LaMDA: our breakthrough conversation technology
In-context Learning and Induction Heads
"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induc
Thinking Like Transformers
What is the computational model behind a Transformer? Where recurrent neural networks have direct parallels in finite state machines, allowing clear discussion and thought around architecture variants or trained models, Transformers have no such familiar parallel. In this paper we aim to change that, proposing a computational model for the transformer-encoder in the form of a programming language.
Thinking like Transformer
Thinking Like Transformers Paper by Gail Weiss, Yoav Goldberg, Eran Yahav Blog by Sasha Rush and Gail Weiss Library and Interactive Notebook: srush/raspy Transformer models are foundational to AI systems. There are now countless explanations of “how transformers work?” in the sense of the architecture diagram at the heart of transformers. svg However this diagram does not provide any intuition int
Visual Question Answering
VQA is a new dataset containing open-ended questions about images. These questions require an understanding of vision, language and commonsense knowledge to answer. 265,016 images (COCO and abstract scenes) At least 3 questions (5.4 questions on average) per image 10 ground truth answers per question 3 plausible (but likely incorrect) answers per question Automatic evaluation metric
Unsupervised Speech Recognition
Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe. This paper describes wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Transformer Circuits Thread
Relating transformers to models and neural representations of the hippocampal formation
Transformer-based normative modelling for anomaly detection of early schizophrenia
Diagnostic Spatio-temporal Transformer with Faithful Encoding
New Tool: the Residual Stream Viewer — AI Alignment Forum
Image Captioners Are Scalable Vision Learners Too
Self-attention Does Not Need $O(n^2)$ Memory
DeBERTa: Decoding-enhanced BERT with Disentangled Attention