A Primer on the Inner Workings of Transformer-based Language Models
The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architect
PowerPoint Presentation
NTECH2023 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Novel Transformer Model Based Clustering Method for Standard Cell Design Automation Chia-Tung (Mark) Ho, Ajay Chandna, David Guan, Alvin Ho, Minsoo Kim, Yaguang Li and Haoxing (Mark) Ren | ISPD 2024 NTECH2023 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. OUTLINE • Introduction • Novel Transformer Model Based Clustering Method • Experimental Results • Con
NeurIPS 2023 Tutorial: Reconsidering Overfitting in the Age of Overparameterized Models
NeurIPS 2023 Tutorial: Reconsidering Overfitting in the Age of Overparameterized Models ` Large, overparameterized models such as neural networks are now the workhorses of modern machine learning. These models are often trained to near-zero error on noisy datasets and simultaneously generalize well to unseen data, in contrast to the textbook intuition regarding the perils of overfitting. At the sa
Zoology (Blogpost 2): Simple, Input-Dependent, and Sub-Quadratic Sequence Mixers
Table 1: Perplexity of 355 million parameter models trained for 10 billion tokens on the Pile. Yet, some subquadratic gated-convolutions match attention on the non AR slice! Can we capture the strengths of both gated convolutions and attention in one purely sub-quadratic architecture? We find the AR gap is because gated convolution models (e.g. Hyena, H3, RWKV, RetNet) need model dimension that sc
はじめての自然言語処理 Transformer 系モデルの推論高速化の検証 | オブジェクトの広場
今回は Transformer 系のモデル、具体的には BERT, T5, GPT の推論を高速化してみます。高速化手法として FasterTransformer, Torch-TensorRT, AWS Neuron を用い、素 の transfomers に比べ、どの程度速くなるか（ならないか）、利点・欠点を確認してみましょう。 1. はじめに 今回は Transformer 系のモデル、具体的には BERT, T5, GPT の推論を様々な技術を使って高速化してみます。 高速化の元ネタは Hugging Face の transformers1 縛りとして、素の transformers で推論する場合に比べ、 どの程度速くなるか（ならないか）見てみましょう。 推論を高速化する技術としては FasterTransfomer2, Torch-TensorRT3, AWS Neuron(
On Layer Normalization in the Transformer Architecture
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers carefully (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand $\textit{what complicates Transformer training}$ from both empirical and theoretical perspectives. Our analy
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
RMT: Retentive Networks Meet Vision Transformers
Personal and Psychological Dimensions of AI Researchers Confronting AI Catastrophic Risks - Yoshua Bengio
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
GCVit/models/gc_vit.py at main · NVlabs/GCVit
ResiDual: Transformer with Dual Residual Connections
Accelerated PyTorch 2 Transformers
GitHub - NVlabs/GCVit: [ICML 2023] Official PyTorch implementation of Global Context Vision Transformers
https://openreview.net/pdf?id=JroZRaRw7Eu
GitHub - Dao-AILab/flash-attention: Fast and memory-efficient exact attention
DETRs Beat YOLOs on Real-time Object Detection
GitHub - NVIDIA/FasterTransformer: Transformer related optimization, including BERT, GPT
bigscience/train/lessons-learned.md at master · bigscience-workshop/bigscience
Transformer Engine documentation — Transformer Engine 1.5.0 documentation