A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using transformers, accelerate and bitsandbytes

Introduction Language models are becoming larger all the time. At the time of this writing, PaLM has 540B parameters, OPT, GPT-3, and BLOOM have around 176B parameters, and we are trending towards even larger models. Below is a diagram showing the size of some recent language models. Therefore, these models are hard to run on easily accessible devices. For example, just to do inference on BLOOM-17

[deejayroka]

“大規模なモデルのメモリ フットプリントを 1/2 に削減する Int8 推論”