Running BLOOM 176B inference on ABCI - Qiita

With evergrowing size of recent pretrained language models (or foundation models), running inference of these models poses many challenges. In this post, I will explain steps needed to run GPT-3 sized models (e.g., BLOOM) with a single 8xA100 (40GB) machine on ABCI using Huggingface inference server, DeepSpeed and bitsandbytes. Since dependent libraries are under rapid development, it is subject t