Scaling-up BERT Inference on CPU (Part 1)

1. Context and Motivations Back in October 2019, my colleague Lysandre Debut published a comprehensive (at the time) inference performance benchmarking blog (1). Since then, 🤗 transformers (2) welcomed a tremendous number of new architectures and thousands of new models were added to the 🤗 hub (3) which now counts more than 9,000 of them as of first quarter of 2021. As the NLP landscape keeps tr

[rawwell]

“Instead of throwing more cores to the task as you would do in the core count scaling setup, now we will be using more model instances. ”