Scaling Laws for Neural Language Models

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence

[Ryobot]

素晴らしい研究．パラメータ数が重要であり幅や深さは重要じゃない．lossが各変数のべき法則に従ってることを表すプロットに感心した． “The loss scales as a power-law with model size, dataset size, and the amount of compute used for training”