[1811.03600] Measuring the Effects of Data Parallelism on Neural Network Training

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by

[elu_18]

Measuring the Effects of Data Parallelism on Neural Network Training: Large empirical study investigating the tradeoffs between batch sizes, optimal learning rates, and training steps required. https://t.co/vxlsJN37kS https://t.co/2pdnCdAy6N

[stealthinu]

色々な問題やネットワーク毎のバッチサイズと学習回数の関係グラフ。思いのほかバッチサイズ大きくても効率悪くならない。TPUはでかいバッチサイズ指定出来るからでかくしたほうが効率良さそう。