On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the mod

[elu_18]

汎化については、どの局所解がより汎化するかみたいなのに興味があって、例えば みたいにフラットな解が汎化するのではという話があるけど、こういうのちゃんと調べるにはどういうこと考えたらいいかよくわからない