The Marginal Value of Adaptive Gradient Methods in Machine Learning

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient desc

[stealthinu]

Adamみたいなニューラルネット学習の最適化手法を使うとSGDよりも汎化能力が下がってたよという論文…　マジすか。また知見が塗り替えられるんだろうか。

[johshisha]

adamよりSGDのほうがよいらしい

[lanius]

SGD. step-sizeのチューニング方法が参考になる。