Alec Radford has created some great animations comparing optimization algorithms SGD, Momentum, NAG, Adagrad, Adadelta, RMSprop (unfortunately no Adam) on low dimensional problems. Also check out his presentation on RNNs. "Noisy moons: This is logistic regression on noisy moons dataset from sklearn which shows the smoothing effects of momentum based techniques (which also results in over shooting
![Alec Radford's animations for optimization algorithms](https://cdn-ak-scissors.b.st-hatena.com/image/square/422afe6e4cf7848b6ab8070366a7747319fd9279/height=288;version=1;width=512/https%3A%2F%2Fblogger.googleusercontent.com%2Fimg%2Fb%2FR29vZ2xl%2FAVvXsEjUvhiRVEra6dZlJFFoR-lYbtVpFCmM41IYACOLukmWUHSnsqABa517IuKtYLHMNXj75wTQtHbr_AlGxYVItrUA3QHBvecLUjpxdyGAiJxXO0uMMV9bD7NW54ePr9OH_JJx3x5I%2Fw1200-h630-p-k-no-nu%2Fs25RsOr%2B-%2BImgur.gif)