Answer (1 of 5): From my practical point of view, AdaDelta yields slower but better local minima in most cases and its convergence speed is depended on the initial learning rate since a bad choice spends to much time for stabilizing. AdaGrad is more robust to initial learning rate, it converges ...