Allreduce (or MPI) vs. Parameter server approaches – Machine Learning (Theory)

In the last 7 years or so there has been quite a bit of work on parallel machine learning approaches, enough that I felt like a summary might be helpful both for myself and others. In each case, I put in the earliest known citation. If I missed something please comment. One basic dividing line between parallel approaches is single-machine vs. multi-machine. Multi-machine approaches offer the poten