Bringing HPC Techniques to Deep Learning - Andrew Gibiansky

Note: This blog post was originally written for the Baidu Research technical blog, and is reproduced here with their permission. Since then, these ideas have evolved and been incorporated into the excellent Horovod library by Uber, which is the easiest way to use MPI or NCCL for multi-GPU or multi-node deep learning applications. Summary: Neural networks have grown in scale over the past several y