Decoupled Neural Interfaces using Synthetic Gradients

Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated. In

[rjj]

少々の誤差を許容することで大きな並列性を実現？（Approximate computingとかも同じ発想。）　なあに、少しくらいの誤差はかえってrobustになる、みたいなノリ？

[rishida]

RNNとかを高速に収束できる

[elu_18]

Synthetic Gradients すごそう…．BPのようにグローバルで誤差伝播するとモジュールのパラメータ更新が同期的になり非効率だが，これを各レイヤにローカルなNNを導入して近似した誤差でレイヤ間の非同期更新を実現． https://t.