Orthogonal Initialization in Convolutional Layers · Hendrik J. Weideman

Orthogonal Initialization in Convolutional Layers 12 Dec 2015 In Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Saxe, McClelland, and Ganguli investigate the question of how to initialize the weights in deep neural networks by studying the learning dynamics of deep linear neural networks. In particular, they suggest that the weight matrix should be chosen as a