Here is a more mathematical way of dealing with nonlinearities in DNNs.

This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider M=1mYY∗ with Y=f(WX) where W and X are random rectangular matrices with i.i.d. centered entries. The function f is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where W and X have sub-Gaussian tails and f is real analytic. This extends a previous result where the case of Gaussian matrices W and X is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:

**: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup**

*Paris Machine Learning***: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog**

*About LightOn*__: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv__

**About myself**