Unfolded ISTA and Orthogonality Regularization -implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Xiaohan sent me the following a few months ago:

Dear Igor,
I'm a long-time fan of your blog and want to share our two recent NIPS papers with you. One is a theory paper on unfolding sparse recovery algorithms into deep networks (and a spotlight oral of NIPS'18); the other is an empirical exploration of applying orthogonality regularizations to training deep CNNs, with many techniques inspired by sparse optimization.
The first paper proves the theoretical linear convergence (as upper bound) of unfolded ISTA networks (LISTA), and proposes two new structures (weight and threshold) to facilitate that fast converegnce and significantly boost performance. The work is done in collaboration with Jialin Liu (http://www.math.ucla.edu/~liujl11/) and Wotao Yin (http://www.math.ucla.edu/~wotaoyin/) in Math@UCLA.
Preprint: https://arxiv.org/abs/1808.10038Github: https://github.com/xchen-tamu/linear-lista-cpss 

The second paper proposes several orthogonality regularizations on CNN weights, by penalizing the distance between the Gram matrix of weights and identity under different metrics. We show that orthogonality evidently accelerates and stabilizes the empirical training convergence, as well as improve as final accuracies. The mose powerful regularization was derived from Restrcited Isometry Property (RIP).

Preprint: https://arxiv.org/abs/1810.09102Github: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality

It would be great if you could distribute their informtion to potential interested audience on nuit-blanche.
Best regards,
--
Xiaohan Chen
Dept. Computer Science & Engineering
Texas A&M University, College Station, TX, U.S.
Webpage: http://people.tamu.edu/~chernxh/

Thanks Xiaohan !

Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds by Xiaohan Chen, Jialin Liu, Zhangyang Wang, Wotao Yin

In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery. We introduce a weight structure that is necessary for asymptotic convergence to the true sparse signal. With this structure, unfolded ISTA can attain a linear convergence, which is better than the sublinear convergence of ISTA/FISTA in general cases. Furthermore, we propose to incorporate thresholding in the network to perform support selection, which is easy to implement and able to boost the convergence rate both theoretically and empirically. Extensive simulations, including sparse vector recovery and a compressive sensing experiment on real image data, corroborate our theoretical results and demonstrate their practical usefulness. We have made our codes publicly available: this https URL.

Can We Gain More from Orthogonality Regularizations in Training Deep CNNs? by Nitin Bansal, Xiaohan Chen, Zhangyang Wang

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: this https URL.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv