Wednesday, May 22, 2019

Uncertainty Quantification for high-dimensional inverse problems

** Nuit Blanche is now on Twitter: @NuitBlog **


Jason sent me the following a few months ago: 

Hi Igor,
Just wanted to draw your attend to a couple of papers we had published recently on uncertainty quantification for high-dimensional inverse problems:
Uncertainty quantification for radio interferometric imaging: I. proximal MCMC methods
Xiaohao Cai, Marcelo Pereyra, Jason D. McEwen
https://arxiv.org/abs/1711.04818
http://dx.doi.org/10.1093/mnras/sty2004


Uncertainty quantification for radio interferometric imaging: II. MAP estimation
Xiaohao Cai, Marcelo Pereyra, Jason D. McEwen
https://arxiv.org/abs/1711.04819
http://dx.doi.org/10.1093/mnras/sty2015


These articles target radio interferometric imaging but the techniques are general so I thought they might be of interest to your readers.

Many thanks!

Best,
Jason
--
www.jasonmcewen.org

Thanks Jason !


Uncertainty quantification is a critical missing component in radio interferometric imaging that will only become increasingly important as the big-data era of radio interferometry emerges. Since radio interferometric imaging requires solving a high-dimensional, ill-posed inverse problem, uncertainty quantification is difficult but also critical to the accurate scientific interpretation of radio observations. Statistical sampling approaches to perform Bayesian inference, like Markov Chain Monte Carlo (MCMC) sampling, can in principle recover the full posterior distribution of the image, from which uncertainties can then be quantified. However, traditional high-dimensional sampling methods are generally limited to smooth (e.g. Gaussian) priors and cannot be used with sparsity-promoting priors. Sparse priors, motivated by the theory of compressive sensing, have been shown to be highly effective for radio interferometric imaging. In this article proximal MCMC methods are developed for radio interferometric imaging, leveraging proximal calculus to support non-differential priors, such as sparse priors, in a Bayesian framework. Furthermore, three strategies to quantify uncertainties using the recovered posterior distribution are developed: (i) local (pixel-wise) credible intervals to provide error bars for each individual pixel; (ii) highest posterior density credible regions; and (iii) hypothesis testing of image structure. These forms of uncertainty quantification provide rich information for analysing radio interferometric observations in a statistically robust manner.

Uncertainty quantification is a critical missing component in radio interferometric imaging that will only become increasingly important as the big-data era of radio interferometry emerges. Statistical sampling approaches to perform Bayesian inference, like Markov Chain Monte Carlo (MCMC) sampling, can in principle recover the full posterior distribution of the image, from which uncertainties can then be quantified. However, for massive data sizes, like those anticipated from the Square Kilometre Array (SKA), it will be difficult if not impossible to apply any MCMC technique due to its inherent computational cost. We formulate Bayesian inference problems with sparsity-promoting priors (motivated by compressive sensing), for which we recover maximum a posteriori (MAP) point estimators of radio interferometric images by convex optimisation. Exploiting recent developments in the theory of probability concentration, we quantify uncertainties by post-processing the recovered MAP estimate. Three strategies to quantify uncertainties are developed: (i) highest posterior density credible regions; (ii) local credible intervals (cf. error bars) for individual pixels and superpixels; and (iii) hypothesis testing of image structure. These forms of uncertainty quantification provide rich information for analysing radio interferometric observations in a statistically robust manner. Our MAP-based methods are approximately 105 times faster computationally than state-of-the-art MCMC methods and, in addition, support highly distributed and parallelised algorithmic structures. For the first time, our MAP-based techniques provide a means of quantifying uncertainties for radio interferometric imaging for realistic data volumes and practical use, and scale to the emerging big-data era of radio astronomy.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Tuesday, May 21, 2019

Unfolded ISTA and Orthogonality Regularization -implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Xiaohan sent me the following a few months ago:
Dear Igor,
I'm a long-time fan of your blog and want to share our two recent NIPS papers with you. One is a theory paper on unfolding sparse recovery algorithms into deep networks (and a spotlight oral of NIPS'18); the other is an empirical exploration of applying orthogonality regularizations to training deep CNNs, with many techniques inspired by sparse optimization.
The first paper proves the theoretical linear convergence (as upper bound) of unfolded ISTA networks (LISTA), and proposes two new structures (weight and threshold) to facilitate that fast converegnce and significantly boost performance. The work is done in collaboration with Jialin Liu (http://www.math.ucla.edu/~liujl11/) and Wotao Yin (http://www.math.ucla.edu/~wotaoyin/) in Math@UCLA.
Preprint: https://arxiv.org/abs/1808.10038Github: https://github.com/xchen-tamu/linear-lista-cpss 

The second paper proposes several orthogonality regularizations on CNN weights, by penalizing the distance between the Gram matrix of weights and identity under different metrics. We show that orthogonality evidently accelerates and stabilizes the empirical training convergence, as well as improve as final accuracies. The mose powerful regularization was derived from Restrcited Isometry Property (RIP).

Preprint: https://arxiv.org/abs/1810.09102Github: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality

It would be great if you could distribute their informtion to potential interested audience on nuit-blanche.
Best regards,
--
Xiaohan Chen
Dept. Computer Science & Engineering
Texas A&M University, College Station, TX, U.S.
Webpage: http://people.tamu.edu/~chernxh/

Thanks Xiaohan !




In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery. We introduce a weight structure that is necessary for asymptotic convergence to the true sparse signal. With this structure, unfolded ISTA can attain a linear convergence, which is better than the sublinear convergence of ISTA/FISTA in general cases. Furthermore, we propose to incorporate thresholding in the network to perform support selection, which is easy to implement and able to boost the convergence rate both theoretically and empirically. Extensive simulations, including sparse vector recovery and a compressive sensing experiment on real image data, corroborate our theoretical results and demonstrate their practical usefulness. We have made our codes publicly available: this https URL.

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: this https URL.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Image Completion Using Low Rank Tensor Method - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Yipeng sent me the following a few months ago:

Dear Igor,

We have a paper on image completion using low rank tensor tree method, which is available at:
https://ieeexplore.ieee.org/abstract/document/8421084The codes are available at:
https://github.com/yipengliu/sttc

Besides, we have another paper which reviews the recent advances in tensor completion for images: https://www.sciencedirect.com/science/article/pii/S0165168418303232

Could you help to post them on your nuit blanche please?

Thank you very much!

Best Regards,

yipeng


Prof. Dr. Yipeng Liu (刘翼鹏)

School of Information and Communication Engineering,
University of Electronic Science and Technology of China (UESTC).
Xiyuan Avenue 2006, Western High-Tech Zone, Chengdu, 611731, China.
webpage1: http://faculty.uestc.edu.cn/yipengliu
webpage2: https://sites.google.com/view/yipeng-liu


Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup
About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Monday, May 20, 2019

Deep Compressed Sensing - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **




Compressed sensing (CS) provides an elegant framework for recovering sparse signals from compressed measurements. For example, CS can exploit the structure of natural images and recover an image from only a few random measurements. CS is flexible and data efficient, but its application has been restricted by the strong assumption of sparsity and costly reconstruction process. A recent approach that combines CS with neural network generators has removed the constraint of sparsity, but reconstruction remains slow. Here we propose a novel framework that significantly improves both the performance and speed of signal recovery by jointly training a generator and the optimisation process for reconstruction via meta-learning. We explore training the measurements with different objectives, and derive a family of models based on minimising measurement errors. We show that Generative Adversarial Nets (GANs) can be viewed as a special case in this family of models. Borrowing insights from the CS perspective, we develop a novel way of improving GANs using gradient information from the discriminator.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn


Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup
 About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog

Saturday, May 18, 2019

Saturday Morning Video: Learning Representations Using Causal Invariance, Léon Bottou

** Nuit Blanche is now on Twitter: @NuitBlog **


The video of Leon Bottou's talk is here and starts at 12 minutes. Coverage on MIT Tech Review and on April's blog.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Saturday Morning Videos: Workshop on the Interface of Machine Learning and Statistical Inference, BANFF

** Nuit Blanche is now on Twitter: @NuitBlog **





Here are the videos of the Workshop on the Interface of Machine Learning and Statistical Inference

Organizers



Description


The Banff International Research Station will host the "Workshop on the Interface of Machine Learning and Statistical Inference" workshop from January 14th to January 19th, 2018.


Over the past thirty years, Machine Learning has proved enormously successful in using large databases to produce automatic prediction methods; they have been used in fields from handwriting recognition to automatic share market investments. However, these techniques produce little insight into the underlying mechanisms the result in the outcomes, nor do they provide statistical quantification of uncertainty. This workshop will bring together statisticians, mathematicians, and computer scientists to build on recent advances that seek to integrate machine learning with more traditional statistical models to obtain both highly accurate and understandable models while quantifying uncertainty about their predictions and conclusions.







Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Friday, May 17, 2019

On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training


Abstract: We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
The attendant video at ICLR is here and it starts at 52 minutes.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Thursday, May 16, 2019

Jobs: Several Postdocs in machine learning and information processing, LIONS, EPFL, Switzerland

** Nuit Blanche is now on Twitter: @NuitBlog **

Volkan just let me know of several openings:

Dear Igor,

I hope this email finds you well.

I would really appreciate it if you could advertise my postdoc position at NB.

best,
=====
Prof. Volkan Cevher
Laboratory for Information and Inference Systems
http://lions.epfl.ch

Here is the announcement:

The LIONS group at EPFL has several openings for postdoctoral fellows for research in machine learning and information processing. Please see our research interests at
https://lions.epfl.ch.

We are looking for candidates with a strong theory background in machine learning, discrete optimization, information theory, statistics, compressive sensing, or other related areas. Strong coding skills is a big plus.
There are two positions that revolve around the following two topics:

1) Bayesian optimization, bandits, and reinforcement learning
We seek to develop online algorithms for Bayesian optimization, as well as related problems such as multi-armed bandits, level-set estimation, and reinforcement learning. The algorithms will be characterized theoretically, and also tested in real-world applications including automated hyperparameter optimization with neural networks and personalized education.
2) Discrete optimization and submodularity with applications to subsampling
We seek to develop techniques for discrete optimization, with submodularity and related concepts playing a key role. These techniques will be targeted at the application of using data in order to optimally subsample for the purpose of performing a given task, such as estimation in compressive sensing or classification in machine learning. Specific applications will also be explored, including medical resonance imaging (MRI) with multiple coils.
3) Continuous optimization theory and methodology
We seek to develop gradient and linear minimization oracle based algorithms for convex and non-convex problems. In particular, we are interested in the marriage of online and offline optimization, universal adaptation, and storage optimal solutions to difficult training problems that range from semidefinite programming to neural network training.

LIONS provides a stimulating, collaborative and fun research environment with state-of-the-art facilities at EPFL. Personal initiative and independent research tasks related with the candidate’s interests are also encouraged.
The working language at EPFL is English.
Candidates should have or be close to finishing a PhD degree in electrical engineering, computer science, applied mathematics, or a related field. Candidates should send their CV, a research statement outlining their expertise and interests, any supplemental information, and a list of at least three references with full contact information to the LIONS Lab Administrator:
Gosia Baltaian (gosia.baltaian@epfl.ch)
======



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

A New Theory for Sketching in Linear Regression -implementation-



Large datasets create opportunities as well as analytic challenges. A recent development is to use random projection or sketching methods for dimension reduction in statistics and machine learning. In this work, we study the statistical performance of sketching algorithms for linear regression. Suppose we randomly project the data matrix and the outcome using a random sketching matrix reducing the sample size, and do linear regression on the resulting data. How much do we lose compared to the original linear regression? The existing theory does not give a precise enough answer, and this has been a bottleneck for using random projections in practice.
In this paper, we introduce a new mathematical approach to the problem, relying on very recent results from asymptotic random matrix theory and free probability theory. This is a perfect fit, as the sketching matrices are random in practice. We allow the dimension and sample sizes to have an arbitrary ratio. We study the most popular sketching methods in a unified framework, including random projection methods (Gaussian and iid projections, uniform orthogonal projections, subsampled randomized Hadamard transforms), as well as sampling methods (including uniform, leverage-based, and greedy sampling). We find precise and simple expressions for the accuracy loss of these methods. These go beyond classical Johnson-Lindenstrauss type results, because they are exact, instead of being bounds up to constants. Our theoretical formulas are surprisingly accurate in extensive simulations and on two empirical datasets.
Implementation is here: https://github.com/liusf15/Sketching-lr

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Wednesday, May 15, 2019

Ce soir, Paris Machine Learning Meetup: Fraud & Bank attacks, Robust Learning, Data Olympics Paris Berlin 2019

** Nuit Blanche is now on Twitter: @NuitBlog **
** The Paris Machine Leanring meetup is on Twitter @ParisMLGroup **


Thank you to Morning coworking for hosting us.

As usual, first come, first served, the room capacity is 120 seats so there is ample space.

Schedule: 6:45PM : doors open / 7PM : talks / 9PM : networking / 10PM : end

Here is the page on the meetup.com platform to register. The streaming will be here:



Our speakers tonight will be (presentations will be uploaded here before the meetup)


Antoine Thorin, BRED-Banque Populaire, From reactivity to proactivity in ATM maintenance
How BRED-Banque Populaire, understand the functioning of ATM, by collecting data from different sources. Visualizes and analyzes data in order to improve the reliability and the availability of ATM.
Gael Varoquaux, INRIA, Learning with missing values
In many application settings, the data have missing features which make data analysis challenging. An abundant literature addresses missing data in an inferential framework: estimating parameters and their variance from incomplete tables. Here, we consider supervised-learning settings: predicting a target when missing values appear in both training and testing data.
https://hal.archives-ouvertes.fr/hal-02024202v2

Gregoire Martinon, Quantmetry, Real-time detection of attacks on ATMs
The general philosophy is to convert video into time series. The standard anomaly detection artillery can then be deployed over the generated time series. To make such a conversion, a combination of tracking detector is used: a deep learning detector allows to locate with great precision an area of interest (face, hand, skeleton), while a tracking device (MOSSE) is in charge of tracking the target from one frame to another. The advantage of the tracker is that it is extremely fast and compensates for the high calculation time of the detector.


Fabien Vauchelles, Zelros
Data Science Olympics 2019, Paris and Berlin



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Printfriendly