Wednesday, May 22, 2019

Uncertainty Quantification for high-dimensional inverse problems

** Nuit Blanche is now on Twitter: @NuitBlog **


Jason sent me the following a few months ago: 

Hi Igor,
Just wanted to draw your attend to a couple of papers we had published recently on uncertainty quantification for high-dimensional inverse problems:
Uncertainty quantification for radio interferometric imaging: I. proximal MCMC methods
Xiaohao Cai, Marcelo Pereyra, Jason D. McEwen
https://arxiv.org/abs/1711.04818
http://dx.doi.org/10.1093/mnras/sty2004


Uncertainty quantification for radio interferometric imaging: II. MAP estimation
Xiaohao Cai, Marcelo Pereyra, Jason D. McEwen
https://arxiv.org/abs/1711.04819
http://dx.doi.org/10.1093/mnras/sty2015


These articles target radio interferometric imaging but the techniques are general so I thought they might be of interest to your readers.

Many thanks!

Best,
Jason
--
www.jasonmcewen.org

Thanks Jason !


Uncertainty quantification is a critical missing component in radio interferometric imaging that will only become increasingly important as the big-data era of radio interferometry emerges. Since radio interferometric imaging requires solving a high-dimensional, ill-posed inverse problem, uncertainty quantification is difficult but also critical to the accurate scientific interpretation of radio observations. Statistical sampling approaches to perform Bayesian inference, like Markov Chain Monte Carlo (MCMC) sampling, can in principle recover the full posterior distribution of the image, from which uncertainties can then be quantified. However, traditional high-dimensional sampling methods are generally limited to smooth (e.g. Gaussian) priors and cannot be used with sparsity-promoting priors. Sparse priors, motivated by the theory of compressive sensing, have been shown to be highly effective for radio interferometric imaging. In this article proximal MCMC methods are developed for radio interferometric imaging, leveraging proximal calculus to support non-differential priors, such as sparse priors, in a Bayesian framework. Furthermore, three strategies to quantify uncertainties using the recovered posterior distribution are developed: (i) local (pixel-wise) credible intervals to provide error bars for each individual pixel; (ii) highest posterior density credible regions; and (iii) hypothesis testing of image structure. These forms of uncertainty quantification provide rich information for analysing radio interferometric observations in a statistically robust manner.

Uncertainty quantification is a critical missing component in radio interferometric imaging that will only become increasingly important as the big-data era of radio interferometry emerges. Statistical sampling approaches to perform Bayesian inference, like Markov Chain Monte Carlo (MCMC) sampling, can in principle recover the full posterior distribution of the image, from which uncertainties can then be quantified. However, for massive data sizes, like those anticipated from the Square Kilometre Array (SKA), it will be difficult if not impossible to apply any MCMC technique due to its inherent computational cost. We formulate Bayesian inference problems with sparsity-promoting priors (motivated by compressive sensing), for which we recover maximum a posteriori (MAP) point estimators of radio interferometric images by convex optimisation. Exploiting recent developments in the theory of probability concentration, we quantify uncertainties by post-processing the recovered MAP estimate. Three strategies to quantify uncertainties are developed: (i) highest posterior density credible regions; (ii) local credible intervals (cf. error bars) for individual pixels and superpixels; and (iii) hypothesis testing of image structure. These forms of uncertainty quantification provide rich information for analysing radio interferometric observations in a statistically robust manner. Our MAP-based methods are approximately 105 times faster computationally than state-of-the-art MCMC methods and, in addition, support highly distributed and parallelised algorithmic structures. For the first time, our MAP-based techniques provide a means of quantifying uncertainties for radio interferometric imaging for realistic data volumes and practical use, and scale to the emerging big-data era of radio astronomy.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Tuesday, May 21, 2019

Image Completion Using Low Rank Tensor Method - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Yipeng sent me the following a few months ago:

Dear Igor,

We have a paper on image completion using low rank tensor tree method, which is available at:
https://ieeexplore.ieee.org/abstract/document/8421084The codes are available at:
https://github.com/yipengliu/sttc

Besides, we have another paper which reviews the recent advances in tensor completion for images: https://www.sciencedirect.com/science/article/pii/S0165168418303232

Could you help to post them on your nuit blanche please?

Thank you very much!

Best Regards,

yipeng


Prof. Dr. Yipeng Liu (刘翼鹏)

School of Information and Communication Engineering,
University of Electronic Science and Technology of China (UESTC).
Xiyuan Avenue 2006, Western High-Tech Zone, Chengdu, 611731, China.
webpage1: http://faculty.uestc.edu.cn/yipengliu
webpage2: https://sites.google.com/view/yipeng-liu


Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup
About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Monday, May 20, 2019

Deep Compressed Sensing - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **




Compressed sensing (CS) provides an elegant framework for recovering sparse signals from compressed measurements. For example, CS can exploit the structure of natural images and recover an image from only a few random measurements. CS is flexible and data efficient, but its application has been restricted by the strong assumption of sparsity and costly reconstruction process. A recent approach that combines CS with neural network generators has removed the constraint of sparsity, but reconstruction remains slow. Here we propose a novel framework that significantly improves both the performance and speed of signal recovery by jointly training a generator and the optimisation process for reconstruction via meta-learning. We explore training the measurements with different objectives, and derive a family of models based on minimising measurement errors. We show that Generative Adversarial Nets (GANs) can be viewed as a special case in this family of models. Borrowing insights from the CS perspective, we develop a novel way of improving GANs using gradient information from the discriminator.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn


Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup
 About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog

Saturday, May 18, 2019

Saturday Morning Video: Learning Representations Using Causal Invariance, Léon Bottou

** Nuit Blanche is now on Twitter: @NuitBlog **


The video of Leon Bottou's talk is here and starts at 12 minutes. Coverage on MIT Tech Review and on April's blog.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Saturday Morning Videos: Workshop on the Interface of Machine Learning and Statistical Inference, BANFF

** Nuit Blanche is now on Twitter: @NuitBlog **





Here are the videos of the Workshop on the Interface of Machine Learning and Statistical Inference

Organizers



Description


The Banff International Research Station will host the "Workshop on the Interface of Machine Learning and Statistical Inference" workshop from January 14th to January 19th, 2018.


Over the past thirty years, Machine Learning has proved enormously successful in using large databases to produce automatic prediction methods; they have been used in fields from handwriting recognition to automatic share market investments. However, these techniques produce little insight into the underlying mechanisms the result in the outcomes, nor do they provide statistical quantification of uncertainty. This workshop will bring together statisticians, mathematicians, and computer scientists to build on recent advances that seek to integrate machine learning with more traditional statistical models to obtain both highly accurate and understandable models while quantifying uncertainty about their predictions and conclusions.







Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Friday, May 17, 2019

On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training


Abstract: We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
The attendant video at ICLR is here and it starts at 52 minutes.



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Thursday, May 16, 2019

Jobs: Several Postdocs in machine learning and information processing, LIONS, EPFL, Switzerland

** Nuit Blanche is now on Twitter: @NuitBlog **

Volkan just let me know of several openings:

Dear Igor,

I hope this email finds you well.

I would really appreciate it if you could advertise my postdoc position at NB.

best,
=====
Prof. Volkan Cevher
Laboratory for Information and Inference Systems
http://lions.epfl.ch

Here is the announcement:

The LIONS group at EPFL has several openings for postdoctoral fellows for research in machine learning and information processing. Please see our research interests at
https://lions.epfl.ch.

We are looking for candidates with a strong theory background in machine learning, discrete optimization, information theory, statistics, compressive sensing, or other related areas. Strong coding skills is a big plus.
There are two positions that revolve around the following two topics:

1) Bayesian optimization, bandits, and reinforcement learning
We seek to develop online algorithms for Bayesian optimization, as well as related problems such as multi-armed bandits, level-set estimation, and reinforcement learning. The algorithms will be characterized theoretically, and also tested in real-world applications including automated hyperparameter optimization with neural networks and personalized education.
2) Discrete optimization and submodularity with applications to subsampling
We seek to develop techniques for discrete optimization, with submodularity and related concepts playing a key role. These techniques will be targeted at the application of using data in order to optimally subsample for the purpose of performing a given task, such as estimation in compressive sensing or classification in machine learning. Specific applications will also be explored, including medical resonance imaging (MRI) with multiple coils.
3) Continuous optimization theory and methodology
We seek to develop gradient and linear minimization oracle based algorithms for convex and non-convex problems. In particular, we are interested in the marriage of online and offline optimization, universal adaptation, and storage optimal solutions to difficult training problems that range from semidefinite programming to neural network training.

LIONS provides a stimulating, collaborative and fun research environment with state-of-the-art facilities at EPFL. Personal initiative and independent research tasks related with the candidate’s interests are also encouraged.
The working language at EPFL is English.
Candidates should have or be close to finishing a PhD degree in electrical engineering, computer science, applied mathematics, or a related field. Candidates should send their CV, a research statement outlining their expertise and interests, any supplemental information, and a list of at least three references with full contact information to the LIONS Lab Administrator:
Gosia Baltaian (gosia.baltaian@epfl.ch)
======



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

A New Theory for Sketching in Linear Regression -implementation-



Large datasets create opportunities as well as analytic challenges. A recent development is to use random projection or sketching methods for dimension reduction in statistics and machine learning. In this work, we study the statistical performance of sketching algorithms for linear regression. Suppose we randomly project the data matrix and the outcome using a random sketching matrix reducing the sample size, and do linear regression on the resulting data. How much do we lose compared to the original linear regression? The existing theory does not give a precise enough answer, and this has been a bottleneck for using random projections in practice.
In this paper, we introduce a new mathematical approach to the problem, relying on very recent results from asymptotic random matrix theory and free probability theory. This is a perfect fit, as the sketching matrices are random in practice. We allow the dimension and sample sizes to have an arbitrary ratio. We study the most popular sketching methods in a unified framework, including random projection methods (Gaussian and iid projections, uniform orthogonal projections, subsampled randomized Hadamard transforms), as well as sampling methods (including uniform, leverage-based, and greedy sampling). We find precise and simple expressions for the accuracy loss of these methods. These go beyond classical Johnson-Lindenstrauss type results, because they are exact, instead of being bounds up to constants. Our theoretical formulas are surprisingly accurate in extensive simulations and on two empirical datasets.
Implementation is here: https://github.com/liusf15/Sketching-lr

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Wednesday, May 15, 2019

Ce soir, Paris Machine Learning Meetup: Fraud & Bank attacks, Robust Learning, Data Olympics Paris Berlin 2019

** Nuit Blanche is now on Twitter: @NuitBlog **
** The Paris Machine Leanring meetup is on Twitter @ParisMLGroup **


Thank you to Morning coworking for hosting us.

As usual, first come, first served, the room capacity is 120 seats so there is ample space.

Schedule: 6:45PM : doors open / 7PM : talks / 9PM : networking / 10PM : end

Here is the page on the meetup.com platform to register. The streaming will be here:



Our speakers tonight will be (presentations will be uploaded here before the meetup)


Antoine Thorin, BRED-Banque Populaire, From reactivity to proactivity in ATM maintenance
How BRED-Banque Populaire, understand the functioning of ATM, by collecting data from different sources. Visualizes and analyzes data in order to improve the reliability and the availability of ATM.
Gael Varoquaux, INRIA, Learning with missing values
In many application settings, the data have missing features which make data analysis challenging. An abundant literature addresses missing data in an inferential framework: estimating parameters and their variance from incomplete tables. Here, we consider supervised-learning settings: predicting a target when missing values appear in both training and testing data.
https://hal.archives-ouvertes.fr/hal-02024202v2

Gregoire Martinon, Quantmetry, Real-time detection of attacks on ATMs
The general philosophy is to convert video into time series. The standard anomaly detection artillery can then be deployed over the generated time series. To make such a conversion, a combination of tracking detector is used: a deep learning detector allows to locate with great precision an area of interest (face, hand, skeleton), while a tracking device (MOSSE) is in charge of tracking the target from one frame to another. The advantage of the tracker is that it is extremely fast and compensates for the high calculation time of the detector.


Fabien Vauchelles, Zelros
Data Science Olympics 2019, Paris and Berlin



Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

One-shot distributed ridge regression in high dimensions - implementation-


In many areas, practitioners need to analyze large datasets that challenge conventional single-machine computing. To scale up data analysis, distributed and parallel computing approaches are increasingly needed. Datasets are spread out over several computing units, which do most of the analysis locally, and communicate short messages. Here we study a fundamental and highly important problem in this area: How to do ridge regression in a distributed computing environment? Ridge regression is an extremely popular method for supervised learning, and has several optimality properties, thus it is important to study. We study one-shot methods that construct weighted combinations of ridge regression estimators computed on each machine. By analyzing the mean squared error in a high dimensional random-effects model where each predictor has a small effect, we discover several new phenomena.
1. Infinite-worker limit: The distributed estimator works well for very large numbers of machines, a phenomenon we call "infinite-worker limit".
2. Optimal weights: The optimal weights for combining local estimators sum to more than unity, due to the downward bias of ridge. Thus, all averaging methods are suboptimal.
We also propose a new optimally weighted one-shot ridge regression algorithm. We confirm our results in simulation studies and using the Million Song Dataset as an example. There we can save at least 100x in computation time, while nearly preserving test accuracy.
An implementation can be found here: https://github.com/dobriban/dist


Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine LearningMeetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOnNewsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myselfLightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Printfriendly