Thursday, June 22, 2017

Videos: Structured Regularization for High-Dimensional Data Analysis: Submodular Functions,computing the Non-computable via sparsity, the SDP approach to graph clustering, the hidden clique problem, robust deconvolution




On foundational computation problems in l1 and TV regularization
A.Hansen - 3/4 - 20/06/2017



Computing the Non-computable via sparsity
On foundational computation barriers in l1 and TV regularization
A.Hansen - 4/4 - 20/06/2017


Submodular Functions: from Discrete to Continuous Domains , Francis Bach (INRIA)

Abstract: Submodular set-functions have many applications in combinatorial optimization, as they can be minimized and approximately maximized in polynomial time. A key element in many of the algorithms and analyses is the possibility of extending the submodular set-function to a convex function, which opens up tools from convex optimization. Submodularity goes beyond set-functions and has naturally been considered for problems with multiple labels or for functions defined on continuous domains, where it corresponds essentially to cross second-derivatives being nonpositive. In this talk, I will show that most results relating submodularity and convexity for set-functions can be extended to all submodular functions. In particular, (a) I will naturally define a continuous extension in a set of probability measures, (b) show that the extension is convex if and only if the original function is submodular, (c) prove that the problem of minimizing a submodular function is equivalent to a typically non-smooth convex optimization problem. Most of these extensions from the set-function situation are obtained by drawing links with the theory of multi-marginal optimal transport, which provides also a new interpretation of existing results for set-functions. I will then provide practical algorithms to minimize generic submodular functions on discrete domains, with associated convergence rates, and an application to proximal operators for non-convex penalty functions. Preprint available here.



Andrea Montanari (Stanford): The semidefinite programming approach to graph clustering.

Andrea Montanari (Stanford): Local algorithms and graphical models. The hidden clique problem.

Carlos Fernandez-Granda (NYU): A sampling theorem for robust deconvolution

Abstract: In the 70s and 80s geophysicists proposed using l1-norm regularization for deconvolution problem in the context of reflection seismology. Since then such methods have had a great impact in high-dimensional statistics and in signal-processing applications, but until recently their performance on the original deconvolution problem was not well understood theoretically. In this talk we provide an analysis of optimization-based methods for the deconvolution problem, including results on irregular sampling and sparse corruptions that highlight the modeling flexibility of these techniques.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, June 21, 2017

Videos: Structured Regularization for High-Dimensional Data Analysis: Compressed Sensing: Structure and Imaging & Matrix and graph estimation



Lectures 1: Compressed Sensing: Structure and Imaging
   
 Lectures 2: Compressed Sensing: Structure and Imaging Anders Hansen (Cambridge) 

Lectures 1 and 2: Compressed Sensing: Structure and Imaging Abstract: The above heading is the title of a new book to be published by Cambridge University Press. In these lectures I will cover some of the main issues discussed in this monograph/textbook. In particular, we will discuss how the key to the success of compressed sensing applied in imaging lies in the structure. For example images are not just sparse in an X-let expansion, they have a very specific sparsity structure in levels according to the X-let scales. Similarly, when considering Total Variation, the gradient coefficients are also highly structured. Moreover, in most realistic sampling scenarios, the sampling operator combined with any X-let transform yields a matrix with a very specific coherence structure. The key to successfully use compressed sensing is therefore to understand how to utilise these structures in an optimal way, in particular in the sampling procedure. In addition, as the coherence and sparsity structures have very particular asymptotic behaviour, the performance of compressed sensing varies greatly with dimension, and so does the optimal way of sampling. Fortunately, there is now a developed theory that can guide the user in detail on how to optimise the use of compressed sensing in inverse and imaging problems. I will cover several of the key aspects of the theory accompanied with real-world examples from Magnetic Resonance Imaging (MRI), Nuclear Magnetic Resonance (NMR), Surface Scattering, Electron Microscopy, Fluorescence Microscopy etc. Recommended readings: (lectures 1 and 2) Chapter 4, 6 and 12 in “A mathematical introduction to compressed sensing” (Foucard/Rauhut) Breaking the coherence barrier: A new theory for compressed sensing On asymptotic structure in compressed sensing Structure dependent sampling in compressed sensing: theoretical guarantees for tight frames
   

Andrea Montanari (Stanford): Matrix and graph estimation 

 Abstract: Many statistics and unsupervised learning problems can be formalized as estimating a structured matrix or a graph from noisy or incomplete observations. These problems present a large variety of challenges, and an intriguing interplay between computational and statistical barriers. I will provide an introduction to recent work in the area, with an emphasis on general methods and unifying themes. 1) Random matrix theory and spectral methods. 2) The semidefinite programming approach to graph clustering. 3) Local algorithms and graphical models. The hidden clique problem. 4) Non-negative matrix factorization.






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Tuesday, June 20, 2017

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Here is another way of cutting down Deep Neural Nets training time.

Learning Deep ResNet Blocks Sequentially using Boosting Theory by Furong Huang, Jordan Ash, John Langford, Robert Schapire
Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct T weak module classifiers, each contains two of the T layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of T "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth T if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with l1 norm bounded weights.  






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, June 19, 2017

Job; Postdoc Opening in Statistical Learning and Optimization, USC

Jason just sent me the following last week:


Dear Igor, 
I am looking for a postdoc. Could you please post the following on Nuit Blanche? BTW, I greatly enjoy reading your blog.

Postdoc Opening in Statistical Learning and Optimization 
Applications are invited for a postdoc position to work with Jason Lee at the University of Southern California. Prospective applicants should have a PhD in statistics, machine learning, signal processing, or applied mathematics. The successful candidate will have the flexibility to choose a focus within non-convex optimization, deep learning, and statistical inference and learning theory. Applications from candidates with a strong background in Optimization, Statistics, and Machine Learning are particularly welcome. 
Applicants are requested to send a CV, 3 representative papers, and the contact details of three references. 
Please send applications and informal inquiries to jasonlee(AT)marshall(DOT)usc(DOT)edu.

Best,
Jason


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Job: Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence, University of Surrey, UK

Mark just me the folloing last week:

Dear Igor,
This job opportunity may interest some readers of Nuit Blanche?
Best wishes,
Mark
---
Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence
Department of Computer Science, University of Surrey
Salary: GBP 39,324 to GBP 57,674 per annum
Closing Date: Wednesday 28 June 2017
The Department of Computer Science at the University of Surrey invites applications for a post of Lecturer/Senior Lecturer (similar to Assistant/Associate Professor) in Machine Learning and Computational Intelligence. We aim to attract outstanding candidates to the Nature Inspired Computing and Engineering (NICE) Group, who will have strong visions for research, a growing international research profile, a passion for teaching, and who value collaborative research and working in a team. This is an exciting opportunity in a department that is growing its reputation for delivering quality interdisciplinary and applied research based on strong foundations.
The post-holder will enhance or complement one or more of the following areas: evolutionary computation, computational intelligence, machine learning, and computational neuroscience, with applications to data-driven optimisation, data analytics and big data, secure machine learning, self-organising and autonomous systems, healthcare and bioinformatics. It is expected that the post-holder will also contribute to high quality teaching at undergraduate and post-graduate level, for example in data science.
Applicants to the post should have a PhD in a relevant subject or equivalent professional experience. An ability to produce high quality outputs is also required. The appointed candidate will be expected to contribute to all aspects of the Department's activities.
We are looking for individuals that can inspire students through their curiosity for leading-edge aspects of technology. In particular, the teaching duties of the role include delivering high quality teaching to all levels of students, supervising undergraduate project students and postgraduate dissertations and contributing to the teaching of computational intelligence, machine learning, data science, as well as other practical areas of Computer Science, such as object-oriented programming and advanced algorithms.
The Department of Computer Science embodies the ethos of "applying theory into practice" across its research and teaching activities. Its research activities are focused into two research groups: Nature Inspired Computing and Engineering (NICE) and Secure Systems. The research on evolutionary optimization and computational intelligence is internationally recognized and has a close collaboration across the University and internationally with academia and industry. The department has excellent links with departments across the University, as well as with industry, such as Honda, The Pirbright Institute, National Physical Laboratory, and Moorfields Eye Hospital.
This is a full-time and permanent position. The post is available from September 2017 but there is flexibility in the start date. The University is located close to the Surrey Research Park in Guildford and within easy commuting distance of London.
For further details and an informal discussion please contact Professor Yaochu Jin, Head of the NICE Group, yaochu.jin@surrey.ac.uk, or Professor Mark Plumbley, Head of Department at m.plumbley@surrey.ac.uk.
The University and the Department specifically are committed to building a culturally diverse organisation and strongly encourages applications from female and minority candidates. The Department shares the Athena SWAN ideals with respect to the equality and diversity agenda.
Presentations and interviews will take place on 13 July 2017. During the interview process there will be an opportunity to meet members of the NICE group in the evening of 12 July 2017 for an informal dinner and to meet with other colleagues within the Department on an informal basis.
For further details and information on how to apply, visit https://jobs.surrey.ac.uk/039217
--
Prof Mark D Plumbley
Interim Head of Department of Computer Science
Professor of Signal Processing
Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey - Celebrating 50 years in Guildford
Email (Head of Department matters): cs-hod@list.surrey.ac.uk
Email (Other matters): m.plumbley@surrey.ac.uk
University of Surrey, Guildford, Surrey, GU2 7XH, UK



Sunday, June 18, 2017

Sunday Morning Videos: NIPS 2016 workshop on nonconvex optimizations.



Hossein just mentioned this on his twitter:

Here is the list of talks and videos:
  • Nando de Freitas (Learning to Optimize) [Slides] [Video]
  • Morning Poster Spotlight (papers #1 to #8) [Slides] [Video]
  • Jean Lasserre (Moment-LP and Moment-SOS Approaches in Optimization) [Slides] [Video]
  • Surya Ganguli (Non-convexity in the error landscape and the expressive capacity of deep neural networks) [Slides] [Video
  • Ryan Adams (Leveraging Structure in Bayesian Optimization) [Slides] [Video]
  • Stefanie Jegelka (Submodular Optimization and Nonconvexity) [Slides] [Video]
  • Suvrit Sra (Taming Non-Convexity via Geometry) [Slides] [Video]
  • Francis Bach (Submodular Functions: from Discrete to Continuous Domains) [Slides] [Video]
  • Panel Discussion [Video]
  • Afternoon Poster Spotlight (papers #9 to #16) [Slides] [Video]





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, June 16, 2017

FreezeOut: Accelerate Training by Progressively Freezing Layers - implementation -

What initially looks like playing with hyperparameters brings new life to a somewhat older approach. From Alex's tweet:







The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.


DenseNet is at: http://github.com/bamos/densenet.pytorch 
while FreezeOut is here: http://github.com/ajbrock/FreezeOut 




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, June 15, 2017

Self-Normalizing Neural Networks - implementation -

There is some commotion on the interweb about this new nonlinear activation function that removes the need for tricks like Batch Normalization and seem to beat recent architectures like Resnets ...and there is the long appendix thingy too. Wow !




Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.
Some mention on the web:


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, June 14, 2017

CSjob: Postdoc, Optimisation for Matrix Factorisation, Toulouse, France

Cedric came to visit us yesterday at LightOn and he let us know about his search for a postdoc, here is the announcement:

Postdoc opening Optimisation for matrix factorisation
Project FACTORY 
 

New paradigms for latent factor estimation Announcement Applications are invited for a 2-year postdoc position to work with Cedric Fevotte (CNRS senior scientist) on matrix factorisation techniques for data processing. The position is part of project FACTORY (New paradigms for latent factor estimation), funded by the European Research Council under a Consolidator Grant (2016-2021). The successful candidate will be based in Toulouse, France.  
Project description  
The project concerns matrix factorisation and dictionary learning for data analysis at large, with an emphasis on statistical estimation in mean-parametrised exponential models, non-convex optimisation, stochastic algorithms & approximate inference, representation learning, and applications to audio signal processing, remote sensing & data mining.
The European Research Council offers highly competitive funding for scientific excellence. The successful candidate will enjoy an inspiring and resourceful environment, with the possibility of travelling to conferences and visiting other national or international labs.
More information at http://www.irit.fr/~Cedric.Fevotte/factory/  
Host institution and place of work 
The successful candidate will be employed by the Centre National de la Recherche Scientifique (CNRS, the National Center for Scientific Research). CNRS is the largest state-funded research organisation in France, involved in all scientific fields. FACTORY is hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), a joint laboratory of CNRS and Toulouse universities & engineering schools. IRIT is among the largest computer & information sciences labs in France. Toulouse is the fourth largest city in France, the capital of the Midi-Pyr´en´ees region in the South-West of France, and is praised for its high quality of living. The physical location for the project is the ENSEEIHT campus (Signal & Communications group), in a lively neighbourhood of the city center.  
Candidate profile and application  

Prospective applicants should have a PhD in machine learning, signal processing, applied mathematics, statistics, or a related discipline, good programming skills, and good communication skills in English, both written and oral. The successful candidate will have the flexibility to choose a topic within the range of the project, according to his/her experience and preferences. Applications from candidates with a good background in optimisation or stochastic simulation are particularly encouraged. The net monthly salary is 2300e for researchers with less than 2 years of professional experience after the PhD, and starts from 2700e in other cases. The position comes with health insurance & other social benefits. Applicants are requested to send a CV, a brief statement of research interests and the contact details of two referees in a single PDF file. Applications and informal enquiries are to be emailed to cedric(dot)fevotte(at)irit(dot)fr




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, June 13, 2017

Paris Machine Learning meetup "Hors Série" #13: Automatic Machine Learning



So tonight is going to be the 13th Hors Série of Season 4 of the Paris Machine Learning meetup. This will bring about to a record 23 the number of meetups Franck and I  have made happen this season!

Tonight will be a presentation of two toolboxes around the issue of Automated Machine Learning. One toolbox is by Axel de Romblay and the other is by Alexis Bondu.

There is limited seating however we will have a streaming (see below).

Thank to Sewan for hosting us. Here is the video for the streaming:

Program:
  • 0 - Preview of the slides
    • A preview of the slides is now available:
    • - French version (link)
    • - English version (link)
  • 1  - Introduction
    • How to ?
    • What's important ?
  • 2 - Theories and Baseline to automate Machine Learning
    • Overview of the different approaches to automate Machine Learning (bayesian, ...)
  • 3 - Demo & Coding
    • Edge ML &  MLBox
    • Comparison of these tools with the same dataset
  • 3-1 - MLBox (Machine Learning Box)
    • MLBox is a powerful Automated Machine Learning python library. It provides the following functionalities:
      • - Fast reading and distributed data preprocessing / cleaning / formatting
      • - Highly robust feature selection and leak detection
      • - Accurate hyper-parameter optimization in high-dimensional space
      • - State-of-the art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,...)
    • - Prediction with models interpretation
    • To learn more about this tool and how to get it installed, please refer to:
    • https://github.com/AxeldeRomblay/MLBox
  • 3.2 - Edge ML
    • Edge ML is an Automated Machine Learning software which implements the MODL approach, which is unique in two respects :
      • i) it is highly scalable and avoids empirical optimization of the models by grid-search;
      • ii) it provides accurate and very robust models.
    • To participate in the interactive demonstration, you can install Edge ML :
      • 1 - Install the shareware version: link
      • 2 - Create and download your licence file: link
      • 3 - Copy / Paste your licence file to your home directory
    • 4 - Download the Python wrapper (with demo example): link


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Printfriendly