A Short Review of 2014 on Nuit Blanche: On Compressive Sensing, Machine Learning, Advanced Matrix Factorization and many beautiful things.

2014 was an exceptional year! One that seemed focused on exploration.

First and foremost, in order to get a real sense of what was accomplished you probably need to read the last 12 Nuit Blanche Monthly Reviews.

This year saw the Ten Year Anniversary of compressive sensing.

The most important event of the year, is in my view the release of two genome sequencers that enable nearly any polynomial time algorithms to perform sequence alignment (Sunday Morning Insight: Crossing into P territory). It is quite simply a momentous event. In terms of data and algorithm most think of it as a faraway mountain range that will eventually need to be explored and studied. It's the opposite, think of it as the wave on Miller. Which is why one of the most fascinating video of the year was that of Craig Venter entitled Life at the Speed of Light. More on that later.

In terms of physical exploration, Philae landed on a comet, but one of the image I saw on my twitter feed this year provided me with the equivalent of what Nature does: Bacteriophage

 

The human made robot bumped three times to eventually land on the comet (we also lost it) but in Nature, do Bacteriophages have similar issues ? I'd love to know more about what we know about the  bacteriophage navigation algorithm!

Anyway, several trends could be witessed from a few of the 536 entries of 2014. Let me note the arrival of a few Machine Learning subjects as there seems to be a convergence between some supervised learning and compressive sensing (and my involvement as a co-organizer of one of the largest Machine Learning meetup).Without further ado, here is my biased take, we witnessed:

• A narrowing gap between reconstruction solvers in compressive sensing and the current deep (or not) architectures used in Machine learning to perform classification. the possibility of one-bit compressive sensing to provide some way of understanding neural networks or how real Neurons could be thought as Signal Processing Devices.
• An increase in Matrix and Tensor Factorizations of all kinds with their attendant phase diagrams (MF, Tensor , RandNLA tag on Nuit Blanche)
• We noticed a larger number of mapmakers, people who, while developing algorithms, also realized that these have bounds / limits could be used as a positive tool. But first and foremost, we saw that empirically some of these bounds could allow us to explore nonlinear reconstructions solvers such as those found in Machine Learning ( Some thoughts on invertibility: Signal recovery from Pooling Representations, Determination of Nonlinear Genetic Architecture using Compressed Sensing ). Sharp phase transition such as Donoho-Tanner was also seen as a way to probe good from bad neural networks in Compressive Sensing and Short Term Memory / Visual Nonclassical Receptive Field Effects or provide Sharp Performance Bounds for Graph Clustering . See also the more in-depth: Sunday Morning Insight: It's Complicated ...On Games, NP-hardness, Grothendieck, Learning and Friends. It is quite interesting to see these moments where pure mathematics has a real an immediate impact on actual engineering feats (Another Donoho-Tao Moment ? ). 
• We saw the growth of Multiple Regularizers (and recent results presented by Maryam Fazel)
• Randomization as a key for designing new algorithms.
• The actual rise of Convolutional Neural Networks as per their winning entries in Kaggle challenges. This indicates an increased robustness of these algorithms (a higher technology readiness level). But there is a high level of activity that aims at figuring out if those a really necessary (See the regularization architecture) or how they can be robustified through adverserial noise.
• Kaggle Challenges point to Random Forests as a first technique to use ( see also Why Kaggle Changes Everything yet don't really allow exploration)
• Vowpal wabbit (as seen in the recent score on the Tradeshift Kaggle challenge)
• Julia as a programming language used in our areas.
• A growing interest in extracting causality from correlation measurements.
• The Paris Machine Learning Meetup has had over 20 meetups already and can now boast to be the largest Machine Learning meetup outside the US (Top 6 overall). I have tried to scheduled a few research subjects that have a particular bearing on some of the theme mentioned here on Nuit Blanche. In at least one case, I tried to clinch the magic of at least one of these meetups on Unfiltered Access. Another meetup stands in my mind, it is the one we organized in June (Europe Wide Machine Learning Meetup and Paris Machine Learning #12: Season 1 Finale, Andrew Ng and More... ). The organization alone was interesting to say the least. But what's not to like when you are being made aware of the intricacies of decyphering Dolphin communications. All the slides and some videos of these meetups can be found here. I'll try to do a summary of what we earned from all these meetups next year.
• The mention and comment of the panel video entitled Is Deep Learning the Final Frontier and the End of Signal Processing ? produced a strong reaction from one of the leaders in Deep Learning. His response can be found in Yoshua Bengio's view on Deep Learning. 
• When asked about how Compressive Sensing helped in scientific discovery, I had a few ideas but here is another one: avoid a 200 Million Dollar Assumption
• The rise of data driven sensors ( Sunday Morning Insight: Zero Knowledge Sensor Design / Data Driven Sensor Design , Sunday Morning Insight: Physics Driven Sensor Design ? , Quick Panorama of Sensing from Direct Imaging to Machine Learning, Blind Deconvolution tag) as well as additional compressive sensing related hardware (including hyperspectral sensors)
• The potential for new fields to be explored (Sunday Morning Insight: Note By Note Cooking)

Personally, I also got back into publishing mode. Our paper also resulted in a few blog entries:

• Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium
• The Post Publication Peer Review of "Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium"
• Additional Photos and Video to "Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium"
• Follow up on "Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium"

I also did an open review (... There will be a "before" and "after" this paper ...) and did a poster presentation at JIONC on the connection between Direct Imaging to Machine Learning. I also provided some tips on Organizing Meetups and got Nuit Blanche featured on La Recherche of February.

Some additional stats can be found in:

• The Long Distance Blogger
• Nuit Blanche in Numbers
• Three and a half million page views: a million here, a million there and soon enough we're talking real readership...
• 1250 posts on the CompressiveSensing Subreddit.
• CitingNuitBlanche tag.

 Credits:
- Courtesy of NASA/SDO and the AIA, EVE, and HMI science teams.
- "PhageExterior" by Adenosine - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:PhageExterior.svg#mediaviewer/File:PhageExterior.svg
-  ESA / Wikipedia
  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Nuit Blanche in Review ( December 2014 )

Since the last Nuit Blanche in Review ( November 2014 )
we had much machine learning related entries as we are seeing a decreasing gap between what we currently see as reconstruction solvers in compressive sensing and the current deep (or not) architectures used in Machine learning to perform classification. Without further ado :

We had two implementations this month:

• Learning with Fredholm Kernels - implementation
• Learning Multidimensional Fourier Series With Tensor Trains - implementation - 

 Three "insights"

• Sunday Morning Insight: The Stuff of Discovery
• Sunday Morning Insight: The Regularization Architecture
• Sunday Morning Insight: An exact mapping between the Variational Renormalization Group and Deep Learning

a few books and theses:

• Book: Sketching as a Tool for Numerical Linear Algebra
• Thesis : Convex Optimization Algorithms and Recovery Theories for Sparse Models in Machine Learning, Bo Huang
• Thesis: Quantum Algorithms for Linear Algebra and Machine Learning; Anupam Prakash
• Book: The Discrepancy Method by Bernard Chazelle

Some more in-depth coverage:

Compressive Sensing

• Single-shot compressed ultrafast photography at one hundred billion frames per second
• How little data is enough? Phase-diagram analysis of sparsity-regularized X-ray CT
• Compressive Hyperspectral Imaging with Side Information
• More Oil and Compressive Sensing Connections
• Another Donoho-Tao Moment ?
• Hamming's time: Scientific Discovery Enabled by Compressive Sensing and related fields Does Compressed Sensing have applications in Robust Statistics?
• Detecting defects in solar cells using compressive sensing 

 Machine Learning

• Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems
• Streaming Anomaly Detection Using Online Matrix Sketching
• Unsupervised Learning of Spatiotemporally Coherent Metrics
• Tag-Aware Ordinal Sparse Factor Analysis for Learning and Content Analytics
• A la Carte - Learning Fast Kernels
• On the Stability of Deep Networks
• NIPS2014 Poster papers   
• Deep Fried Convnets
• Non-parametric PSF estimation from celestial transit solar images using blind deconvolution  
• Distinguishing Cause from Effect using Observational Data: Methods and Benchmarks
• Cauchy Principal Component Analysis  
• Cone-constrained Principal Component Analysis
  Sparse Polynomial Learning and Graph Sketching
  A statistical model for tensor PCA 
• L_1 regularization in Machine Learning: Memory Bounded Deep Convolutional Networks / Sparse Random Features Algorithm as Coordinate Descent in Hilbert Space

Paris Machine Learning Meetup

• Paris Machine Learning Meetup #4 Season 2: Tips and advices for machine learning challenges, Biochemical Probabilistic Computation, Bias-variance decomposition in Random Forests 

 Video and Slides / Slides

• Video : Statistical and Causal Approaches to Machine Learning, Bernhard Schölkopf.
• Video: Exact Recovery via Convex Relaxations 
• Saturday Morning Videos : Spectral Algorithms: From Theory to Practice (Simons Institute @ Berkeley)
• Saturday Morning Videos : Semidefinite Optimization, Approximation and Applications (Simons Institute @ Berkeley)
• Video and Slides: Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints Video: A Statistical Model for Tensor Principal Component Analysis
• Video and Slides: Random Embeddings, Matrix-valued Kernels and Deep Learning 
• Slides: Curse of Dimensionality with Convex Neural Networks by Francis Bach 

Jobs:

• CSjobs: Marie Curie Early Stage Researchers in Sparse Representations and Compressed Sensing
• Job: Research positions (PostDocs and PhD studentship) at MSSL UCL
• CSjob: Postdoc in Mathematics
• Job: One year postdoc in the area of Applied Nonlinear Fourier Analysis, TU Delft
• CSjobs: Multiple positions @ BASP Edinburgh  

Other videos:

• Saturday Morning Video: Jeff Iliff: One more reason to get a good night’s sleep
• Video: Alexander Gerst’s Earth timelapses

A few new statistics

• Nuit Blanche in Numbers
• Three and a half million page views: a million here, a million there and soon enough we're talking real readership...

 Credit: Courtesy of NASA/SDO and the AIA, EVE, and HMI science teams.

Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Nuit Blanche in Numbers

 

 

Total number of unique visits on Nuit Blanche per year since 2008. 

Other stats can be found here.

Affiliated communities

• Google+ Community (1320 members), 
• the CompressiveSensing subreddit (639 members), 
• the LinkedIn Compressive Sensing group (3150 members) or 
• the Advanced Matrix Factorization Group (995 members)

Reference pages include

• The Big Picture in Compressive Sensing (and Learning Compressive Sensing),
• the Advanced Matrix Factorization Jungle Page and 
• the Reproducible Research page
• Paris Machine Learning Meetup Archives (1640 members), 
• Highly Technical Reference Pages - Aggregators,
• These Technologies Do Not Exist,
• CAI: Cable And Igor's Adventures in Matrix Factorization

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Slides: Curse of Dimensionality with Convex Neural Networks by Francis Bach

I just stumbled on Breaking the Curse of Dimensionality with Convex Neural Networks by Francis Bach

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Book: The Discrepancy Method by Bernard Chazelle

In a recent video presentation when Vikas Sindhwani talked about Random Embeddings, Matrix-valued Kernels and Deep Learning, he mentioned that instead of using purely random features we ought to be looking at Quasi-Monte Carlo series of numbers [1]. Sebastien Bubeck just hosted a blog post pn the subject entitled Beating Monte Carlo a guest post by Sasho Nikolov which linked to the pdf version of this book by Bernard Chazelle on the Discrepancy Method.


[1] Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels Jiyan Yang, Vikas Sindhwani, Haim Avron, Michael Mahoney

Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday Morning Video: Jeff Iliff: One more reason to get a good night’s sleep

     

Of interest: Brain-wide pathway for waste clearance captured by contrast-enhanced MRI, Jeffrey J Iliff, Hedok Lee, Mei Yu, Tian Feng, Jean Logan, Maiken Nedergaard, Helene Benveniste

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Video: Alexander Gerst’s Earth timelapses

  of specific interest is the video starting at about 4 minutes and 38 seconds, with the overflight of France in about 8 seconds but real time video would really last about 2 minutes and 15 seconds.       Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Thesis: Quantum Algorithms for Linear Algebra and Machine Learning; Anupam Prakash

Quantum Algorithms for Linear Algebra and Machine Learning by Anupam Prakash

Most quantum algorithms o ering speedups over classical algorithms are based on the three techniques of phase estimation, amplitude estimation and Hamiltonian simulation. In spite of the linear algebraic nature of the postulates of quantum mechanics, until recent work by Lloyd and coauthors (23; 22; 24) no quantum algorithms achieving speedups for linear algebra or machine learning had been proposed. A quantum machine learning algorithm must address three issues: encoding of classical data into a succinct quantum representation, processing the quantum representation and extraction of classically useful information from the processed quantum state. In this dissertation, we make progress on all three aspects of the quantum machine learning problem and obtain quantum algorithms for low rank approximation and regularized least squares.

The oracle QRAM, the standard model studied in quantum query complexity, requires time O(pn) to encode vectors v2Rn into quantum states. We propose simple hardware augmentations to the oracle QRAM, that enable vectors v2Rn to be encoded in time O(logn), with pre-processing. The augmented QRAM incurs minimal hardware overheads, the pre-processing can be parallelized and is a exible model that allows storage of multiple vectors and matrices. It provides a framework for designing quantum algorithms for linear algebra and machine learning.

Using the augmented QRAM for vector state preparation, we present two di erent algorithms for singular value estimation where given singular vector jvi forA2 Rm n, the singular value i is estimated within additive error kAkF. The rst algorithm requires time eO(1= 3 ) and uses the approach for simulating e CX and CUR decompositions. Classical algorithms for these problems require time O(mnlog n+poly(1= )), the quantum algorithms have running time O(p mpoly(1 = ;k; )) where k; are the rank and spectral gap. The running time of the quantum CX decomposition algorithm does not depend on m, it is polynomial in problem parameters. We also provide quantum algorithms for`2 regularized regression problems, the quantum ridge regression algorithm requires time eO(1= 2 )to output a quantum state that is close to the solution, where is the regularization parameter.

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Unsupervised Learning of Spatiotemporally Coherent Metrics

 

 From the paper:

 

Non-linear operators consisting of a redundant linear transformation followed by a point-wise non-linearity and a local pooling, are fundamental building blocks in deep convolutional networks. This is due to their capacity to generate local invariance while preserving discriminative information Le-Cun et al. (1998); Bruna & Mallat (2013).

 

 

Unsupervised Learning of Spatiotemporally Coherent Metrics by Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, Yann LeCun

Current state-of-the-art object detection and recognition algorithms rely on supervised training, and most benchmark datasets contain only static images. In this work we study feature learning in the context of temporally coherent video data. We focus on training convolutional features on unlabeled video data, using only the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity. We show that nearest neighbors of a query frame in the learned feature space are more likely to be its temporal neighbors.  

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tag-Aware Ordinal Sparse Factor Analysis for Learning and Content Analytics

So it looks like Andrew Lan et al improved on SPARFA, a matrix factorization technique to help in education analytics. We had Andew talk at the Paris Machine Learning meetup #11 season 1.

Tag-Aware Ordinal Sparse Factor Analysis for Learning and Content Analytics by Andrew S. Lan, Christoph Studer, Andrew E. Waters, Richard G. Baraniuk

Machine learning offers novel ways and means to design personalized learning systems wherein each student's educational experience is customized in real time depending on their background, learning goals, and performance to date. SPARse Factor Analysis (SPARFA) is a novel framework for machine learning-based learning analytics, which estimates a learner's knowledge of the concepts underlying a domain, and content analytics, which estimates the relationships among a collection of questions and those concepts. SPARFA jointly learns the associations among the questions and the concepts, learner concept knowledge profiles, and the underlying question difficulties, solely based on the correct/incorrect graded responses of a population of learners to a collection of questions. In this paper, we extend the SPARFA framework significantly to enable: (i) the analysis of graded responses on an ordinal scale (partial credit) rather than a binary scale (correct/incorrect); (ii) the exploitation of tags/labels for questions that partially describe the question{concept associations. The resulting Ordinal SPARFA-Tag framework greatly enhances the interpretability of the estimated concepts. We demonstrate using real educational data that Ordinal SPARFA-Tag outperforms both SPARFA and existing collaborative filtering techniques in predicting missing learner responses.  

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

A la Carte - Learning Fast Kernels

Here is a specific tweak to FastFood in order to enable a large set of kernels to better fit group theoretic transformation. It is a little bit a similar path taken in the ScatNet approach.

A la Carte - Learning Fast Kernels by Zichao Yang, Alexander J. Smola, Le Song, Andrew Gordon Wilson

Kernel methods have great promise for learning rich statistical representations of large modern datasets. However, compared to neural networks, kernel methods have been perceived as lacking in scalability and flexibility. We introduce a family of fast, flexible, lightly parametrized and general purpose kernel learning methods, derived from Fastfood basis function expansions. We provide mechanisms to learn the properties of groups of spectral frequencies in these expansions, which require only O(mlogd) time and O(m) memory, for m basis functions and d input dimensions. We show that the proposed methods can learn a wide class of kernels, outperforming the alternatives in accuracy, speed, and memory consumption.  

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

On the Stability of Deep Networks

Wow! after this, this and this, we are now converging:

On the Stability of Deep Networks by Raja Giryes, Guillermo Sapiro, Alex M. Bronstein

In this work we study the properties of deep neural networks with random weights. We formally prove that these networks perform a distance-preserving embedding of the data. Based on this we then draw conclusions on the size of the training data and the networks' structure.

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

CSjobs: Marie Curie Early Stage Researchers in Sparse Representations and Compressed Sensing

Mark just sent me the following:

 

Dear Igor,

We thought that readers of Nuit Blanche might be interested in these "Early Stage Researcher" (ESR) positions for PhD study, as part of our new "SpaRTaN" EU-funded Marie Curie Initial Training Network in Sparse Representations and Compressed Sensing. These positions come with a very competitive salary compared to regular PhD scholarships, so might also be of interest to young researchers who had not otherwise thought of taking a PhD.
Merry Christmas! Mark Plumbley
----

Marie Curie Early Stage Researchers in Sparse Representations and Compressed Sensing

EU FP7 Marie Curie Initial Training Network "SpaRTaN: Sparse Representations and Compressed Sensing Training Network" (FP7-PEOPLE-2013-ITN 607290)

Project Website: http://spartan-itn.eu/

Applications are invited to a number of Marie Curie Early Stage Researcher (ESR) positions as part of the new EU-funded Marie Curie Initial Training Network (ITN) "SpaRTaN: Sparse Representations and Compressed Sensing Training Network".

The SpaRTaN ITN (http://spartan-itn.eu/) brings together leading academic and industry groups to train a new generation of interdisciplinary researchers in sparse representations and compressed sensing, with applications in areas such as hyperspectral imaging, audio signal processing and video analytics.

Early Stage Researcher (ESR) positions allow the researcher to work towards a PhD, for a duration of 36 months. ESRs should be within four years of the diploma granting them access to doctorate studies at the time of recruitment, and must not have spent more than 12 months in the host country in the 3 years prior to starting. Marie Curie ESRs are paid a competitive salary which is adjusted for their host country.

Each of the ESR posts being recruited across SpaRTaN has its own application process and closing date in early 2015. The full list of Early Stage Researcher (ESR) Positions (recruiting early 2015) is as follows:

* ESR1 : Sparse Time-Frequency methods for Audio Source Separation - CVSSP, University of Surrey, United Kingdom
* ESR2 : Automatic Music Transcription using Structured Sparse Dictionary Learning - CVSSP, University of Surrey, United Kingdom
* ESR3 : Sparse Representations and Compressed Sensing - University of Edinburgh, United Kingdom
* ESR4 : Task Based Dictionary Learning for Audio-Visual Tagging - LTS2, EPFL,Switzerland
* ESR5 : 1-bit Compressive Imaging - LTS2, EPFL,Switzerland
* ESR6 : Analysis Dictionary Learning Beyond Gaussian Denoising - Instituto de Telecomunicações, Portugal
* ESR7 : Compressed Sensing for Hyperspectral Imaging - Instituto de Telecomunicações, Portugal
* ESR8 : Large-scale signal processing - INRIA, France

For further details, see http://spartan-itn.eu/#1

The following experienced Researcher (ER) positions will also be recruiting later in 2015:

* ER1 : Video Analytics for Large Camera Networks - VisioSafe, Switzerland
* ER2 : Image and Video Restoration with Adaptive Transforms - Noiseless Imaging, Finland

For more details of all ESR positions and future ER positions, and information on how to apply, see http://spartan-itn.eu/#1

--
Prof Mark D Plumbley
Professor of Signal Processing
Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey
Guildford, Surrey, GU2 7XH, UK
Email: m.plumbley@surrey.ac.uk 

  Thanks Mark and Merry Festivus for the rest of us.
  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

How little data is enough? Phase-diagram analysis of sparsity-regularized X-ray CT

Jakob, a map maker, just sent me the following:

Dear Igor,


Thank you for keeping the Nuit Blanche an important CS resource that I have been following for several years now.


I thought you and some readers might be interested in an update on our work on phase transitions in computed tomography that you recently posted about:

http://nuit-blanche.blogspot.dk/2014/09/testable-uniqueness-conditions-for.html

That paper has been published online (with open access) by Inverse Problems in Science and Engineering:

http://dx.doi.org/10.1080/17415977.2014.986724

Furthermore, we have just put on arXiv our latest results:
How little data is enough? Phase-diagram analysis of sparsity-regularized X-ray CT by Jakob S. Jørgensen and Emil Y. Sidky

http://arxiv.org/abs/1412.6833

where we compare the empirically observed phase transitions in CT with the theoretical results for Gaussian matrices i.e. the Donoho-Tanner phase transition as well as the results by Amelunxen, Lotz, McCoy and Tropp:

http://arxiv.org/abs/1303.6672

Surprisingly, at least to us, we find essentially the same phase-transition behavior in CT as in Gaussian case!


Further we study the use of randomized sampling strategies in CT and give preliminary results on how well the phase transitions can predict the sufficient number of samples to acquire in a large-scale CT setting depending on the sparsity.


Happy holidays and best wishes,
Jakob

--
Jakob Sauer Jørgensen

Postdoc Scientific Computing

------------------------------------------------------

Technical University of Denmark Department of Applied Mathematics and Computer Science Matematiktorvet Building 303 B
2800 Kgs. Lyngby

Thanks Jakob ! Here is the paper: How little data is enough? Phase-diagram analysis of sparsity-regularized X-ray CT by Jakob S. Jørgensen, Emil Y. Sidky

We introduce phase-diagram analysis, a standard tool in compressed sensing, to the X-ray CT community as a systematic method for determining how few projections suffice for accurate sparsity-regularized reconstruction. In compressed sensing a phase diagram is a convenient way to study and express certain theoretical relations between sparsity and sufficient sampling. We adapt phase-diagram analysis for empirical use in X-ray CT for which the same theoretical results do not hold. We demonstrate in three case studies the potential of phase-diagram analysis for providing quantitative answers to questions of undersampling: First we demonstrate that there are cases where X-ray CT empirically performs comparable with an optimal compressed sensing strategy, namely taking measurements with Gaussian sensing matrices. Second, we show that, in contrast to what might have been anticipated, taking randomized CT measurements does not lead to improved performance compared to standard structured sampling patterns. Finally, we show preliminary results of how well phase-diagram analysis can predict the sufficient number of projections for accurately reconstructing a large-scale image of a given sparsity by means of total-variation regularization.  

 

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Deep Fried Convnets

Nando de Freitas just commented on the Sunday Morning Insight on The Regularization Architecture

Very interesting. Thanks for this summary. I loved the paper of Jimmy Ba and Rich Caruana. Here's one of our recent takes on the issue of kernels, randomization and new forms of deep layerd: http://arxiv.org/abs/1412.7149

In all these, I think it's important to analyse performance on large datasets such as imagenet.

Thanks Nando !  another piece of the puzzle for the construction of reconstruction solvers:

Deep Fried Convnets by Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, Ziyu Wang

The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important for operating deep neural networks in memory constrained environments such as GPUs or embedded devices.
In this paper we show how kernel methods, in particular a single Fastfood layer, can be used to replace all fully connected layers in a deep convolutional neural network. This novel Fastfood layer is also end-to-end trainable in conjunction with convolutional layers, allowing us to combine them into a new architecture, named deep fried convolutional networks, which substantially reduces the memory footprint of convolutional networks trained on MNIST and ImageNet with no drop in predictive performance.

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Non-parametric PSF estimation from celestial transit solar images using blind deconvolution

Using blind deconvolution and a Venus transit to figure out the calibration of working camera onboard a current spacecraft. This is the feat of this paper: 

Non-parametric PSF estimation from celestial transit solar images using blind deconvolution by Adriana Gonzalez, Véronique Delouille, Laurent Jacques

Context: Characterization of instrumental effects in astronomical imaging is important in order to extract accurate physical information from the observations. Optics are never perfect and the non-ideal path through the telescope is usually represented by the convolution of an ideal image with a Point Spread Function (PSF). Other sources of noise (read-out, Photon) also contaminate the image acquisition process. The problem of estimating both the PSF filter and a denoised image is called blind deconvolution and is ill-posed.
Aims: We propose a blind deconvolution scheme that relies on image regularization. Contrarily to most methods presented in the literature, it does not assume a parametric model of the PSF and can thus be applied to any telescope.
Methods: Our scheme uses a wavelet analysis image prior model and weak assumptions on the PSF filter's response. We use the observations from a celestial body transit where such object can be assumed to be a black disk. Such constraints limits the interchangeability between the filter and the image in the blind deconvolution problem.
Results: Our method is applied on synthetic and experimental data. We compute the PSF for SECCHI/EUVI instrument using the 2007 Lunar transit, and for SDO/AIA with the 2012 Venus transit. Results show that the proposed non-parametric blind deconvolution method is able to estimate the core of the PSF with a similar quality than parametric methods proposed in the literature. We also show that, if these parametric estimations are incorporated in the acquisition model, the resulting PSF outperforms both the parametric and non-parametric methods.

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Video : Statistical and Causal Approaches to Machine Learning, Bernhard Schölkopf.

Found the following recently released video of Bernhard Schölkopf (see previous Distinguishing Cause from Effect using Observational Data: Methods and Benchmarks)

h/t Alung     Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Distinguishing Cause from Effect using Observational Data: Methods and Benchmarks

More work paralleling that of David Lopez-Paz et al's s Non-linear Causal Inference using Gaussianity Measures - implementation - ( We featured David at Paris Machine Learning #2 Season 2 ). A simple note when one looks at this from the signal processing standpoint. There, we usually set problems as 

y = A x + epsilon

where epsilon is some gaussian noise. In these papers on causality, that specific shape for the noise would tend to imply that y is not the result of x as "residuals in the anti-causal direction are more Gaussian than the residuals in the actual causal direction". Anyway, today we have the following (including a cause and effects pairs database ).

Distinguishing cause from effect using observational data: methods and benchmarks by Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, Bernhard Schölkopf

The discovery of causal relationships from purely observational data is a fundamental problem in science. The most elementary form of such a causal discovery problem is to decide whether X causes Y or, alternatively, Y causes X, given joint observations of two variables X, Y . This was often considered to be impossible. Nevertheless, several approaches for addressing this bivariate causal discovery problem were proposed recently. In this paper, we present the benchmark data set CauseEffectPairs that consists of 88 different "cause-effect pairs" selected from 31 datasets from various domains. We evaluated the performance of several bivariate causal discovery methods on these real-world benchmark data and on artificially simulated data. Our empirical results provide evidence that additive-noise methods are indeed able to distinguish cause from effect using only purely observational data. In addition, we prove consistency of the additive-noise method proposed by Hoyer et al. (2009).

Of additional interest is a Database with cause-effect pairs 

Related:

Causal Discovery with Continuous Additive Noise Models by Jonas Peters, Joris Mooij, Dominik Janzing, Bernhard Schölkopf

We consider the problem of learning causal directed acyclic graphs from an observational joint distribution. One can use these graphs to predict the outcome of interventional experiments, from which data are often not available. We show that if the observational distribution follows a structural equation model with an additive noise structure, the directed acyclic graph becomes identifiable from the distribution under mild conditions. This constitutes an interesting alternative to traditional methods that assume faithfulness and identify only the Markov equivalence class of the graph, thus leaving some edges undirected. We provide practical algorithms for finitely many samples, RESIT (Regression with Subsequent Independence Test) and two methods based on an independence score. We prove that RESIT is correct in the population setting and provide an empirical evaluation.  

 Implementations are available here.


Learning Sparse Causal Models is not NP-hard by Tom Claassen, Joris M. Mooij, Tom Heskes

This paper shows that causal model discovery is not an NP-hard problem, in the sense that for sparse graphs bounded by node degree k the sound and complete causal model can be obtained in worst case order N2(k+2) independence tests, even when latent variables and selection bias may be present. We present a modi cation of the well-known FCI algorithm that implements the method for an independence oracle, and suggest improvements for sample/real-world data versions. It does not contradict any known hardness results, and does not solve an NP-hard problem: it just proves that sparse causal discovery is perhaps more complicated, but not as hard as learning minimal Bayesian networks

Proof Supplement - Learning Sparse Causal Models is not NP-hard (UAI2013) by Tom Claassen, Joris M. Mooij, Tom Heskes

This article contains detailed proofs and additional examples related to the UAI-2013 submission `Learning Sparse Causal Models is not NP-hard'. It describes the FCI+ algorithm: a method for sound and complete causal model discovery in the presence of latent confounders and/or selection bias, that has worst case polynomial complexity of order N2(k+1) in the number of independence tests, for sparse graphs over N nodes, bounded by node degree k. The algorithm is an adaptation of the well-known FCI algorithm by (Spirtes et al., 2000) that is also sound and complete, but has worst case complexity exponential in N.

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.