Saturday Morning Video: "Can the brain do back-propagation?" - Geoffrey Hinton of Google & University of Toronto

  Geoffrey Hinton wonders if the brain can do backpropagation:

 h/t Andrej

Neat talk by Geoff Hinton https://t.co/e0xUq6eXP7 "STDP is the signature of backprop using temporal derivatives as error signal". hm.

— Andrej Karpathy (@karpathy) 29 avril 2016

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday Morning Videos: Mars, Falcon landing and "A Beautiful Planet"

An animation tracking Curiosity’s path on Mars, along Bridger Basin while looking at the Bob Marshall Ridge. Composed in Blender from 41 Navcam greyscale/xyz image pairs taken by the rover at sols 1094 to 1108. Conversion done with the Navcam importer which is available at: github.com/phaseIV/Blender-Navcam-Importer Following Curiosity from Captain Video on Vimeo.
360 View, First Stage Landing on Droneship for Falcon 9    Window to Earth: NASA Partners with IMAX for 'A Beautiful Planet'    Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex / Memory and Information Processing in Recurrent Neural Networks

Here are two interesting papers on Recurrent Neural Networks. Interestingly, here is an excerpt from the second preprint:

However, here we see that even without the sparse input assumption and L1 optimization in the output (a computationally expensive optimization used in compressive sensing) the network can achieve capacity greater than its degrees of freedom N. 

 Enjoy !

Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex by Qianli Liao, Tomaso Poggio

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 dataset.

Memory and Information Processing in Recurrent Neural Networks by Alireza Goudarzi, Sarah Marzen, Peter Banda, Guy Feldman, Christof Teuscher, Darko Stefanovic

Recurrent neural networks (RNN) are simple dynamical systems whose computational power has been attributed to their short-term memory. Short-term memory of RNNs has been previously studied analytically only for the case of orthogonal networks, and only under annealed approximation, and uncorrelated input. Here for the first time, we present an exact solution to the memory capacity and the task-solving performance as a function of the structure of a given network instance, enabling direct determination of the function--structure relation in RNNs. We calculate the memory capacity for arbitrary networks with exponentially correlated input and further related it to the performance of the system on signal processing tasks in a supervised learning setup. We compute the expected error and the worst-case error bound as a function of the spectra of the network and the correlation structure of its inputs and outputs. Our results give an explanation for learning and generalization of task solving using short-term memory, which is crucial for building alternative computer architectures using physical phenomena based on the short-term memory principle.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Video: Sparse Identification of Nonlinear Dynamics (SINDy)

We mentioned the approach earlier in Discovering governing equations from data: Sparse identification of nonlinear dynamical systems - implementation - but Steve sent me the following as an add-on:

 Hi Igor,

I am attaching a link to a youtube video abstract of our recent paper on sparse identification of nonlinear dynamics (SINDy) in PNAS.  Hope you enjoy, and please feel free to share with anyone who may be interested.

Also, I saw that you mentioned our algorithm on your blog — thanks very much!!  It is awesome to hear the others like the work.

   Video abstract:  https://www.youtube.com/watch?v=gSCa78TIldg
   Paper [open access]:  http://www.pnas.org/content/113/15/3932.abstract

Best Regards,
Steve

Thanks  Steve ! Here is the video:

 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Agnostic Estimation of Mean and Covariance

 

 

The reason we have a zoo of matrix factorizations stems in part with the need to deal with different adversarial noises. From the paper:

The Achilles heel of algorithms for generative models is the assumption that data is exactly from the model. This is crucial for known guarantees, and relaxations of it are few and specialized, e.g., in ICA, data could by noisy, but the noise itself is assumed to be Gaussian. Assumptions about rank and sparsity are made in a technique that is now called Robust PCA [CSPW11, CLMW11]. There have been attempts [Kwa08, MT+11] at achieving robustness by L1 minimization, but they don’t give any error bounds on the output produced. A natural, important and wide open problem is estimating the parameters of generative models in the presence of arbitrary, i.e., malicious noise, a setting usually referred to as agnostic learning. The simplest version of this problem is to estimate a single Gaussian in the presence of malicious noise. Alternatively, this can be posed as the problem of finding a best-fit Gaussian to data or agnostically learning a single Gaussian.

Agnostic Estimation of Mean and Covariance  by Kevin A. Lai, Anup B. Rao, Santosh Vempala

We consider the problem of estimating the mean and covariance of a distribution from iid samples in $\mathbb{R}^n$, in the presence of an $\eta$ fraction of malicious noise; this is in contrast to much recent work where the noise itself is assumed to be from a distribution of known type. The agnostic problem includes many interesting special cases, e.g., learning the parameters of a single Gaussian (or finding the best-fit Gaussian) when $\eta$ fraction of data is adversarially corrupted, agnostically learning a mixture of Gaussians, agnostic ICA, etc. We present polynomial-time algorithms to estimate the mean and covariance with error guarantees in terms of information-theoretic lower bounds. As a corollary, we also obtain an agnostic algorithm for Singular Value Decomposition.

and previously the Recursive Fourier PCA algorithm

Max vs Min: Tensor Decomposition and ICA with nearly Linear Sample Complexity by Santosh S. Vempala, Ying Xiao

We present a simple, general technique for reducing the sample complexity of matrix and tensor decomposition algorithms applied to distributions. We use the technique to give a polynomial-time algorithm for standard ICA with sample complexity nearly linear in the dimension, thereby improving substantially on previous bounds. The analysis is based on properties of random polynomials, namely the spacings of an ensemble of polynomials. Our technique also applies to other applications of tensor decompositions, including spherical Gaussian mixture models.

 

    Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

A Tutorial on Libra: R package for the Linearized Bregman Algorithm in High Dimensional Statistics - implementation -

Here is an implementation in R of an l1 solver. All compressive sensing solvers implementation show up under the CS and implemntation tags.

A Tutorial on Libra: R package for the Linearized Bregman Algorithm in High Dimensional Statistics by Jiechao Xiong, Feng Ruan, Yuan Yao

The R package, Libra, stands for the LInearized BRegman Al- gorithm in high dimensional statistics. The Linearized Bregman Algorithm is a simple iterative procedure to generate sparse regularization paths of model estimation, which are rstly discovered in applied mathematics for image restoration and particularly suitable for parallel implementation in large scale problems. The limit of such an algorithm is a sparsity-restricted gradient descent ow, called the Inverse Scale Space, evolving along a par- simonious path of sparse models from the null model to over tting ones. In sparse linear regression, the dynamics with early stopping regularization can provably meet the unbiased Oracle estimator under nearly the same condition as LASSO, while the latter is biased. Despite their successful applications, statistical consistency theory of such dynamical algorithms remains largely open except for some recent progress on linear regression. In this tutorial, algorithmic implementations in the package are discussed for several widely used sparse models in statistics, including linear regression, logistic regres- sion, and several graphical models (Gaussian, Ising, and Potts). Besides the simulation examples, various application cases are demonstrated, with real world datasets from diabetes, publications of COPSS award winners, as well as social networks of two Chinese classic novels, Journey to the West and Dream of the Red Chamber.

 The packahe is here: https://cran.r-project.org/web/packages/Libra/index.html

  Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

An information theoretic formulation of the Dictionary Learning and Sparse Coding Problems on Statistical Manifolds

  

An information theoretic formulation of the Dictionary Learning and Sparse Coding Problems on Statistical Manifolds by Rudrasis Chakraborty, Monami Banerjee, Victoria Crawford, Baba C. Vemuri

In this work, we propose a novel information theoretic framework for dictionary learning (DL) and sparse coding (SC) on a statistical manifold (the manifold of probability distributions). Unlike the traditional DL and SC framework, our new formulation {\it does not explicitly incorporate any sparsity inducing norm in the cost function but yet yields SCs}. Moreover, we extend this framework to the manifold of symmetric positive definite matrices, $\mathcal{P}_n$. Our algorithm approximates the data points, which are probability distributions, by the weighted Kullback-Leibeler center (KL-center) of the dictionary atoms. The KL-center is the minimizer of the maximum KL-divergence between the unknown center and members of the set whose center is being sought. Further, {\it we proved that this KL-center is a sparse combination of the dictionary atoms}. Since, the data reside on a statistical manifold, the data fidelity term can not be as simple as in the case of the vector-space data. We therefore employ the geodesic distance between the data and a sparse approximation of the data element. This cost function is minimized using an acceleterated gradient descent algorithm. An extensive set of experimental results show the effectiveness of our proposed framework. We present several experiments involving a variety of classification problems in Computer Vision applications. Further, we demonstrate the performance of our algorithm by comparing it to several state-of-the-art methods both in terms of classification accuracy and sparsity.

 
 
 
 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Paris Machine Learning Meetup #12 Season 3: ML Hardware

Mozilla France is hosting us. Mathworks is sponsoring the networking event afterwards. Thank you to them.

Hardware is becoming key in getting computations performed at scale and using specific techniques such as Deep Learning. Today's Paris Machine Learning meetup is focused on the type of hardware and software combination that is currently and may eventually run this Machine Learning/Deep Learning/AI infrastructure.

Streaming video is below:

 

Right now, these are the presentations we should have tonight. Undoubtedly, we should have more meetups on this topic in the future.
Arjun Bansal, Nervana Systems, Nervana and the Future of Computing

"Nervana delivers fast, scalable AI on demand using cloud computing, deep learning and custom hardware. These trends have recently been identified as the basis for the future of computing in a recent issue of The Economist. In this talk, I will provide an overview of Nervana’s technology."

Marc Wolff, Amine El Helou , Mathworks, Un algorithme distribué de forêt aléatoire pour du risque de crédit/ A Random Forest distributed algorithm for credit risk computations.

L'algorithme Random Forest consiste à entraîner, généralement en parallèle, plusieurs arbres de décision. Une des limitations bien connues de cette méthode est la nécessité de charger voire de répliquer l’intégralité du dataset en mémoire. Dans le cas d’importantes volumétries de données, cela peut se révéler problématique. Une alternative possible consiste à développer un algorithme distribué d’arbre décisionnel en s’appuyant sur du parallélisme de type SPMD (Single Program Multiple Data) et sur l’API MPI (Message Passing Interface). Cette approche permet d’exploiter au mieux toute la puissance de traitement (machine multi-cœurs ou cluster de calcul) et d’opérer sur des données distribuées en mémoire. Nous présenterons une implémentation de ce type et son application à une analyse de risque de crédit auprès d'un groupe bancaire (Groupe de Risque Opérationnel du Crédit Agricole). 

Olivier Guillaune,  Any computer for my machine learning?

"When the learning phase begin to be too long, what are the current material solutions to accelerate my calculations and which elements to assemble a reliable calculation server, powerful and at the best cost ?"

Igor Carron, LightOn.io, Approximating kernels at the speed of light

We will talk about how we use optics to perform some specific operation of interest in Machine Learning. The basis of this talk relies on this preprint.

André Reinald, Mozilla, 

En introduction, nous expliquerons que le lien entre machine learning et Mozilla ce sont les données personnelles, auxquelles nous accordons toute notre attention, et que nous aimerions voir traitées avec une certaine éthique. On parlera de trois sujets:

1. retour sur les tentatives dans les "content services",
2. projets dans les objets connectés (limités à la domotique pour le moment),
3. projet peerstorage.org

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Sketching and Neural Networks

  Two subjects we used to not see in the same field. Interesting!

Sketching and Neural Networks by Amit Daniely, Nevena Lazic, Yoram Singer, Kunal Talwar

High-dimensional sparse data present computational and statistical challenges for supervised learning. We propose compact linear sketches for reducing the dimensionality of the input, followed by a single layer neural network. We show that any sparse polynomial function can be computed, on nearly all sparse binary vectors, by a single layer neural network that takes a compact sketch of the vector as input. Consequently, when a set of sparse binary vectors is approximately separable using a sparse polynomial, there exists a single-layer neural network that takes a short sketch as input and correctly classifies nearly all the points. Previous work has proposed using sketches to reduce dimensionality while preserving the hypothesis class. However, the sketch size has an exponential dependence on the degree in the case of polynomial classifiers. In stark contrast, our approach of using improper learning, using a larger hypothesis class allows the sketch size to have a logarithmic dependence on the degree. Even in the linear case, our approach allows us to improve on the pesky $O({1}/{{\gamma}^2})$ dependence of random projections, on the margin $\gamma$. We empirically show that our approach leads to more compact neural networks than related methods such as feature hashing at equal or better performance.

  Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure

Some new Compressive Sensing hardware, woohoo ! 

 

Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure by Jie Zhang, Tao Xiong, Trac Tran, Sang Chin, and Ralph Etienne-Cummings
Abstract

We present a low power all-CMOS implementation of temporal compressive sensing with pixel-wise coded exposure. This image sensor can increase video pixel resolution and frame rate simultaneously while reducing data readout speed. Compared to previous architectures, this system modulates pixel exposure at the individual photo-diode electronically without external optical components. Thus, the system provides reduction in size and power compare to previous optics based implementations. The prototype image sensor (127 × 90 pixels) can reconstruct 100 fps videos from coded images sampled at 5 fps. With 20× reduction in readout speed, our CMOS image sensor only consumes 14μW to provide 100 fps videos.

 
 
 
 
 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Heavy hitters via cluster-preserving clustering

Interesting, clustering the most frequent elementsin a turnstile model:

Heavy hitters via cluster-preserving clustering by Kasper Green Larsen, Jelani Nelson, Huy L. Nguyen, Mikkel Thorup

In turnstile $\ell_p$ $\varepsilon$-heavy hitters, one maintains a high-dimensional $x\in\mathbb{R}^n$ subject to $\texttt{update}(i,\Delta)$ causing $x_i\leftarrow x_i + \Delta$, where $i\in[n]$, $\Delta\in\mathbb{R}$. Upon receiving a query, the goal is to report a small list $L\subset[n]$, $|L| = O(1/\varepsilon^p)$, containing every "heavy hitter" $i\in[n]$ with $|x_i| \ge \varepsilon \|x_{\overline{1/\varepsilon^p}}\|_p$, where $x_{\overline{k}}$ denotes the vector obtained by zeroing out the largest $k$ entries of $x$ in magnitude.
For any $p\in(0,2]$ the CountSketch solves $\ell_p$ heavy hitters using $O(\varepsilon^{-p}\log n)$ words of space with $O(\log n)$ update time, $O(n\log n)$ query time to output $L$, and whose output after any query is correct with high probability (whp) $1 - 1/poly(n)$. Unfortunately the query time is very slow. To remedy this, the work [CM05] proposed for $p=1$ in the strict turnstile model, a whp correct algorithm achieving suboptimal space $O(\varepsilon^{-1}\log^2 n)$, worse update time $O(\log^2 n)$, but much better query time $O(\varepsilon^{-1}poly(\log n))$.
We show this tradeoff between space and update time versus query time is unnecessary. We provide a new algorithm, ExpanderSketch, which in the most general turnstile model achieves optimal $O(\varepsilon^{-p}\log n)$ space, $O(\log n)$ update time, and fast $O(\varepsilon^{-p}poly(\log n))$ query time, and whp correctness. Our main innovation is an efficient reduction from the heavy hitters to a clustering problem in which each heavy hitter is encoded as some form of noisy spectral cluster in a much bigger graph, and the goal is to identify every cluster. Since every heavy hitter must be found, correctness requires that every cluster be found. We then develop a "cluster-preserving clustering" algorithm, partitioning the graph into clusters without destroying any original cluster.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Deep Online Convex Optimization with Gated Games

David sent me the following a while back:

 

 

Dear Igor,


I’ve enjoyed following your blog for a long time, and think the following preprint may be of interest: “Deep Online Convex Optimization with Gated Games,” http://arxiv.org/abs/1604.01952. It analyzes the convergence of gradient-based methods on rectifier neural networks, which are neither smooth nor convex, using ideas from game-theory.


Abstract:
Methods from convex optimization are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since modern convolutional networks (convnets), incorporating rectifier units and max-pooling, are neither smooth nor convex. Standard guarantees therefore do not apply. This paper provides the first convergence rates for gradient descent on rectifier convnets. The proof utilizes the particular structure of rectifier networks which consists in binary active/inactive gates applied on top of an underlying linear network. The approach generalizes to max-pooling, dropout and maxout. In other words, to precisely the neural networks that perform best empirically. The key step is to introduce gated games, an extension of convex games with similar convergence properties that capture the gating function of rectifiers. The main result is that rectifier convnets converge to a critical point at a rate controlled by the gated-regret of the units in the network. Corollaries of the main result include: (i) a game-theoretic description of the representations learned by a neural network; (ii) a logarithmic-regret algorithm for training neural nets; and (iii) a formal setting for analyzing conditional computation in neural nets that can be applied to recently developed models of attention.

cheers,
David 

 

 

Thanks David !
 

Image Credit: NASA/JPL-Caltech/Space Science Institute  

W00097359.jpg was taken on April 14, 2016 and received on Earth April 14, 2016. The camera was pointing toward SATURN, and the image was taken using the MT2 and CL2 filters. This image has not been validated or calibrated.
 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

The 2016 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2016) is now open

 Michael just let me know that registration for MMDS (2016 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2016 ) is now open (Previous MMDSs were covered under the MMDS tag)

Registration for the 2016 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2016) is now available at:

http://mmds-data.org/home/registration

Early registration is available until May 1, 2016.

In addition to four days of talks on algorithmic and statistical aspects of modern large-scale data analysis, MMDS 2016 will have a contributed poster session one evening.  The registration fee is waived for student poster presenters.  You may apply to present a poster at the link above.

Event: MMDS 2016: Workshop on Algorithms for Modern Massive Data Sets
Dates: June 21-24, 2016
Location: UC Berkeley, Berkeley, CA
Website: http://mmds-data.org
Contact: organizers@mmds-data.org

Synopsis: The 2016 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2016) will address algorithmic, mathematical, and statistical challenges in modern statistical data analysis. The goals of MMDS 2016 are to explore novel techniques for modeling and analyzing massive, high-dimensional, and nonlinearly-structured scientific and internet data sets, and to bring together computer scientists, statisticians, mathematicians, and data analysis practitioners to promote cross-fertilization of ideas.

Organizers: Michael Mahoney (UC Berkeley), Alex Shkolnik (Stanford), and Petros Drineas (RPI)

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wooden blocks and rails - David MacKay ( Information, Inference, and Energy A symposium to celebrate the work of Professor Sir David MacKay FRS)

        

A month ago, the Cambridge community held a symposium in honor of David MacKay who left us this past week. They just released the videos:

• Brief Reminiscences I: Neil Lawrence, John Skilling, John Bridle, Graeme Mitchison
• Brief Reminiscences II: Hanna Wallach, Robert MacKay, Chris Williams, Carl Rasmussen

 Here is the full set of videos:

Information, Inference, and Energy

A symposium to celebrate the work of Professor Sir David MacKay FRS

 

Time and Location

14–15 March 2016
Computer Laboratory, University of Cambridge, UK

Links and downloads

• Symposium booklet (PDF, 316 kB)
• Symposium booklet cover (PDF, 2819 kB)
• Symposium welcome screen (PNG, 312 kB, 2560×1440 pixels)
• David MacKay's website

 

Monday 14th

Andy Hopper	Welcome address
Zoubin Ghahramani	Introduction
David Spiegelhalter	Risk in whole numbers
Tom Counsell	Global Calculator
Amin Shokrollahi	Choral codes
Jossy Sayir	Storing data on DNA
Brief reminiscences I
Christian Steinruecken	Inverse calculators
Mark Lynas	Science and Society
Brendan Frey	Deep learning in medicine
Brief reminiscences II
Oliver Stegle	Disentangling cell diversity
Richard Durbin	Analysis of genome data

Tuesday 15th

John Hopfield	Emergent dynamics
Carlos Brody	Decision-making in rats
Andreas Herz	Activity of grid cells
Per Ola Kristensson	Next-generation text entry
Brief reminiscences III
Alan Blackwell	Dasher videos
John Winn	Democratising data
Iain Murray	Slice sampling
Geoff Hinton	Sensory cortex
David MacKay	Surprise lecture

    Godspeed David...
      Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

"The Great Convergence" or How ML/DL is Disrupting Sensor Design

On Wednesday night, we had the Paris Machine Learning meetup. It was, shall we say, unusual and eventful. More on that later. Because of the flu, two of our speakers could not make it and we had to scramble to put more presentations in the schedule in short order. I rapidily put together one presentation that summarized some of my current thoughts about sensor/hardware design. Because the meetup was eventful, I decided that after three hours, we really needed to go eat and be merry and shelved that presentation. I am not sure I will be able to present it anytime soon as the next meetup will be about hardware for Machine Learning and will probably not be a good fit. So here it is:

 Igor Carron, "The Great Convergence" or How ML/DL is disrupting sensor design.

I describe through three examples how sensor design is being disrupted by new Machine Learning Techniques.

References:

• Tomasz Malisiewicz, From feature descriptors to deep learning: 20 years of computer vision
• Jack Clark, Why AI development is going to get even faster ,
• David Beyer, Future of Machine Intelligence,
• The Great Convergence tag on Nuit Blanche,  
• « Making Hyperspectral Imaging Mainstream » 
• Sunday Morning Insight: A Quick Panorama of Sensing from Direct Imaging to Machine Learning
  

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Exploiting Correlations Among Channels in distributed Compressive Sensing With Convolutional Deep Stacking Networks

Here is another instance of The Great Convergence in action. Today MMV reconstruction with a Deep architecture. From the graphs, it looks like the Deep architecture is not gaining much in terms of phase transition although it does better than the greedy algorithm.

Exploiting Correlations Among Channels in distributed Compressive Sensing With Convolutional Deep Stacking Networks by Hamid Palangi, Rabab Ward, Li Deng

This paper addresses the compressive sensing with Multiple Measurement Vectors (MMV) problem where the correlation amongst the different sparse vectors (channels) are used to improve the reconstruction performance. We propose the use of Convolutional Deep Stacking Networks (CDSN), where the correlations amongst the channels are captured by a moving window containing the “residuals” of different sparse vectors. We develop a greedy algorithm that exploits the structure captured by the CDSN to reconstruct the sparse vectors. Using a natural image dataset, we compare the performance of the proposed algorithm with two types of reconstruction algorithms: Simultaneous Orthogonal Matching Pursuit (SOMP) which is a greedy solver and the model-based Bayesian approaches that also exploit correlation among channels. We show experimentally that our proposed method outperforms these popular methods and is almost as fast as the greedy methods. 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Paris Machine Learning #11 S3: Rogue Waves, Dataiku, eLum, Human Resources, DNN on Matlab, Data and Cars, The Great Convergence

The program has changed slightly and will probably change until later today. Due to the flu, the Mars presentation is postponed.

Interested in making a presentation tonight ? go fill this form here, we'll get back to you shortly.

 Le meetup sera à l'Etoile Saint Honoré au 21/25 rue de Balzac (http://www.etoile-prestige.fr/). Merci a Xebia d'organiser cette événement. Pour ce meetup nous devrions avoir (programme qui changera): 

 Laurence Vachon will briefly present Mathworks' Mission On Mars Robot Challenge 2016. Think of it as a Kaggle for robots. In a different area,Themis Sapsis will show how to predict rogue/killer waves using a new technique based on ML. Martin Prillard will talk to us about company cultures and Machine Learning. Amine El Helou will present the new Deep Learning libraries within Matlab. Florent Pignal will finally briefly mention how his start-up wants to make use of data from computers that are already in cars.

Interested in making a presentation tonight ? go fill this form here, we'll get back to you shortly.

Here are the presentations:

• Matthieu Blanc ou Yoann Benoit, Presentation Xebia

• Paul-Henri Hincelin, Dataiku, Putting Data science in production

  What strategies to bridge the gap between development and production

• Martin Prillard, Talentoday, How psychometrics and machine learning can identify corporate cultures and business success factors

Comment la psychométrie et le machine learning peuvent identifier les cultures d'entreprise et les facteurs de réussite professionnels.

• Amine El Helou, Deep Learning in Matlab.

Deep learning is becoming ubiquitous. In this example we are going to train a Convolutional Neural Network from scratch in order to classify the popular dataset CIFAR-10 using MATLAB. 

Igor Carron, "The Great Convergence" or How ML/DL is disrupting sensor design.

I describe through three examples how sensor design is being disrupted by new Machine Learning Techniques.

• Themis Sapsis (MIT) Robust prediction of extreme wave events in realistic seas

The objective of this work is the development of robust computational algorithms for the prediction of the location and time of rare and extreme wave events in realistic seas. Today’s algorithms for the prediction of rare events rely on the direct computation of the wave field by accurately solving the governing wave equations. This is a computationally very demanding task that requires a large number of computational nodes working in parallel for long times, even for a few minutes of prediction. In addition, due to the chaotic nature of the wave dynamics, the results are very sensitive to noise. The latter is inevitable during the measurement/scanning process. These properties make these direct numerical simulations impractical for real-time prediction. We use a radically different approach for the prediction of rare events in water waves. Instead of trying to accurately solve the full equations, we analyze their dynamical properties and combine them with statistics that characterize the wave field. Specifically, we use the wave equations to understand which wave-groups (i.e. with what shape) are dynamically prone on evolving to extreme waves in the near future. This analysis gives a wide range of possible waves that indeed can evolve to extreme events. However, the sea state conditions are those that define the probability for the formation of each of those critical wave-groups. Combining these two pieces of information we identify the pattern of waves that i) has the highest likelihood to appear in given sea state conditions, and ii) if it appears it will evolve into an extreme wave. Having identified the critical wave-group, the next step is to use scanning technologies of the surrounding wave field in order to identify where this wave-group may appear. This process requires trivial computational effort and is very robust to measurement noise.

• • Pitch:  Laurence Vachon (Mathworks) Mission On Mars Robot Challenge 2016

  Présentation de la Compétition Mission on Mars, dont les inscriptions sont ouvertes jusqu'au 15 Avril 2016. La Compétition consiste à optimiser les algorithmes qui régissent le comportement du robot Rover sur Mars, afin qu'il détecte de manière autonome des sites d'exploration et qu'il évite des obstacles.

• Pitch: Florent Pignal, (drust.io) Drust: Application de la data science à des données du véhicule connecté!
• Pitch: Cyril Colin, Karim Elalami , eLum, Artificial Intelligence Driven Energy Management, Elum Energy - Load forecasting for nano-grid management
  
  

  "At Elum we leverage the power of our learning algorithms to store the solar power and deliver it when needed the most. Our technology allows to lower the electricity bill, ensures a reliable renewable energy supply and participates in the grid stability"

  
  

  Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Fast and Space-optimal Low-rank Factorization in the Streaming Model With Application in Differential Privacy

Streaming Matrix Factorization and differential privacy, interesting !

Fast and Space-optimal Low-rank Factorization in the Streaming Model With Application in Differential Privacy by Jalaj Upadhyay

In this paper, we consider the problem of computing a low-rank factorization of an m×n matrix in the general turnstile update model. We consider both the private and non-private setting. In the non-private setting, we give a space-optimal algorithm that computes a low-rank factorization. Our algorithm maintains three sketches of the matrix instead of five as in Boutsidis {\it et al.} (STOC 2016). Our algorithm takes O˜(1) time to update the sketch and computes the factorization in time linear in the sparsity and the dimensions of the matrix. In the private setting, we study low-rank factorization in the framework of differential privacy and under turnstile updates. We give two algorithms with respect to two levels of privacy. Both of our privacy levels are stronger than earlier studied privacy levels, namely that of Blocki {\it et al.} (FOCS 2012), Dwork {\it et al.} (STOC 2014), Hardt and Roth (STOC 2012, STOC 2013), and Hardt and Price (NIPS 2014).

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Around the blogs in 78 hours (Part Deux)

I have just updated the previous Around the blogs in 78 hours to reflect on my missing the entries of the excellent "Off the convex path" blog and Ben Recht's new blog Argmin.

Off the convex path

• Semantic Word Embeddings
• Tensor Methods in Machine Learning
• Nature, Dynamical Systems and Optimization
• NIPS 2015 workshop on non-convex optimization
• Word Embeddings: Explaining their properties
• Evolution, Dynamical Systems and Markov Chains
• Stability as a foundation of machine learning
• Escaping from Saddle Points
• Saddles Again
• Markov Chains Through the Lens of Dynamical Systems: The Case of Evolution

Ben

• The Hardest Part

h/t Sebastien

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Around the blogs in 78 hours

A rising, rotating column of hot air on Mars.

David McKay of "Information theory, inference and learning algorithms" fame writes

• Perhaps my last post - we'll see
• Index for the first 23 Cancer Chapters

I briefly met David back at lunch in 2006 during a Bayesian conference in Paris. One thing for sure, he is impressively quick, witty and smart: The discussion went from how he built Dasher to figuring out how to go about detecting potential earth smashing meteorites by digitizing old astronomical photos in repositories around the world (this is still an awesome idea). The underlying algorithm is the one used by star trackers on satellites. However since each of the photographs are taken by different instruments there is an additional layer of parameters to guess before figuring out where the picture is pointing at. Obviously, the probabilities for such an event are rather small compared to other more highly probable catastrophic events. In effect, David produced the much needed:

• Global Calculator
• Sustainable Energy Without the Hot Air

David continues tweeting.

In other news, here are some other blog entries (in no particular order):

John

• ICML registration is live 
• AlphaGo is not the solution to AI
   

Anand

• DIMACS Network Coding Workshop talks posted
• IHP “Nexus” Workshop on Privacy and Security: Day 1
• LabTV Profiles Are Up!
• Signal boost: IBM Social Good Fellowship for data science

Tomasz

• The Future of Real-Time SLAM and "Deep Learning vs SLAM"
• ICCV 2015: Twenty one hottest research papers

Andrew

• Black Box Challenge

John

• Well, F = ma.
• Frequentist properties of Bayesian methods

Dustin

• The Voronoi Means Conjecture
• The HRT Conjecture
• On the low-rank approach for semidefinite programs arising in synchronization and community detection
• Clustering noisy data with semidefinite relaxations
• Recent developments in equiangular tight frames
• Compressed Sensing and its Applications 2015

Fabian

• Lightning v0.1

Dirk

• Completing a measure space with respect to transport norms I

Joshua

• Hamiltonian Monte Carlo with Stan

Vladimir

• PIDA on CMOS Sensor Market
• Lytro Exits Consumer Camera Business

Suresh

• Time to cluster !
• Who owns a review, and who's having the conversation ?
• ITA FTW: Bayesian surprise and eigenvectors in your meal.
• Making all my reviews public (and annotated): A question.
• On "the moral hazard of complexity-theoretic assumptions"
• White Elephant parties and fair division
• Fairness and The Good Wife

Sebastien

• Notes on least-squares, part I
• AlphaGo is born

Bob

• Machine Listening Lab | Queen Mary University of London

Afonso

• SDP for Community Detection with many communities
• Generalized Power Method for SO(2) Synchronization
• Below the Kesten-Stigum bound for Community Detection
• Ten Lectures and Forty-Two Open Problems in the Mathematics of Data Science
• 18.S096: Synchronization Problems and Alignment
• 18.S096: Compressed Sensing and Sparse Recovery

Off the convex path

• Semantic Word Embeddings
• Tensor Methods in Machine Learning
• Nature, Dynamical Systems and Optimization
• NIPS 2015 workshop on non-convex optimization
• Word Embeddings: Explaining their properties
• Evolution, Dynamical Systems and Markov Chains
• Stability as a foundation of machine learning
• Escaping from Saddle Points
• Saddles Again
• Markov Chains Through the Lens of Dynamical Systems: The Case of Evolution

Ben

• The Hardest Part

 On Reddit, there were two interesting AMAs

• OpenAI Research Team (1/9/2016)
• Nando de Freitas (12/26/2015)

 On Quora a few people asnwered quite a few interesting questions:

Anima Anandkumar 

• What are the best tutorials/videos about Tensor Factorization/Tensor Decomposition?
• What are some introductory resources on tensors in machine learning?
• When would you recommend Tensor Factorization over Factorization Machines?
• What is the state of the art of Unsupervised Learning, and is human-like Unsupervised Learning possible in the near future?

Alex Smola

• Do I need to use deep learning to build a product recommandation engine in ecommerce?
• What are the major factors that motivate us to use Neural networks over Kernel methods for large datasets in layman terms?
• Are stochastic variational approaches the way to do large scale Bayesian ML or do you see any hope of scaling up MCMC-based algorithms?
• Which is your favorite Machine Learning Algorithm?

Xavier Amatriain

• I have seen Alternating Least Squares (ALS), SVD and NMF (Non negative matrix factorization) used for recommendation systems. Apart from the optimization function how are the approaches different? In what situations would one be better than the other?
• What is the best way to learn a machine learning algorithm?
• How would you define "data science"?
• Can Deep learning techniques be useful for recommender systems?
• What are some of the best recommendation algorithms?
• What is the most important unresolved problem in machine learning?

Image Credit: NASA/JPL-Caltech

Source: NASA/JPL-Caltech

Published: 4 April 2016

From its perch high on a ridge, NASA's Mars Exploration Rover Opportunity recorded this image of a Martian dust devil twisting through the valley below. The view looks back at the rover's tracks leading up the north-facing slope of "Knudsen Ridge," which forms part of the southern edge of "Marathon Valley."
Opportunity took the image using its navigation camera (Navcam) on March 31, 2016, during the 4,332nd Martian day, or sol, of the rover's work on Mars.
Dust devils were a common sight for Opportunity's twin rover, Spirit, in its outpost at Gusev Crater. Dust devils have been an uncommon sight for Opportunity though.
Just as on Earth, a dust devil is created by a rising, rotating column of hot air. When the column whirls fast enough, it picks up tiny grains of dust from the ground, making the vortex visible.
During the uphill drive to reach the top of Knudsen Ridge, Opportunity's tilt reached 32 degrees, the steepest ever for any rover on Mars.
NASA's Jet Propulsion Laboratory, a division of the California Institute of Technology in Pasadena, manages the Mars Exploration Rover Project for NASA's Science Mission Directorate, Washington.
For more information about Opportunity, visit http://www.nasa.gov/rovers and http://mars.nasa.gov/mer/.

  Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.