Nuit Blanche

Saturday, August 27, 2016

Saturday Morning Videos: ICML 2016 Plenary, Tutorials and more....

The videos for ICML 2016 are out ! They are all here: http://techtalks.tv/icml/2016/orals/

Thank you to the organizing committee to making these videos available. Here are the plenary and tutorial talks

Plenary

Tutorials

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, August 26, 2016

Lecture Notes on Randomized Linear Algebra / Spectral Graph Methods by Michael Mahoney

Michael Mahoney just made available 188 pages of his lecture notes on Randomized Linear Algebra and 257 pages of his lecture notes on Spectral Graph Methods . These are lecture notes that are based on the lectures from a class Michael taught on the topic of Randomized Linear Algebra (RLA) at UC Berkeley during the Fall 2013 semester. The second lecture notes are based on the lectures from a class Michael taught on the topic of Spectral Graph Methods at UC Berkeley during the Spring 2015 semester.

This blog post is filed under the Overviews tag.

Credit: NASA, Discovr

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Factorized Binary Codes for Large-ScaleNearest Neighbor Search

Factorized Binary Codes for Large-ScaleNearest Neighbor Search by Frederick Tung , James J. Little

Hashing algorithms for fast large-scale nearest neighbor search transform data points into compact binary codes by applying a set of learned or randomly generated hash functions. Retrieval accuracy generally increases with the number of hash functions, but increasing the number of hash functions also increases the storage requirements of the resulting binary codes. We present a novel factorized binary codes approach that uses an approximate matrix factorization of the binary codes to increase the number of hash functions while maintaining the original storage requirements. The proposed approach does not assume a particular algorithm for generating the hash functions, and requires only that we can discover and take advantage of correlations among the hash functions. Experiments on publicly available datasets suggest that factorized binary codes work particularly well for locality-sensitive hashing algorithms.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, August 25, 2016

Videos: Deep Learning Summer School 2016

The videos and slides of the Deep Learning Summer School 2016 are out on VideoLectures.net:

Invited Talks

Machine Learning Doina Precup
Neural Networks Hugo Larochelle
Introduction to Theano Pascal Lamblin
Convolutional Neural Networks and Computer Vision Rob Fergus
Learning to See Antonio Torralba
Introduction to Torch Alex Wiltschko
Recurrent Neural Networks Yoshua Bengio
Reasoning, Attention and Memory Sumit Chopra
Large Scale Deep Learning with TensorFlow Jeffrey Dean
Deep Natural Language Understanding Kyunghyun Cho
Beyond Seq2Seq with Augmented RNNs Edward Grefenstette
GPU programming for Deep Learning Julie Bernauer, Ryan Olson
Introduction to Reinforcement Learning Joelle Pineau
Deep Reinforcement Learning Pieter Abbeel
Learning Deep Generative Models Ruslan Salakhutdinov
Building Machines that Imagine and Reason: Principles and Applications of Deep Generative Models Shakir Mohamed
Beyond inspiration: Five lessons from biology on building intelligent machines Bruno Olshausen
Theoretical neuroscience and deep learning theory Surya Ganguli

Contributed Talks

Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks Rajarshi Das
Professor Forcing: A New Algorithm for Training Recurrent Networks Anirudh Goyal
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations Tegan Maharaj
Deep multi-view representation learning of brain responses to natural stimuli Leila Wehbe
Advanced Topics in LR Doina Precup
Advanced Topics in LR Joelle Pineau
Learning to Communicate with Deep Multi-Agent Reinforcement Learning Jakob Foerster
Model-Based Relative Entropy Stochastic Search Abbas Abdolmaleki
Learning Nash Equilibrium for General-Sum Markov Games from Batch Dat Julien Pérolat
A Network-based End-to-End Trainable Task-oriented Dialogue System Tsung-Hsien Wen
Inference Learning Patrick Putzky
Variational Autoencoders with PixelCNN Decoders Ishaan Gulrajani
An Infinite Restricted Boltzmann Machine Marc-Alexandre Côté
Deep siamese neural network for prediction of long-range interactions in chromatin Davide Chicco
Beam Search Message Passing in Bidirectional RNNs: Applications to Fill-in-the-Blank Image Captioning Qing Sun
Analyzing the Behavior of Deep Visual Question Answering Models Aishwarya Agrawal
Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation Sina Honari

h/t Edward Grefenstette

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Wednesday, August 24, 2016

How to Fake Multiply by a Gaussian Matrix

You know what happens when Random Matrices Are Too Damn Large and so yuuuggge, they cannot be held in memory nor processed ? well, the answer is to fake them till you make it. I like how the authors used the phase transition diagrams to show the similarity of two algorithms.

How to Fake Multiply by a Gaussian Matrix by Michael Kapralov, Vamsi K. Potluru, David P. Woodruff

Have you ever wanted to multiply an n×d matrix X, with n≫d, on the left by an m×n matrix G~ of i.i.d. Gaussian random variables, but could not afford to do it because it was too slow? In this work we propose a new randomized m×n matrix T, for which one can compute T⋅X in only O(nnz(X))+O~(m1.5⋅d3) time, for which the total variation distance between the distributions T⋅X and G~⋅X is as small as desired, i.e., less than any positive constant. Here nnz(X) denotes the number of non-zero entries of X. Assuming nnz(X)≫m1.5⋅d3, this is a significant savings over the na\"ive O(nnz(X)m) time to compute G~⋅X. Moreover, since the total variation distance is small, we can provably use T⋅X in place of G~⋅X in any application and have the same guarantees as if we were using G~⋅X, up to a small positive constant in error probability. We apply this transform to nonnegative matrix factorization (NMF) and support vector machines (SVM).

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Direct inference on compressive measurements using convolutional neural networks

Here is another instance of the great convergence. Here the first layer of a deep network is a compressive sensing system. The results are Ok, it is just the beginning.

Direct inference on compressive measurements using convolutional neural networks by Suhas Lohit Kuldeep Kulkarni , Pavan Turaga

Compressive imagers, e.g. the single-pixel camera (SPC), acquire measurements in the form of random projections of the scene instead of pixel intensities. Compressive Sensing (CS) theory allows accurate reconstruction of the image even from a small number of such projections. However, in practice, most reconstruction algorithms perform poorly at low measurement rates and are computationally very expensive. But perfect reconstruction is not the goal of high-level computer vision applications. Instead, we are interested in only determining certain properties of the image. Recent work has shown that effective inference is possible directly from the compressive measurements, without reconstruction, using correlational features. In this paper, we show that convolutional neural networks (CNNs) can be employed to extract discriminative non-linear features directly from CS measurements. Using these features, we demonstrate that effective high-level inference can be performed. Experimentally, using hand written digit recognition (MNIST dataset) and image recognition (ImageNet) as examples, we show that recognition is possible even at low measurement rates of about 0.1.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Tuesday, August 23, 2016

Stable Reinforcement Learning with Autoencoders for Tactile and Visual Data

Here is an interesting bit where random features are used to approximate control policy modeling in reinforcement learning and deep auto-encoders are used to figure out the latent dimension of the model.

Stable Reinforcement Learning with Autoencodersfor Tactile and Visual Data by Herke van Hoof, Nutan Chen, Maximilian Karl , Patrick van der Smagt, Jan Peters

For many tasks, tactile or visual feedback is helpful or even crucial. However, designing controllers that take such high-dimensional feedback into account is non-trivial. Therefore, robots should be able to learn tactile skills through trial and error by using reinforcement learning algorithms. The input domain for such tasks, however, might include strongly correlated or non-relevant dimensions, making it hard to specify a suitable metric on such domains. Auto-encoders specialize in finding compact representations, where defining such a metric is likely to be easier. Therefore, we propose a reinforcement learning algorithm that can learn non-linear policies in continuous state spaces, which leverages representations learned using auto-encoders. We first evaluate this method on a simulated toytask with visual input. Then, we validate our approach on a real-robot tactile stabilization task.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Monday, August 22, 2016

Decoupled Neural Interfaces using Synthetic Gradients

Last week in Stacked Approximated Regression Machine: A Simple Deep Learning Approach we mentioned a way for neural networks to be built without a backpropagation step. Another NIPS paper proposes to do the same using a different technique. Delip talks about it on his blog. The Great Convergence continues...

Decoupled Neural Interfaces using Synthetic Gradients by Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu

Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated. In this work we break this constraint by decoupling modules by introducing a model of the future computation of the network graph. These models predict what the result of the modelled subgraph will produce using only local information. In particular we focus on modelling error gradients: by using the modelled synthetic gradient in place of true backpropagated error gradients we decouple subgraphs, and can update them independently and asynchronously i.e. we realise decoupled neural interfaces. We show results for feed-forward models, where every layer is trained asynchronously, recurrent neural networks (RNNs) where predicting one's future gradient extends the time over which the RNN can effectively model, and also a hierarchical RNN system with ticking at different timescales. Finally, we demonstrate that in addition to predicting gradients, the same framework can be used to predict inputs, resulting in models which are decoupled in both the forward and backwards pass -- amounting to independent networks which co-learn such that they can be composed into a single functioning corporation.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Full Resolution Image Compression with Recurrent Neural Networks

The real question is not whether we are seeing another instance of the Great Convergence, but when is Deepmind going to beat the Weissmann score of Pied Piper Inc ?

Full Resolution Image Compression with Recurrent Neural Networks by George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, Michele Covell

This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study "one-shot" versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%-8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Friday, August 19, 2016

Reviews: Low-Rank Semidefinite Programming:Theory and Applications / A Primer on Reproducing Kernel Hilbert Spaces

In the same spirit as the Highly Technical Reference pages, I have created an "Overviews" tag that links back to all the entries that provide a review or an overview of a specific field. Today we have two:

Low-Rank Semidefinite Programming:Theory and Applications by Alex Lemon, Anthony Man-Cho So, Yinyu Ye

Finding low-rank solutions of semidefinite programs is important in many applications. For example, semidefinite programs that arise as relaxations of polynomial optimization problems are exact relaxations when the semidefinite program has a rank-1 solution. Unfortunately, computing a minimum-rank solution of a semidefinite program is an NP-hard problem. In this paper we review the theory of low-rank semidefinite programming, presenting theorems that guarantee the existence of a low-rank solution, heuristics for computing low-rank solutions, and algorithms for finding low-rank approximate solutions. Then we present applications of the theory to trust-region problems and signal processing.

A Primer on Reproducing Kernel Hilbert Spaces by Jonathan H. Manton, Pierre-Olivier Amblard

Reproducing kernel Hilbert spaces are elucidated without assuming prior familiarity with Hilbert spaces. Compared with extant pedagogic material, greater care is placed on motivating the definition of reproducing kernel Hilbert spaces and explaining when and why these spaces are efficacious. The novel viewpoint is that reproducing kernel Hilbert space theory studies extrinsic geometry, associating with each geometric configuration a canonical overdetermined coordinate system. This coordinate system varies continuously with changing geometric configurations, making it well-suited for studying problems whose solutions also vary continuously with changing geometry. This primer can also serve as an introduction to infinite-dimensional linear algebra because reproducing kernel Hilbert spaces have more properties in common with Euclidean spaces than do more general Hilbert spaces.

Image Credit: NASA/JPL-Caltech/Space Science Institute

W00100570.jpghttp://saturnraw.jpl.nasa.gov/multimedia/raw/?start=1#raw-364127 was taken on 2016-08-16 22:59 (UTC) and received on Earth 2016-08-17 07:42 (UTC). The camera was pointing toward SATURN, and the image was taken using the MT3 and CL2 filters.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !