Highly Technical Reference Page: "the GAN Zoo" and "Delving deep into Generative Adversarial Networks (GANs)"

Much like what happened with the Advanced Matrix Factorization Jungle, here is a new Highly Technical Reference page, on a subject of increased interest that is difficult to follow for even a specialist: GANs. 

If you wonder what GANs are, take a look at the tutorial on Generative Adversarial Networks by Ian Goodfellow (and his NIPS slides) or John Glover's entry last August on the subject 'with TF code).

Avinash Hindupur who is behind deephunt.in recently listed the log series of GANs techniques in the GAN Zoo. From the page: 

Every week, new papers on Generative Adversarial Networks (GAN) are coming out and it’s hard to keep track of them all, not to mention the incredibly creative ways in which researchers are naming these GANs! You can read more about GANs in this Generative Models post by OpenAI or this overview tutorial in KDNuggets.

Avinash also mentions that the list can be expanded: 

You can visit the Github repository to add more links via pull requests or create an issue to lemme know something I missed or to start a discussion.

The subject is so hot that there is an earlier and somewhat more complete page on the subject
Delving deep into Generative Adversarial Networks (GANs) by Grigorios Kalliatakis

A curated list of state-of-the-art publications and resources about Generative Adversarial Networks (GANs) and their applications.....

Contributions are welcome !! If you have any suggestions (missing or new papers, missing repos or typos) you can pull a request or start a discussion.

The blog post introducing the page is here.


Both pages are added to the Highly Technical Reference page

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Jobs: Four Engineering positions at NVIDIA

Anita just sent me the following just before GTC2017 and subsequent discussion of their three billion dollars investment in the new chip effort (Tesla V100), the Volta Tensor Unit, Inference optimizers and their new cloud. Looks like the ML Hardware is eating the world !

Hi Igor,

Thanks for posting last time! I really appreciate. I have some other exciting roles I wanted to see if you are interested in posting, including a manager role! Thanks!

• Deep Learning Software Engineering Manager

• Sr Deep Learning Software Development Engineer

• Deep Learning Performance Architect

• Deep Learning Performance Architect

Thanks! Best Regards,

Anita Rexinger

NVIDIA Corporation

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Speckle-based hyperspectral imaging combining multiple scattering and compressive sensing in nanowire mats

The paper has been pusblished but it was also just put on arxiv since Optics Letters is a Romeo green journal and allows preprints and even postprints to be archived. Enjoy !

Speckle-based hyperspectral imaging combining multiple scattering and compressive sensing in nanowire mats by Rebecca French, Sylvain Gigan, Otto L. Muskens

Encoding of spectral information onto monochrome imaging cameras is of interest for wavelength multiplexing and hyperspectral imaging applications. Here, the complex spatio-spectral response of a disordered material is used to demonstrate retrieval of a number of discrete wavelengths over a wide spectral range. Strong, diffuse light scattering in a semiconductor nanowire mat is used to achieve a highly compact spectrometer of micrometer thickness, transforming different wavelengths into distinct speckle patterns with nanometer sensitivity. Spatial multiplexing is achieved through the use of a microlens array, allowing simultaneous imaging of many speckles, ultimately limited by the size of the diffuse spot area. The performance of different information retrieval algorithms is compared. A compressive sensing algorithm exhibits efficient reconstruction capability in noisy environments and with only a few measurements.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Paris Machine Learning Meetup #9 Season 4 @ IHP, Bias, Ethics & Fair Algorithms

video of the streaming is here:

Ce soir le meetup se fera à l'IHP, Cette soirée sera présidée par Cédric Villani.

Merci à l'IHP pour nous accueillir et remerciement à Quantmetry qui offre le buffet de clôture.

Pour cette soirée, nous souhaitons mettre l’accent sur les aspects sociaux, éthiques, philosophiques et juridiques.

Ceci afin de changer notre point de vue de Data Scientist et nous extraire de la technique.

Les développements récents (Amazon Go, voitures autonomes, bots conversationnels, objets connectés, drones militaires autonomes, ventes & police prédictives, ...) nous montre qu’on peut difficilement concevoir des algorithmes prédictifs sans s’interroger sur leur finalité, leur biais ainsi que sur les répercussions sociales de ces techniques.

Les décideurs politiques de tous les pays se penchent sur ces questions qui deviennent centrales. Ainsi, pour la France, la commission OPECST a publié un rapport dont voici les préconisations (le rapport en entier se trouve http://www.senat.fr/notice-rapport/2016/r16-464-1-notice.html, http://www.senat.fr/notice-rapport/2016/r16-464-2-notice.html ).

Il y aura une petite introduction par Franck Bardol et moi sur le sujet.

Les invités

• Cedric Villani
• Dominique Cardon
• Cynthia Fleury

Programme

Prise de parole libre des intervenants pendant 20 min chacun.

Suivi de questions - réponses avec l'auditoire




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Video: "Can the brain do back-propagation?" Goeff Hinton

The talk by Goeff was given a year ago at Stanford. I liked that sentence at 20 minutes and 30 seconds:

 ...I think it is a good idea trying to always try make the data look small by using a huge model, now this relies on you having more almost free computations...

I added below two papers mentioned in the talk  

Learning Representation by Recirculation by Geoffrey E. Hinton, James L. McClelland

We describe a new learning procedure for networks that contain groups of nonlinear units arranged in a closed loop. The aim of the learning is to discover codes that allow the activity vectors in a "visible" group to be represented by activity vectors in a "hidden" group. One way to test whether a code is an accurate representation is to try to reconstruct the visible vector from the hidden vector. The difference between the original and the reconstructed visible vectors is called the reconstruction error, and the learning procedure aims to minimize this error. The learning procedure has two passes. On the first pass, the original visible vector is passed around the loop, and on the second pass an average of the original vector and the reconstructed vector is passed around the loop. The learning procedure changes each weight by an amount proportional to the product of the "presynaptic" activity and the difference in the post-synaptic activity on the two passes. This procedure is much simpler to implement than methods like back-propagation. Simulations in simple networks show that it usually converges rapidly on a good set of codes, and analysis shows that in certain restricted cases it performs gradient descent in the squared reconstruction error.

Random feedback weights support learning in deep neural networks by Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, Colin J. Akerman

The brain processes information through many layers of neurons. This deep architecture is representationally powerful, but it complicates learning by making it hard to identify the responsible neurons when a mistake is made. In machine learning, the backpropagation algorithm assigns blame to a neuron by computing exactly how it contributed to an error. To do this, it multiplies error signals by matrices consisting of all the synaptic weights on the neuron's axon and farther downstream. This operation requires a precisely choreographed transport of synaptic weight information, which is thought to be impossible in the brain. Here we present a surprisingly simple algorithm for deep learning, which assigns blame by multiplying error signals by random synaptic weights. We show that a network can learn to extract useful information from signals sent through these random feedback connections. In essence, the network learns to learn. We demonstrate that this new mechanism performs as quickly and accurately as backpropagation on a variety of problems and describe the principles which underlie its function. Our demonstration provides a plausible basis for how a neuron can be adapted using error signals generated at distal locations in the brain, and thus dispels long-held assumptions about the algorithmic constraints on learning in neural circuits.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

SPARS 2017: The program is out !

Mario just sent me the following:

Dear Igor, 

The program of SPARS 2017 is now available at the workshop website: 

http://spars2017.lx.it.pt 

This may be of interest to the readers of Nuit Blanche (many of which will be attending SPARS). SPARS 2017 will feature an excellent set of 8 plenary speakers, 34 oral presentations, and 111 posters, on the general area of sparsity-related techniques and computational methods, for high dimensional data analysis, signal processing, and related applications. 

Best regards,
Mario

Thanks for the update Mario !





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Sunday Morning Videos: Deep Learning and Artificial Intelligence symposium at NAS 154th Annual Meeting

Yann points to this series of videos of a symposium of the National Academies of Sciences. Noteworthy is Bill Press who introduces the symposium. This is quite fitting as Bill is a major figure behind the Numerical Recipes that has changed algorithm use in Engineering and Science in the mid-90's.

Deep Learning and Artificial Intelligence  

In less than a decade, the field of “artificial intelligence” or “AI” has been jolted by the extraordinary and unexpected success of a set of techniques now called “Deep Learning”. These methods (with some other related rapidly advancing technologies) already exceed average human performance in some kinds of image understanding; spoken word recognition and language translation; and indeed some tasks, like the game of Go, previously thought to require generalized human intelligence. AI may soon replace humans in driving cars, coding new software, robotic caregiving, and making healthcare decisions. The societal implications are enormous. In this session, experts in the field discuss this revolution from five different perspectives. The Symposium has concluded. A recording is available above.

• Introduction and Moderation by William H. Press, Raymer Professor in Computer Science, The University of Texas at Austin 
• The Deep Learning and Artificial Intelligence Revolution Rob Fergus, Research Lead, Facebook AI Research, and Associate Professor, New York University 
• Rethinking the Software Industry Peter Norvig, Director of Research, Google Inc.
• How Machines Learn by Doing Michael Littman, Professor of Computer Science, Brown University
• Autonomous Machines Interacting with Humans Manuela Veloso, Herbert A. Simon University Professor of Computer Science; and Head, Machine Learning Department, Carnegie Mellon University
• The Impact of Artificial Intelligence on Healthcare, Suchi Saria, Assistant Professor of Computer Science, Johns Hopkins University

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday Morning Videos: Computational Challenges in Machine Learning, Simons Institute, May 1 – May 5, 2017

Jupiter’s Swirling ‘Pearl’ Storm

Santosh Vempala, David Blei, Katherine Heller, John Langford and Le Song with the Simons Institute at UC Berkeley just organised a workshop on Computational Challenges in Machine Learning this week. The videos can be accessed by following each link.

The aim of this workshop is to bring together a broad set of researchers looking at algorithmic questions that arise in machine learning. The primary target areas will be large-­scale learning, including algorithms for Bayesian estimation and variational inference, nonlinear and nonparametric function estimation, reinforcement learning, and stochastic processes including diffusion, point processes and MCMC. While many of these methods have been central to statistical modeling and machine learning, recent advances in their scope and applicability lead to basic questions about their computational efficiency. The latter is often linked to modeling assumptions and objectives. The workshop will examine progress and challenges and include a set of tutorials on the state of the art by leading experts.

• Variational Inference: Foundations and Innovations David Blei, Columbia University
• Representational and Optimization Properties of Deep Residual Networks Peter Bartlett, UC Berkeley
• Composing Graphical Models with Neural Networks for Structured Representations and Fast Inference David Duvenaud, University of Toronto
• Scaling Up Bayesian Inference for Big and Complex Data David Dunson, Duke University
• Unbiased Estimation of the Spectral Properties of Large Implicit Matrices Ryan Adams, Harvard University
• Stochastic Gradient MCMC for Independent and Dependent Data Sources Emily Fox, University of Washington
• On Gradient-Based Optimization: Accelerated, Distributed, Asynchronous and Stochastic Michael Jordan, UC Berkeley
• Sampling Polytopes: From Euclid to Riemann Yin Tat Lee, Microsoft Research and University of Washington
• Understanding Generalization in Adaptive Data Analysis Vitaly Feldman, IBM Almaden
• Gradient Descent Learns Linear Dynamical Systems Moritz Hardt, Google
• Your Neural Network Can't Learn Mine John Wilmes, Georgia Institute of Technology
• Efficient Distributed Deep Learning Using MXNet Anima Anandkumar, UC Irvine
• Machine Learning for Healthcare Data Katherine Heller, Duke University
• Machine Learning Combinatorial Optimization Algorithms Dorit Hochbaum, UC Berkeley
• A Cost Function for Similarity-Based Hierarchical Clustering Sanjoy Dasgupta, UC San Diego
• The Imitation Learning View of Structured Prediction Hal Daume, University of Maryland at College Park
• Exponential Computational Improvement by Reduction John Langford, Microsoft Research New York
• Simons Institute Open Lecture: "Does Computational Complexity Restrict Artificial Intelligence (AI) and Machine Learning? Sanjeev Arora, Princeton University
• Embedding as a Tool for Algorithm Design Le Song, Georgia Institute of Technology
• Robust Estimation of Mean and Covariance Anup Rao, Georgia Institute of Technology
• Computational Efficiency and Robust Statistics Ilias Diakonikolas, University of Southern California
• System and Algorithm Co-Design, Theory and Practice, for Distributed Machine Learning Eric Xing, Carnegie Mellon University
• The Polytope Learning Problem Navin Goyal, Microsoft Research
• Computational Challenges and the Future of ML Panel
• Computationally Tractable and Near Optimal Design of Experiments Aarti Singh, Carnegie Mellon University
• Learning from Untrusted Data Gregory Valiant, Stanford University
• The “Tell Me Something New” Model of Computation for Machine Learning Yoav Freund, UC San Diego

Credits: NASA/JPL-Caltech/SwRI/MSSS/Jason Major

This image, taken by the JunoCam imager on NASA’s Juno spacecraft, highlights a swirling storm just south of one of the white oval storms on Jupiter.

The image was taken on March 27, 2017, at 2:12 a.m. PDT (5:12 a.m. EDT), as the Juno spacecraft performed a close flyby of Jupiter. At the time the image was taken, the spacecraft was about 12,400 miles (20,000 kilometers) from the planet.

Citizen scientist Jason Major enhanced the color and contrast in this image, turning the picture into a Jovian work of art. He then cropped it to focus our attention on this beautiful example of Jupiter’s spinning storms.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Hamming's Time: The Desperate Researchers of Computer Vision

Image Credit: NASA/JPL-Caltech/Space Science Institute

Back in August 2014, I mentioned this roundtable at the Technion ( Is Deep Learning the Final Frontier and the End of Signal Processing ? ). Then Yoshua provided his view on Deep Learning. Two years and half later, Miki just wrote the following op-ed in SIAM News: Deep, Deep Trouble: Deep Learning’s Impact on Image Processing, Mathematics, and Humanity. It is difficult to understate how much of The Great Convergence we are witnessing. This is how it starts.

I am really confused. I keep changing my opinion on a daily basis, and I cannot seem to settle on one solid view of this puzzle. No, I am not talking about world politics or the current U.S. president, but rather something far more critical to humankind, and more specifically to our existence and work as engineers and researchers. I am talking about…deep learning.....

.....Now back to the main question: should we be pleased about emerging solutions based on deep learning? Is our frustration justified? What is the role of deep learning in imaging science? These questions present themselves when researchers in the community meet at conferences, and the answers are diverse and confusing. The facts speak loudly for themselves; in most cases, deep learning-based solutions lack mathematical elegance and offer very little interpretability of the found solution or understanding of the underlying phenomena. On the positive side, however, the performance obtained is terrific. This is clearly not the school of research we have been taught, and not the kind of science we want to practice. Should we insist on our more rigorous ways, even at the cost of falling behind in terms of output quality? Or should we fight back and seek ways to fuse ideas from deep learning into our more solid foundations?
To further complicate this story, certain deep learning-based contributions bear some elegance that cannot be dismissed. Such is the case with the style-transfer problem, which yielded amazingly beautiful results, and with inversion ideas of learned networks used to synthesize images out of thin air, as Google’s Deep Dream project does. A few years ago we did not have the slightest idea how to formulate such complicated tasks; now they are solved formidably as a byproduct of a deep neural network trained for the completely extraneous task of visual classification. 

The whole thing is here.
h/t Ulugbeck

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices - implementation -

Mehmet just sent me the following

Hello Igor,

Hope you are all well.

The recent manuscript [1,3] might be of interest to nuit-blanche blog readers. The manuscript has developed an ergodicity measure for spectral analysis using KL divergence and applied to circular ensembles. The software package is also available along with the Python notebook.

Please feel free to use the material in your blog.

Many regards,
Mehmet

[1] https://arxiv.org/abs/1704.08303
[2] https://pypi.python.org/pypi/bristol
https://github.com/msuzen/bristol/blob/master/works/spectralErgodicity/ergodicity_random_matrix_17a.ipynb

[3] Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices by Mehmet Süzen, Cornelius Weber, Joan J. Cerdà

Using random matrix ensembles, mimicking weight matrices from deep and recurrent neural networks, we investigate how increasing connectivity leads to higher accuracy in learning with a related measure on eigenvalue spectra. For this purpose, we quantify spectral ergodicity based on the Thirumalai-Mountain (TM) metric and Kullbach-Leibler (KL) divergence. As a case study, different size circular random matrix ensembles, i.e., circular unitary ensemble (CUE), circular orthogonal ensemble (COE), and circular symplectic ensemble (CSE), are generated. Eigenvalue spectra are computed along with the approach to spectral ergodicity with increasing connectivity size. As a result, it is argued that success of deep learning architectures attributed to spectral ergodicity conceptually, as this property prominently decreases with increasing connectivity in surrogate weight matrices. 

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.