Tuesday, April 10, 2018

Videos: The "Institute for Advanced Study - Princeton University Joint Symposium on 'The Mathematical Theory of Deep Neural Networks'"

Adam followed through with videos of the awesome workshop he co-organized last month:
Hi Igor,

Thanks for posting about our recent workshop --- The "Institute for Advanced Study - Princeton University Joint Symposium on 'The Mathematical Theory of Deep Neural Networks'" --- last month. I just wanted to follow up and let you know that for those that missed the live-stream, we have put videos of all the talks up online:

https://www.youtube.com/playlist?list=PLWQvhvMdDChyI5BdVbrthz5sIRTtqV6Jw

I hope you and your readers enjoy!

Cheers,

-Adam
----------------------------Adam CharlesPost-doctoral associatePrinceton Neuroscience InstitutePrinceton, NJ, 08550 

Thanks Adam ! Here are the videos:

9:10 Adam Charles: Introductory remarks


2
56:17 Sanjeev Arora: Why do deep nets generalize, that is, predict well on unseen data


3
59:34 Sebastian Musslick: Multitasking Capability vs Learning Efficiency in Neural Network Architectures


4
48:01 Joan Bruna: On the Optimization Landscape of Neural Networks


5
59:44 Andrew Saxe: A theory of deep learning dynamics: Insights from the linear case


6
51:13 Anna Gilbert: Toward Understanding the Invertibility of Convolutional Neural Networks


7
1:03:57 Nadav Cohen: On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, April 09, 2018

Efficient Neural Architecture Search via Parameter Sharing - implementation -

Melody mentions on her twitter feed that an implementation of her work is now available.




We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Thanks to parameter sharing between child models, ENAS is fast: it delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new state-of-the-art among all methods without post-training processing. On the CIFAR-10 dataset, ENAS designs novel architectures that achieve a test error of 2.89%, which is on par with NASNet (Zoph et al., 2018), whose test error is 2.65%.

The implementation in TensorFlow is here: https://github.com/melodyguan/enas
and in PyTorch: https://github.com/carpedm20/ENAS-pytorch







Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Saturday, April 07, 2018

Saturday Morning Videos: Bandit Convex Optimization, PGMO Lecture 1 and 2


Sebastien did four lectures on Bandit Convex Optimization for the Gaspard Monge Program in Optimization. Two of them are on Sebastien YouTube channel. Here is the abstract:

The multi-armed bandit and its variants have been around for more than 80 years, with applications ranging from medial trials in the 1930s to ad placement in the 2010s. In this mini-course I will focus on a groundbreaking model introduced in the 1990s which gets rid of the unrealistic i.i.d. assumption that is standard in statistics and learning theory. This paradigm shift leads to exciting new mathematical and algorithmic challenges. I will focus the lectures on the foundational results of this burgeoning field, as well as their connections with classical problems in mathematics such as the geometry of martingales and high dimensional phenomena. 
  • Lecture 1: Introduction to regret. Game theoretic viewpoint (duality, Bayesian version of the game) and derivation of the minimax regret via geometry of martingales (brief recall of type/cotype and entropic proof for ell_1). 
  • Lecture 2: Introduction to the mirror descent algorithm. Connections with competitive analysis in online computations will also be discussed. 
  • Lecture 3: Bandit Linear Optimization. Two proofs of optimal regret: one via low-rank decomposition in the information theoretic argument, and the other via mirror descent with self-concordant barriers. 
  • Lecture 4 : Bandit Convex optimization 1. Kernel methods for online learning, Bernoulli convolution based kernel. 2. Gaussian approximation of Bernoulli convolutions, and restart type strategies.


Bandit Convex Optimization, PGMO Lecture 1 (slides)




Bandit Convex Optimization, PGMO Lecture 2 (slides)



Bandit Convex Optimization, PGMO Lecture 3 slides and Lecture 4 slides..



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, April 06, 2018

Blocked Direct Feedback Alignment: Exploring the Benefits of Direct Feedback Alignment

Interesting exploration of DFA concepts !


Blocked Direct Feedback Alignment:Exploring the Benefits of Direct Feedback Alignment by Mateo Espinosa Zarlenga, Eyvind Niklasson

Backpropagation is undoubtedly the preferred method for training deep feedforward neural networks. While this method has proven its effectiveness on applications ranging over a myriad of different fields, it has some well-known drawbacks. Moreover, this algorithm is arguably far from being biologically plausible, which makes it very unattractive as a crucial step of any attempt for an accurate model of our brain. Alternatives like feedback alignment and direct feedback alignment has then been proposed recently as possible methods that are more biologically plausible than backpropagation while also correcting some of the know drawbacks of this algorithm. For this project, we explore the uses of this last method, direct feedback alignment (DFA), by looking at variants of the same that could lead to improvements in both training convergence times and testing-time accuracies. We present two main variants: Feedback Propagation (FP) and Blocked Direct Feedback Alignment (BDFA). These variants of DFA attempt to find some sort of equilibrium between DFA and backpropagation that takes advantage of the benefits in both methods. In our experiments we manage to empirically show that BDFA outperforms both DFA and backpropagation in terms of convergence time and testing performance when used to train very deep neural networks with fully connected layers on MNIST and notMNIST. 




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

SUNLayer: Stable denoising with generative networks

Dustin just let me know of the following item to be added to The Great Convergence:

Hi Igor, 
I wanted to point you to a recent paper on the arXiv: 
I think you'll like Figure 1 in particular.
Apparently, GANs provide signal models that allow for extremely good denoising in a high-noise regime. To denoise, we hunt for the point in the GAN model that's closest to the noisy image. Surprisingly, local minimization works well in practice. To help explain this, we provide theory for a certain model of neural networks using techniques from spherical harmonics. This is joint work with Soledad Villar (NYU).
Cheers,
Dustin

Yes, you're right, I do like Figure 1 ! 



It has been experimentally established that deep neural networks can be used to produce good generative models for real world data. It has also been established that such generative models can be exploited to solve classical inverse problems like compressed sensing and super resolution. In this work we focus on the classical signal processing problem of image denoising. We propose a theoretical setting that uses spherical harmonics to identify what mathematical properties of the activation functions will allow signal denoising with local methods.



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Thursday, April 05, 2018

Processeurs optiques et traitement de données de grande dimension/ Optical Co-Processors and High Dimensional Data Processing, Paris, April 5, 2018

So today, we'll do a presentation of where we are at LightOn. Both Laurent and I will be speaking at the Paris Science and Data eventHere is the anouncement on Inria's website. Nicolas Keriven is one of one of our first alpha users of LightOn Cloud.




Paris Science & Data est une série d’événements organisés conjointement par le pôle Cap Digital, l’Inria et PSL, et destinés à présenter des recherches concernant la science des données, ainsi que leurs applications dans le monde académique et dans celui des entreprises.
Au programme de cette 8e conférence différents intervenants prendront la parole sur les sujets suivants :
  • From computational imaging to optical computing (Laurent Daudet - Professeur Paris Diderot/Institut Langevin & CTO LightOn)
  • Online sketches with random features (Nicolas Keriven - Chercheur ENS, CFM-ENS ''Laplace'' chair in Data Science)
  • Lighton : une nouvelle génération de coprocesseurs optiques (Igor Carron - CEO LightOn)




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, April 04, 2018

Paris Machine Learning Meetup Newsletter, Avril 2018: “Be the change you wish to see in the world” [In French]


Paris Machine Learning Meetup Newsletter, Avril 2018: “Be the change you wish to see in the world” [In French]
Table of content
  1. Le prochain meetup du Paris Machine Learning, 11 Avril.
  2. “Be the change you wish to see in the world”: Le rapport Villani et l’événement #AIforHumanity et le reste.
  3. On aime vraiment beaucoup
  4. Le Calendrier
  1. Allez on y va

1. Le prochain meetup 

On vient d’ouvrir le meetup maintenant on a besoin d’une salle pour le prochain meetup du 11 Avril. Contactez-nous. Franck ou Igor;

Nous sommes maintenant 6 800 membres, 1983 sur LinkedIn et plus de 2300 sur Twitter.


2. “Be the change you wish to see in the world”: Le rapport Villani et l’événement #AIforHumanity et le reste.


Le rapport Villani se trouve ici.L’autre rapport de France Stratégie de l’IA et le travail se trouve ici. L’événement ainsi que les vidéos de l’événement #AIforHumanity sont ici.


Comme le dit Cédric Villani “il fallait guider et proposer des axes pour la puissance publique mais aussi expliquer que l’#IA est l’affaire de toute la société et que tous les secteurs seront touchés”


A la fin, la société change grâce aux discussions et échanges qui se passent aux différents meetups de Machine Learning dans toute la France et dans le travail mis dans des projets tels que scikit-learn ou Keras.

On remarque aussi qu’à l’opposé de beaucoup de thématiques, la vitesse d’évolution de notre communauté est plus rapide que les constantes de temps des rapports officiels. Pour exemple: Netflix, l’entreprise qui se définit comme faisant majoritairement du Machine Learning (et que nous avions présentée dans l’un des meetups) est en discussion pour reprendre Europacorp de Luc Besson. Dans la même semaine NVIDIA a fait des annonces bluffantes au GTC, et on a vu les différentes annonces de Google sur TensorFlow (dont TensorFlow;js). Localement, cette semaine pas moins de trois meetups/rencontres (voir le calendrier plus bas) se passent à Paris dans la soirée du 4 avril et ils seront tous remplis. Dans la prochaine rubrique, il y a plusieurs projets, plein de dates à retenir dans votre calendrier du mois, deux summer schools et une conférence. En somme, comme Gandhi disait, tout ça commence ici : “Be the change you wish to see in the world”.


3. On aime vraiment beaucoup


Projets:


4. Le Calendrier


Deux Summer schools et une conférence:




Voilà, c’est tout pour aujourd’hui !




PS: N’oubliez pas que vous pouvez aussi suivre le Paris Machine Learning Meetup sur Twitter, LinkedIn, Facebook et Google+ .


Vous pouvez consulter les archives des meet ups précédents.


On travaille aussi sur un nouveau site web : MLParis.org


Le Paris Machine Learning Meetup, c’est 6800 membres ce qui en fait un des plus important du monde avec déjà plus de 80 rencontres et encore cinq dates programmées pour cette saison 5.


. Si vous êtes étudiant, postdoc ou chercheur, le meet up est une belle tribune pour parler de vos travaux avant de les présenter aux conférences NIPS/ICML/ICLR/COLT/UAI/ACL/KDD ;


. Pour les startups, c’est un bon moyen de parler de vos projets ou de recruter les futurs superstars de votre équipe IA/Data Science ;


. Et pour tous, c’est un moyen simple de se tenir informé des derniers développements du domaine et d’avoir des échanges uniques avec les conférenciers et les autres participants.


Comme toujours, premier arrivé, premier entré. Le nombre de places dans les salles est limité. Au-delà de leur capacité, nous ne pourrons pas vous faire rentrer. Vous pouvez suivre le taux de remplissage en suivant #MLParis sur twitter.







Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.


Monday, April 02, 2018

Artificial Intelligence Summer School, Grenoble, 2-6 July 2018



Julien let me know of this Summer school in Grenoble and the attendant deadline which is coming up pretty fast:

Inria and NAVER LABS Europe are co-organizing a summer school on AI during
2 - 6 July 2018 in Grenoble: https://project.inria.fr/paiss.
The deadline for application is April 4th 2018. You can apply at:
https://project.inria.fr/paiss/application.
Our distinguished speakers include:
  • Lourdes Agapito (UCL)
  • Leon Bottou (Facebook)
  • Kyunghyun Cho (NYU)
  • Emmanuel Dupoux (EHESS)
  • Martial Hebert (CMU)
  • Diane Larlus (NAVER LABS Europe)
  • Hugo Larochelle (Google Brain)
  • Yann LeCun (Facebook / NYU)
  • Julien Mairal (Inria)
  • Julien Perez (NAVER LABS Europe)
  • Jean Ponce (Inria)
  • Cordelia Schmid (Inria)
  • Andrew Zisserman (Oxford / Google DeepMind).

This event is the revival of a past series of very successful summer schools, which took place in Grenoble and Paris (see the 2013 edition here: http://www.di.ens.fr/willow/events/cvml2013). While originally focusing on computer vision, the summer school now targets a broader AI audience.
In addition to the exciting technical program, there will be social events, such as a welcome reception at the Bastille (http://www.bastille-grenoble.fr), and a social event at Naver labs Europe's castle.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Saturday, March 31, 2018

Saturday Morning Video: On Characterizing the Capacity of Neural Networks using Algebraic Topology by William Guss



Here is a video from Microsoft Research by William Guss. I love it because it seems to connect to some of the earlier work we have seen in compressive sensing and related field. ( recently here, or earlier here  or here)


Much like one of the commenter on YouTube, I would have loved less questions during the presentation but it is a fascinating subject. From Guss' website:



The learnability of different neural architectures can be characterized directly by computable measures of data complexity. In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias. After suggesting algebraic topology as a measure for data complexity, we show that the power of a network to express the topological complexity of a dataset in its decision region is a strictly limiting factor in its ability to generalize. We then provide the first empirical characterization of the topological capacity of neural networks. Our empirical analysis shows that at every level of dataset complexity, neural networks exhibit topological phase transitions. This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for fully-connected neural networks.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Friday, March 30, 2018

CfP: OIP'21 : Optics for information processing in the 21th century, 23-26 May 2018 Florence (Italy)


Sylvain let me know of the CfP for this meeting "OIP'21 : Optics for information processing in the 21th century" which will take place in Florence (Italy) on 23-26 May 2018.

Scope of the conference
This very focused workshop aims at bringing together key players in the field to discuss the recent advances in the field of :
  • optical computing
  • computational imaging
  • optical information processing,
  • future challenges and opportunities.
It aims at connecting the communities of optics, physics, mathematics and computer science, in an open atmosphere, with long time for lectures and extended discussions, favoring the exchange of ideas and emergence of possible collaborations.
Confirmed Invited Speakers
  • Alexander Gaeta (Columbia)
  • Alexander Jesacher (Medical U. Innsbruck)
  • Anne Sentenac (CNRS – Institut Fresnel)
  • Christophe Moser (EPFL)
  • David Miller (Stanford)
  • Demetri Psaltis (EPFL)
  • Florent Krzakala  (ENS Paris)
  • Georges Barbastathis (MIT)
  • Jason Fleischer (Princeton)
  • Kelvin Wagner (U.Colorado Boulder)
  • Michal Lipson (Columbia)
  • Monika Ritsch-Marte (Medical U. Innsbruck)
  • Ori Katz (Hebrew University)
  • Riccardo Sapienza (Imperial College)
  • Shaya Fainman (U.C. San Diego)
Registration
Beyond the invited talks, there is a limited number of slots for Contributed posters, preferentially  for  PhD and Postdoctoral researcher. Accomodation in shared room will be provided at the Villa for students and postdocs (on a first come first serve basis). participation to the conference with lodging on your own is of course also possible. 
Registration costs: 
  • full registration single room *invited only* (conference fees, lodging 3 nights in single room,  meals) : 600 euros
  • full registration shared room (conference fees, lodging 3 nights in double or triple room,  meals) : 500 euros
  • registration without lodging (conference fees, meals) : 300 euros
  • accompanying persons (meals) : 125 euros 
Registration is open (payment will be available shortly).
The Venue
The Villa Finaly is situated on the heights above Florence, in beautiful Tuscany, Italy. The villa belongs to the "chancellerie des universités de Paris" which devotes it to organize scientific conference and events. The conference as well as most meals will be taken as the villa, which also counts several rooms to host most of the participants to the conference. Additional participants can easily find accomodation in the numerous hotels of downtown florence, and easily reach the villa by bus.
see the official website of the Villa Finaly
Organizers
Sylvain Gigan - Professor Sorbonne Université Paris  
email: sylvain.gigan@lkb.ens.fr
Rafael Piestun - Professor, University of Colorado, Boulder, USA






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Printfriendly