Learning the structure of learning

If anything, there has been a flurry of effort in learning the structure of new learning architectures. Here is an ICLR2017 paper on the subject of meta learning and posters of the recent NIPS symposium on the topic.

Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc Le (Open Review is here)

Abstract: Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.

At NIPS, we had the Symposium on Recurrent Neural Networks and Other Machine that Learns Algorithms

• Jürgen Schmidhuber, Introduction to Recurrent Neural Networks and Other Machines that Learn Algorithms
• Paul Werbos, Deep Learning in Recurrent Networks: From Basics To New Data on the Brain
• Li Deng, Three Cool Topics on RNN
• Risto Miikkulainen, Scaling Up Deep Learning through Neuroevolution
• Jason Weston, New Tasks and Architectures for Language Understanding and Dialogue with Memory
• Oriol Vinyals, Recurrent Nets Frontiers
• Mike Mozer, Neural Hawkes Process Memories
• Ilya Sutskever, Using a slow RL algorithm to learn a fast RL algorithm using recurrent neural networks (Arxiv)
• Marcus Hutter, Asymptotically fastest solver of all well-defined problems
• Nando de Freitas , Learning to Learn, to Program, to Explore and to Seek Knowledge (Video)
• Alex Graves, Differentiable Neural Computer
• Nal Kalchbrenner, Generative Modeling as Sequence Learning
• Panel Discussion Topic: The future of machines that learn algorithms, Panelists: Ilya Sutskever, Jürgen Schmidhuber, Li Deng, Paul Werbos, Risto Miikkulainen, Sepp Hochreiter, Moderator: Alex Graves

Posters of the recent NIPS2016 workshop

• Real-time interactive sequence generation and control with Recurrent Neural Network ensembles  by Memo Akten and Mick Grierson
• A Neural Forth Abstract Machine by Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky and Sebastian Riedel
• Log-Linear RNNs : Towards Recurrent Neural Networks with Flexible Prior Knowledge by Marc Dymetman and Chunyang Xiao
• Similarity-based LSTMs for Time Series Representation Learning in the Presence of Structured Covariates by Madalina Fiterau, Jason Fries, Eni Halilaj, Nopphon Siranart, Suvrat Bhooshan and Christopher Ré
• Neural Machine Translation with Characters and Hierarchical Encoding by Alexander Rosenberg Johansen, Jonas Meinertz Hansen, Elias Khazen Obeid, Casper Kaae Sønderby and Ole Winther
• Supervised learning with information penalties by Artemy Kolchinsky and David H. Wolpert
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations by David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville and Christopher Pal
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision  by Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus and Ni Lao
• Recurrent Highway Networks by Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník and Jürgen Schmidhuber

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.