Meeting: Medical imaging in the era of AI, Yonsei University, Korea

Jong just sent me the following:

Dear Igor,

I would like to bring your attention to upcoming workshop in Korea: "Medical Imaging in the era of AI." This workshop is organized in the form of "Imaging Summit" by inviting many leading researchers in the medical imaging and inverse problems.

http://www.imaging-summit.net/

Date: July 10th, 2017

Place: Grand Ballroom in the Commons , Yonsei University, Korea 

As you know, machine learning techniques have been investigated for various biomedical image reconstruction and inverse problems with encouraging preliminary results. Given the importance of new opportunity of machine learning for image reconstruction, the summit titled "Medical Imaging in the era of AI” is organized to devote to this great convergence between image reconstruction and machine learning.

I believe that many Nuit Blanche readers may be interested in this topic. I appreciate it if you can announce this workshop in Nuit Blanche.

Best,-Jong

Great Jong ! Here is the program.



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday Morning Video: Deep Learning Meets Sparse Coding, Chandra Sekhar Seelamantula

Chandra just sent me the following:

Hi Igor, 

I would like to draw your attention to the following work of ours: 

https://arxiv.org/abs/1705.07290 

The lecture video related to the paper is up on youtube: 

https://www.youtube.com/watch?v=r3q05c1PIRg 

I request you to kindly share these with Nuit Blanche readers, if possible.
Hope you find it interesting.
Thank you so much.
Best regards,
Chandra
==

Chandra, this is great ! Here is the video:

Deep Sparse Coding Using Optimized Linear Expansion of Thresholds by Debabrata Mahapatra, Subhadip Mukherjee, Chandra Sekhar Seelamantula

We address the problem of reconstructing sparse signals from noisy and compressive measurements using a feed-forward deep neural network (DNN) with an architecture motivated by the iterative shrinkage-thresholding algorithm (ISTA). We maintain the weights and biases of the network links as prescribed by ISTA and model the nonlinear activation function using a linear expansion of thresholds (LET), which has been very successful in image denoising and deconvolution. The optimal set of coefficients of the parametrized activation is learned over a training dataset containing measurement-sparse signal pairs, corresponding to a fixed sensing matrix. For training, we develop an efficient second-order algorithm, which requires only matrix-vector product computations in every training epoch (Hessian-free optimization) and offers superior convergence performance than gradient-descent optimization. Subsequently, we derive an improved network architecture inspired by FISTA, a faster version of ISTA, to achieve similar signal estimation performance with about 50% of the number of layers. The resulting architecture turns out to be a deep residual network, which has recently been shown to exhibit superior performance in several visual recognition tasks. Numerical experiments demonstrate that the proposed DNN architectures lead to 3 to 4 dB improvement in the reconstruction signal-to-noise ratio (SNR), compared with the state-of-the-art sparse coding algorithms.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

CSJob: PhD position “Tensors for System Identification” at ELEC, Vrije Universiteit Brussel (VUB)

Philippe just sent me the following: 

Dear Igor,

We have an open PhD position at Vrije Universiteit Brussel, Belgium, see below. We believe the topic is relevant to your readers, as it is in the context of advanced matrix and tensor factorizations. Could you please consider posting it to Nuit Blanche?

Kind regards,

Philippe

Sure Philippe !, Here is the announcement:

PhD position “Tensors for System Identification” at ELEC, Vrije Universiteit Brussel (VUB)




A fully funded PhD position is now available at Department ELEC, Vrije Universiteit Brussel (VUB). The goal is to develop novel methods for nonlinear system identification using tensors. We will exploit the Volterra representation while aiming for interpretable block-oriented models. Tensors play a central role in understanding and taking advantage of the Volterra kernels. See our paper http://homepages.vub.ac.be/~mishteva/papers/voltpWH.pdf





We offer an attractive salary, a job in the heart of Europe, and support from a renowned research group. This is a four year PhD project, with a yearly evaluated and renewable contract. The preferred starting date is as soon as possible and no later than Oct 1, 2017.


The specific PhD topic can be adapted to the interests of the applicant. It is especially suitable for mathematicians interested in engineering applications, and for engineers interested in mathematics.


Requirements: Master’s degree in (Applied) Mathematics, Electrical Engineering, Physics, Computer Science, or a related domain. Further requirements include excellent programming skills (e.g., MATLAB) and excellent English language skills. Experience in tensor methods and system identification is an advantage, but is not required.


Please send a two-page CV and a one-page personal statement (motivation and background knowledge) as a single PDF document to philippe.dreesen@vub.ac.be. Mention “VOLT-FWO-2017N” in the email subject line. Applications received before July 15, 2017 will be given full consideration. Informal inquiries can be sent to the same email address.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Training Quantized Nets: A Deeper Understanding

Ah, here is some insight Christoph, Tom et al. !

Training Quantized Nets: A Deeper Understanding by Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, Tom Goldstein

Currently, deep neural networks are deployed on low-power embedded devices by first training a full-precision model using powerful computing hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized network, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of algorithms for non-convex problems, and we show that training algorithms that exploit high-precision representations have an important annealing property that purely quantized training methods lack, which explains many of the observed empirical differences between these types of algorithms.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Coreset Construction via Randomized Matrix Multiplication

Coreset Construction via Randomized Matrix Multiplication by Jiasen Yang, Agniva Chowdhury, Petros Drineas

Coresets are small sets of points that approximate the properties of a larger point-set. For example, given a compact set \mathcal{S} \subseteq \mathbb{R}^d, a coreset could be defined as a (weighted) subset of \mathcal{S} that approximates the sum of squared distances from \mathcal{S} to every linear subspace of \mathbb{R}^d. As such, coresets can be used as a proxy to the full dataset and provide an important technique to speed up algorithms for solving problems including principal component analysis, latent semantic indexing, etc. In this paper, we provide a structural result that connects the construction of such coresets to approximating matrix products. This structural result implies a simple, randomized algorithm that constructs coresets whose sizes are independent of the number and dimensionality of the input points. The expected size of the resulting coresets yields an improvement over the state-of-the-art deterministic approach. Finally, we evaluate the proposed randomized algorithm on synthetic and real data, and demonstrate its effective performance relative to its deterministic counterpart.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Hyperparameter Optimization: A Spectral Approach

From the introduction

In this paper we introduce a new spectral approach to hyperparameter optimization based on harmonic analysis of Boolean functions. At a high level, the idea is to fit a sparse polynomial function to the discrete, high-dimensional function mapping hyperparameters to loss, and then optimize the resulting sparse polynomial. Using ideas from discrete Fourier analysis and compressed sensing, we can give provable guarantees for a sparse-recovery algorithm that admits an efficient, paralellizable implementation. Here we are concerned with the tradeoff between running time and sample complexity for learning Boolean functions f where sampling uniformly from f is very expensive. This approach appears to be new and allows us to give uniform-distribution learning algorithms for Boolean concept classes such as decision trees that match the state-of-the-art in running time and save dramatically in sample complexity.

Hyperparameter Optimization: A Spectral Approach by Elad Hazan, Adam Klivans, Yang Yuan

We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions. We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm - an iterative application of compressed sensing techniques for orthogonal polynomials - requires only uniform sampling of the hyperparameters and is thus easily parallelizable. Experiments for training deep nets on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases matching what is attainable by hand-tuning. In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and even more so compared to Bayesian Optimization. We also outperform Random Search 5X. Additionally, our method comes with provable guarantees and yields the first quasi-polynomial time algorithm for learning decision trees under the uniform distribution with polynomial sample complexity, the first improvement in over two decades.

h/t François on Twitter

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Kronecker Recurrent Units

Kronecker Recurrent Units by Cijo Jose, Moustpaha Cisse, Francois Fleuret

Our work addresses two important issues with recurrent neural networks: (1) they are over-parameterized, and (2) the recurrence matrix is ill-conditioned. The former increases the sample complexity of learning and the training time. The latter causes the vanishing and exploding gradient problem. We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU). KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix. It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors. Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient. Our experimental results on five standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance. These results in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Book:Artificial Intelligence and Games by Georgios N. Yannakakis and Julian Togelius

From the website:

Welcome to the Artificial Intelligence and Games book. This book aims to be the first comprehensive textbook on the application and use of artificial intelligence (AI) in, and for, games. Our hope is that the book will be used by educators and students of graduate or advanced undergraduate courses on game AI as well as game AI practitioners at large.
First Public Draft
The first draft of the book is available here!
If you spot any typos or inaccurate information, disagree with parts of the text or you have suggestions for papers we should discuss or exercises (and readings) we should include please contact us via email at gameaibook [ at ] gmail [ dot ] com.
We would appreciate your feedback on this first draft by no later than June 20, 2017 so that we meet the publication deadlines.

Here is the introduction of Artificial Intelligence and Games by Georgios N. Yannakakis and Julian Togelius

Introduction 
Artificial Intelligence (AI) has seen an immense progress in recent years. This progress is a result of a vibrant and thriving research field that features an increasing number of important research areas. The success stories of AI can be experienced in our daily lives and also evidenced though its many practical applications. AI nowadays can understand images and speech, detect emotion, drive cars, search the web, support creative design, and play games, among many other tasks; for some of these tasks machines have reached human-level status. In addition to the algorithmic innovation, the progress is often attributed to increasing computational power or to hardware advancements. There is, however, a difference between what machines can do well and what humans are good at. In the early days of AI we envisaged computational systems that deliver aspects of human intelligence and achieve humanlevel problem solving or decision making skills. While these problems can be difficult for most of us they were presented to machines as a set of formal mathematical notions within rather narrow and controlled spaces. The properties of these domains collectively allowed AI to succeed. Naturally, games—especially board games—have been a popular domain for early AI attempts as they are formal and highly constrained yet complex decision making environments. Over the years the focus of much AI research has shifted to tasks that appear simple for us to do, such as remembering a face or recognizing our friend’s voice over the phone. AI researchers have been asking questions such as: How can AI detect and express emotion? How can AI educate people, be creative or artistically novel? How can AI play a game it has not seen before? How can AI learn from minimal amount of trials? How can AI feel guilt?. All these questions pose serious challenges to AI and correspond to tasks that are not easy for us to formalize or define objectively. Unsurprisingly, tasks that require relatively low cognitive effort from us often turn out to be much harder for machines to tackle. Again, games have provided a popular domain to tackle such tasks as they feature aspects of subjective nature that cannot be formalized easily. These include, for instance, the experience of play or the creative process of game design. Games have been helping AI to grow and advance since its birth. Games not only pose interesting and complex problems for AI to solve—e.g. playing a game well; they also offer a canvas for creativity and expression which is experienced by users (people or even machines!). Thus, arguably, games is a rare domain where science (problem solving) meets art and interaction: these ingredients traditionally made games a unique and favorite domain for the study of AI. But it is not only AI that is advanced through games; it is also games that are advanced through AI research. We argue that AI has been helping games to get better in several fronts: in the way we play them, in the way we understand their inner functionalities, in the way we design them, in the way we understand play, interaction and creativity. This book is dedicated to the healthy relationship between games and AI and the numerous ways both games and AI have been challenged, but nevertheless, advanced through this relationship.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

DiracNets: Training Very Deep Neural Networks Without Skip-Connections - implementation -

DiracNets: Training Very Deep Neural Networks Without Skip-Connections by Sergey Zagoruyko, Nikos Komodakis

Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference. We're able to achieve 95.5% accuracy on CIFAR-10 with 34-layer deep plain network, surpassing 1001-layer deep ResNet, and approaching Wide ResNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at this https URL

The implementation is here: https://github.com/szagoruyko/diracnets
h/t Iacopo


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Inexact Gradient Projection and Fast Data Driven Compressed Sensing

Reconstructing from sketches.

Inexact Gradient Projection and Fast Data Driven Compressed Sensing by Mohammad Golbabaee, Mike E. Davies

We study convergence of the iterative projected gradient (IPG) algorithm for arbitrary (possibly nonconvex) sets and when both the gradient and projection oracles are computed approximately. We consider different notions of approximation of which we show that the Progressive Fixed Precision (PFP) and the (1+ϵ)-optimal oracles can achieve the same accuracy as for the exact IPG algorithm. We show that the former scheme is also able to maintain the (linear) rate of convergence of the exact algorithm, under the same embedding assumption. In contrast, the (1+ϵ)-approximate oracle requires a stronger embedding condition, moderate compression ratios and it typically slows down the convergence. We apply our results to accelerate solving a class of data driven compressed sensing problems, where we replace iterative exhaustive searches over large datasets by fast approximate nearest neighbour search strategies based on the cover tree data structure. For datasets with low intrinsic dimensions our proposed algorithm achieves a complexity logarithmic in terms of the dataset population as opposed to the linear complexity of a brute force search. By running several numerical experiments we conclude similar observations as predicted by our theoretical analysis.

h/t Laurent Jacques


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.