Thursday, December 03, 2015

Unlabeled Sensing with Random Linear Measurements

From the introduction:

Unlabeled sensing has potential applications in a number of different fields. Consider the following example. You are blindfolded in a room, and the floor is not flat but a 3 dimensional terrain model. You can sample the height, but you dont know where you take the samples. Is it possible, under some assumption about the terrain model, to recover the location of the samples and the shape of the terrain? This is related to a celebrated problem in robotics called simultaneous location and mapping (SLAM) [8]. Similar data-association problems also arise in the task of assigning observations to targets in multi-target tracking problems that arise in radar applications [9]. More generally, consider the problem of reconstructing a spatial field from samples. Let x denote the representation of the field in some K-dimensional basis. Each measurement can be interpreted as an inner product of x with a “sampling vector” unique to the location where the sample was taken. Consider a mobile sensing scheme [10, 11] where a moving sensor samples the field at N different locations. Further suppose that the mobile sensor does not have access to accurate spatial measurements, although the set of M potential sampling locations and the sampling vectors corresponding to the potential locations are known a priori. The field reconstruction problem one faces in this situation is precisely the unlabeled sensing problem studied in this paper. A similar situation arises in time-domain sampling in the presence of clock jitter [12] which makes it impossible to associate sampled observations to the correct time indices. There is some prior work on reconstruction of bandlimited signals from samples at unknown locations. In [13] an approximate solution to this problem is proposed under the setting of continuous-time measurements and bandlimited signals. In [14], an iterative procedure to reconstruct discrete-time bandlimited signals is proposed. Our work differs from that of these papers in that we do not restrict ourselves to a bandlimited signal model. Our main results are focused on the setting in which the sampling vectors are randomly distributed. In such settings we show that an exact solution to the unlabeled sensing problem is possible when we take twice as many samples as required in classic labeled sensing.
Enjoy the paper:
Unlabeled Sensing with Random Linear Measurements by Jayakrishnan Unnikrishnan, Saeid Haghighatshoar, Martin Vetterli

We study the problem of solving a linear sensing system when the observations are unlabeled. Specifically we seek a solution to a linear system of equations y = Ax when the order of the observations in the vector y is unknown. Focusing on the setting in which A is a random matrix with i.i.d. entries, we show that if the sensing matrix A admits an oversampling ratio of 2 or higher, then with probability 1 it is possible to recover x exactly without the knowledge of the order of the observations in y. Furthermore, if x is of dimension K, then any 2K entries of y are sufficient to recover x. This result implies the existence of deterministic unlabeled sensing matrices with an oversampling factor of 2 that admit perfect reconstruction. The result is universal in that recovery is guaranteed for all possible choices of x. While the proof is constructive, it uses a combinatorial algorithm which is not practical, leaving the question of complexity open. We also analyze a noisy version of the problem and show that local stability is guaranteed by the solution. In particular, for every x, the recovery error tends to zero as the signal-to-noise-ratio tends to infinity. The question of universal stability is unclear. We also obtain a converse of the result in the noiseless case: If the number of observations in y is less than 2K, then with probability 1, universal recovery fails, i.e., with probability 1, there exists distinct choices of x which lead to the same unordered list of observations in y. In terms of applications, the unlabeled sensing problem is related to data association problems encountered in different domains including robotics where it is appears in a method called "simultaneous localization and mapping" (SLAM), multi-target tracking applications, and in sampling signals in the presence of jitter.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, December 02, 2015

Job: PhD studentships and maybe a postdoc, Iowa State University

Namrata just let me know of the following opportunities in her group (for those interested in osting similar job opportunities, do not hesitate to post those on the Compressive Sensing group on LinkedIn with its 3400+ members, also most job entries on Nuit Blanche are collected under the csjob tag)
 

Prof. Namrata Vaswani (http://www.ece.iastate.edu/~namrata/)  Looking for Multiple Ph.D. students for Spring or Fall 2016

Prof. Namrata Vaswani (http://www.ece.iastate.edu/~namrata/)  is looking for multiple Ph.D. students for Spring or Fall 2016. Students with Bachelors or Masters in Electrical or Electrical and Computer Engineering (EE or ECE) or in Mathematics or Applied Mathematics and who have a strong background in linear algebra and probability (undergrad level probability taught in an EE or Math program is enough) are encouraged to apply. Other desirable skills include: (a) the interest and ability to work hard, (b) the ability to think independently, (c) the ability to write well (mathematically).

Her research is in statistical machine learning, data science and signal and information processing. In recent years, her group has worked on developing and analyzing online algorithms for various high-dimensional structured data recovery problems such as online sparse matrix recovery (recursive recovery of sparse vector sequences) or dynamic compressed sensing, online robust principal components' analysis (PCA) and online matrix completion, sparse PCA etc. For more details, see http://www.ece.iastate.edu/~namrata/Summary/index.html

Please email her at namrata@iastate.edu  with the subject line `graduate student applicant'. Please attach a copy of your resume and your transcripts (scanned or unofficial is fine).
Credit: ESA, Rosetta, 1st Earth flyby
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Universality laws for randomized dimension reduction, with applications

It's been mentioned by a few of you, Giuseppe being the first (thanks !)



Universality laws for randomized dimension reduction, with applications by Samet Oymak, Joel A. Tropp

Dimension reduction is the process of embedding high-dimensional data into a lower dimensional space to facilitate its analysis. In the Euclidean setting, one fundamental technique for dimension reduction is to apply a random linear map to the data. This dimension reduction procedure succeeds when it preserves certain geometric features of the set. The question is how large the embedding dimension must be to ensure that randomized dimension reduction succeeds with high probability.
This paper studies a natural family of randomized dimension reduction maps and a large class of data sets. It proves that there is a phase transition in the success probability of the dimension reduction map as the embedding dimension increases. For a given data set, the location of the phase transition is the same for all maps in this family. Furthermore, each map has the same stability properties, as quantified through the restricted minimum singular value. These results can be viewed as new universality laws in high-dimensional stochastic geometry.
Universality laws for randomized dimension reduction have many applications in applied mathematics, signal processing, and statistics. They yield design principles for numerical linear algebra algorithms, for compressed sensing measurement ensembles, and for random linear codes. Furthermore, these results have implications for the performance of statistical estimation methods under a large class of random experimental designs.
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, December 01, 2015

Thesis: High-Dimensional Big Data Processing with Dictionary Learning and Diffusion Maps by Aviv Rotbart

 
Algorithms for modern Big Data analysis deal with both massive amount of samples and a large number of features (high-dimension). One way to cope with these challenges is to assume and discover the existence of localization in the data by uncovering its intrinsic geometry. This approach suggests that different data segments can be analyzed separately and then unified in order to gain an understanding of the whole phenomenon. Methods that utilize efficiently localized data are attractive for high-dimensional big data analysis, because they can be parallelized, and thus the computational resources, which are needed for their utilization, are realistic and affordable. These methods can explore local properties such as intrinsic dimension that vary among different pieces of data. This thesis presents two different methods to locally analyze large datasets for classification, clustering and anomaly detection. The first method localizes dictionary learning based on matrix factorization techniques. We utilize randomized LU decomposition and QR-decomposition algorithms to build dictionaries that describe different types of data. Then, these dictionaries are used to assign new samples to their respective class. One application in cyber security deals with learning of computer files and detecting executable code hidden in PDF files. In a different application, a dictionary learned from a normally behaving computer network data is used to detect anomalies in test data which may imply a cyber threat. 
The second method is localized diffusion process (LDP), which constitutes a coarse-graining of the classic Diffusion Maps algorithm. In LDP, a Markov walk is calculated on small data point clouds instead of the original data points. This work establishes a theoretical foundation for the Localized Diffusion Folders for hierarchical data analysis.
 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Book: Convex Optimization: Algorithms and Complexity, Sébastien Bubeck

Sébastien Bubeck mentioned it on his blog here is his new introduction to convex optimization entitled:  Convex Optimization: Algorithms and Complexity", 

Abstract
This monograph presents the main complexity theorems in convex optimization and their corresponding algorithms. Starting from the fundamental theory of black-box optimization, the material progresses towards recent advances in structural optimization and stochastic optimization. Our presentation of black-box optimization, strongly influenced by Nesterov’s seminal book and Nemirovski’s lecture notes, includes the analysis of cutting plane methods, as well as (accelerated) gradient descent schemes. We also pay special attention to non- Euclidean settings (relevant algorithms include Frank-Wolfe, mirror descent, and dual averaging) and discuss their relevance in machine learning. We provide a gentle introduction to structural optimization with FISTA (to optimize a sum of a smooth and a simple non-smooth term), saddle-point mirror prox (Nemirovski’s alternative to Nesterov’s smoothing), and a concise description of interior point methods. In stochastic optimization we discuss stochastic gradient descent, minibatches, random coordinate descent, and sublinear algorithms. We also briefly touch upon convex relaxation of combinatorial problems and the use of randomness to round solutions, as well as random walks based methods.

 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Sunday, November 29, 2015

Nuit Blanche in Review ( November 2015 )

 Here are the blog entries we had since the last review ( October 2015 )  and some  Hard Questions

  Implementations
Conferences:
In-depth 

ML
Hardware
 
Paris Machine Learning meetup
 
Thesis:
Job:
 

 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, November 27, 2015

Slides and Videos: Randomized Numerical Linear Algebra reading group, Berkeley, Spring 2015

Laura Grigori co-organized with Jim Demmel, Ming Gu, Michael Mahoney, the Randomized Numerical Linear Algebra reading group at Berkeley this past spring. Here are the slides and videos of the presentations that occured then. From her page:
 


Archived video of the lectures may be seen here 
  • Feb 17 - M. Mahoney: Randomized Algorithms for Matrices and Data, slides  
  • Feb 24 - J. Demmel: notes of the lecture , video of the lecture  
  • Mar 3 - J. Demmel, second part of lecture from Feb 24, notes of the lecture , video of the lecture 
  • Mar 10 - M. Gu: Subspace iteration randomization and singular value problems, slides , video of the lecture , arxiv.org 
  • Mar 17: Eric Hallman, based on the paper Sketching as a Tool for Numerical Linear Algebra, Woodruff, video of the lecture 
  • Mar 24 - spring break Mar 31 - Becca Roelofs, video of the lecture based on the paper Blendenpik: Supercharging LAPACK's least-squares solver, Avron et al. 
  • Apr 7 - Arturo Fernandez, slides based on the paper Relative-Error CUR Matrix Decompositions, Drineas et al, and Shivaram Venkataraman, High Performance Linear Algebra using Spark, video of the lecture, lecture starts at 3:59 
  • Apr 14 - Aydin Buluc, based on the paper Low Rank Approximation and Regression in Input Sparsity Time, Clarkson and Woodruff, and Chris Melgaard.video of the lecture 
  • Apr 21: Laura Grigori, CA rank revealing QR and LU factorizations, and Pieter Ghysels, Construction of hierarchically semi-separable matrices using randomized sampling and application in a sparse direct solver, video of the lecture, lecture starts at ~2:44 
  • Apr 28 - Yudong Chen, Fast projected gradient descent algorithms for low-rank estimation video of the lecture, lecture starts at 4:59

Abstract: Fitting a rank-r matrix to noisy data is in general NP-hard. A popular approach is by convex relaxations via nuclear/trace norm minimization. This approach is shown to provide strong (often order-wise unimprovable) statistical guarantees in terms of error bounds and sample complexity. Computationally, while nuclear norm minimization can be solved in polynomial time in principle by semidefinite programming, its time complexity is often too high for large matrices. In this talk, we consider an alternative approach via projected gradient descent over the space of n-by-r matrices, which scales well to large instances. Moreover, we develop a unified framework characterizing the convergence of projected gradient descent for a broad range of non-convex low-rank estimation problems. Our results apply to the problems of matrix sensing, matrix completion, robust PCA, sparse PCA, densest subgraph detection and others, for which we match the best known statistical guarantees provided by convex relaxation methods.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

CSJobs: Postdocs / PhDs / internships opportunities, Saclay, France

Jerome let me know of a few postdoc/PhD/internship opportunities in his lab, here they are:

Stages

Thèses / PhD positions

Post-doc positions


 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, November 25, 2015

Gradual DropIn of Layers to Train Very Deep Neural Networks

Leslie just mentioned to me that of he and his colleague's foray into deep learning. Here it is:

We introduce the concept of dynamically growing a neural network during training. In particular, an untrainable deep network starts as a trainable shallow network and newly added layers are slowly, organically added during training, thereby increasing the network's depth. This is accomplished by a new layer, which we call DropIn. The DropIn layer starts by passing the output from a previous layer (effectively skipping over the newly added layers), then increasingly including units from the new layers for both feedforward and backpropagation. We show that deep networks, which are untrainable with conventional methods, will converge with DropIn layers interspersed in the architecture. In addition, we demonstrate that DropIn provides regularization during training in an analogous way as dropout. Experiments are described with the MNIST dataset and various expanded LeNet architectures, CIFAR-10 dataset with its architecture expanded from 3 to 11 layers, and on the ImageNet dataset with the AlexNet architecture expanded to 13 layers and the VGG 16-layer architecture.
 
 
 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Printfriendly