Thesis: Spectral classification using convolutional neural networks, Pavel Hála - implementation -

I love the approach !

The following is "just" an MS Thesis but I simply found it fascinating for two reasons: Even though the author knows about astrophysics, you can read the thesis and think that he could be foreign to the field. Second, since CNNs are doing well on images,  Pavel does what people have been doing with voice, convert spectra into images. From the thesis:

I was inspired by the way CRT monitors rasterize image on the shadow mask. The output matrices of intensities were normalized using linear normalization and 8-bit grayscale encoding. Finally the product was then converted into an actual PNG image using imagemagick scripting tools, which is the fastest and most reliable approach I have found. The process is illustrated in figures 5.10 and 5.11.

this tells me that hyperspectral imaging is cleary next. Without further ado: Spectral classification using convolutional neural networks by Pavel Hála

There is a great need for accurate and autonomous spectral classification methods in astrophysics. This thesis is about training a convolutional neural network (ConvNet) to recognize an object class (quasar, star or galaxy) from one-dimension spectra only. Author developed several scripts and C programs for datasets preparation, preprocessing and postprocessing of the data. EBLearn library (developed by Pierre Sermanet and Yann LeCun) was used to create ConvNets. Application on dataset of more than 60000 spectra yielded success rate of nearly 95%. This thesis conclusively proved great potential of convolutional neural networks and deep learning methods in astrophysics.  

  The source for Pavel"s code is here: http://physics.muni.cz/~hala/source.tar.gz    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Everybody wants to be shallow: Compact Nonlinear Maps and Circulant Extensions

Everybody wants to be shallow ! Compared to the deep network frenzy, there is still the sense that some sorts of regularization is at play and that a shallow network might hold the key to the right regularization. Here some recent examples featured here: 

• Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network
• Fast, simple and accurate handwritten digit classification using extreme learning machines with shaped input-weights
• Kernel Methods Match Deep Neural Networks on TIMIT
• but also Deep Fried Convnets, Sunday Morning Insight: The Regularization Architecture, Do Deep Nets Really Need to be Deep?

Here is another instance of improving the scalability of  the Random Features approach by choosing better "random projections" so they can be more representative of the dataset being investigated. It slims them up but it ain't doing much with regards to accuracy. I noted from the paper:

6.2 CNM as Neural Networks
One can view the proposed CNM framework from a different angle. If we ignore the original motivation of the work i.e., kernel approximation via Random Fourier Features, the proposed method can be seen as a shallow neural network with one hidden layer, with cos(·) as the activation function, and the SVM objective. It is interesting to note that such a “two-layer neural network”, which simulates certain shift-invariant kernels, leads to very good classification performance as shown in the experimental section. Under the neural network view, one can also use back-propagation as the optimization method, similar to the proposed alternating SGD, or use other types of activation functions such as the sigmoid, and ReLU functions. However the “network” then will no longer correspond to a shift-invariant kernel.

I know I am being provocative here, but "is this really such a bad thing ?" anyway, here is the paper: Compact Nonlinear Maps and Circulant Extensions by Felix X. Yu, Sanjiv Kumar, Henry Rowley, Shih-Fu Chang

Kernel approximation via nonlinear random feature maps is widely used in speeding up kernel machines. There are two main challenges for the conventional kernel approximation methods. First, before performing kernel approximation, a good kernel has to be chosen. Picking a good kernel is a very challenging problem in itself. Second, high-dimensional maps are often required in order to achieve good performance. This leads to high computational cost in both generating the nonlinear maps, and in the subsequent learning and prediction process. In this work, we propose to optimize the nonlinear maps directly with respect to the classification objective in a data-dependent fashion. The proposed approach achieves kernel approximation and kernel learning in a joint framework. This leads to much more compact maps without hurting the performance. As a by-product, the same framework can also be used to achieve more compact kernel maps to approximate a known kernel. We also introduce Circulant Nonlinear Maps, which uses a circulant-structured projection matrix to speed up the nonlinear maps for high-dimensional data.  

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Meeting: missDATA 2015, Rennes, Brittany, France

Julie let me know of the following meeting

  Dear colleagues,

It is my pleasure to announce that the applied mathematics department of Agrocampus Ouest and the French Statistical society organize an international conference on missing values and matrix completion MissData  from the 17 to the 19 June 2015 at Agrocampus Ouest in Rennes. The conference website is:

http://missdata2015.agrocampus-ouest.fr

 
The objective of this conference is to spotlight the latest research in missing values and matrix completion. We aim to bring together theoretical and applied researchers from different fields of application such as in social-science, genetics, medical analysis, survey, collaborative filtering, image reconstruction, etc.

The program comprises R tutorials, invited lectures, ligthning talks and a lively poster session during the reception (with hopefully many French specialties to taste....)! Awards will be given to up to three posters!

We are very proud to have as invited speakers some of the main contributors on the topic from different fields: Mouna Akacha (Novartis, Switzerland), Christophe Biernacki (University Lille 1, France), Stefan van Buuren (University of Utrecht, the Netherlands), John W. Graham (Pennsylvania State University, US), David Haziza, (Montreal University, Canada), Olga Klopp (Paris Ouest University, France), Roderick Little (University of Michigan, US), Patrick Perez (Technicolor), Sandrine Pires (CEA), Joseph L. Schaefer (Census Bureau, Washignton, US), Shaun Seaman (Medical Research Council Biostatistics Unit, UK), Pierre Vandergheynst (EPFL, Switzerland), Recai Yucel (University Albany, US).

Important dates:

Before April 1, 2015: early registration fees – posters and talks submission

We would be glad to have you as a participant or even better as a contributor,
Looking forward to seeing you in brittany (one beautiful part in France...)
The organizing committee.
Email: missdata2015@agrocampus-ouest.fr

 

They even have a Twitter account: @MissData2015

 

Thanks Julie !

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network

It must be the second shallow network featured here recently and all use some sort of random projection techniques for their measurements: Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network by Mark D. McDonnell, Tony Vladusich

We present a neural network architecture and training method designed to enable very rapid training and low implementation complexity. Due to its training speed and very few tunable parameters, the method has strong potential for embedded hardware applications requiring frequent retraining or online training. The approach is characterized by (a) convolutional filters based on biologically inspired visual processing filters, (b) randomly-valued classifier-stage input weights, (c) use of least squares regression to train the classifier output weights in a single batch, and (d) linear classifier-stage output units. We demonstrate the efficacy of the method as an image classifier, obtaining state-of-the-art results on the MNIST (0.37% error) and NORB-small (2.2%) image classification databases, with very fast training times compared to standard deep network approaches. The network's performance on the Google Street View House Number (SVHN) (4%) database is also competitive with state-of-the art methods.  

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Adaptive-Rate Sparse Signal Reconstruction With Application in Compressive Background Subtraction

Using DECOPT, one can envision evaluating the number of measurements for the next video frame by assuming that additional information to the next frame is going to be sparse. This is is what today's paper is looking into: Adaptive-Rate Sparse Signal Reconstruction With Application in Compressive Background Subtraction by Joao F. C. Mota, Nikos Deligiannis, Aswin C. Sankaranarayanan, Volkan Cevher, Miguel R. D. Rodrigues

We propose and analyze an online algorithm for reconstructing a sequence of signals from a limited number of linear measurements. The signals are assumed sparse, with unknown support, and evolve over time according to a generic nonlinear dynamical model. Our algorithm, based on recent theoretical results for $\ell_1$-$\ell_1$ minimization, is recursive and computes the number of measurements to be taken at each time on-the-fly. As an example, we apply the algorithm to compressive video background subtraction, a problem that can be stated as follows: given a set of measurements of a sequence of images with a static background, simultaneously reconstruct each image while separating its foreground from the background. The performance of our method is illustrated on sequences of real images: we observe that it allows a dramatic reduction in the number of measurements with respect to state-of-the-art compressive background subtraction schemes.

    Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

DECOPT - DEcomposition Convex OPTimization - implementation -

Just noticed the following recent solver. From the main page:

DECOPT - DEcomposition Convex OPTimization

Overview

DECOPT is a MATLAB software package for solving the following generic constrained convex optimization problem:

minx∈Rp,r∈Rn{f(x)+g(r):Ax−r=b, l≤x≤u},  (CP)

where f and g are two proper, closed and convex functions, A∈Rn×p, b∈Rn and l,u∈Rp are the lower and upper bounds of x.

Here, we assume that f and g are proximally tractable. By proximal tractability, we mean that the proximal operator proxφ of a given proper, closed and convex function φ:

proxφ(x):=argminy∈Rp{φ(y)+(1/2)∥y−x∥22}

is efficient to compute (e.g., by a closed form solution or by polynomial time algorithms).

DECOPT is implemented by Quoc Tran-Dinh at the Laboratory for Information and Inference Systems (LIONS), EPFL, Lausanne, Switzerland. This is a joint work with Volkan Cevher at LIONS, EPFL.

DECOPT aims at solving (CP) for any convex functions f and g, where their proximal operator is provided. The following special cases have been customized in DECOPT:

• Basis pursuit: 

minx{∥x∥1:Ax=b,l≤x≤u}.

• ℓ1/ℓ2-unconstrained LASSO problem:

minx12∥Ax−b∥22+λ∥x∥1

• ℓ1/ℓ1-convex problem:

minx∥Ax−b∥1+λ∥x∥1.

• Square-root ℓ1/ℓ2 LASSO problem:

minx∥Ax−b∥2+λ∥x∥1.

• ℓ1/ℓ2 - the basis denosing (BPDN) problem:

minx{∥x∥1:∥Ax−b∥2≤δ}.

• ℓ2/ℓ1 - the ℓ1-constrained LASSO problem:

minx{(1/2)∥Ax−b∥22:∥x∥1≤δ}.

Here, λ>0 is a penalty parameter, δ>0 is a given level parameter.

And the references:

References

   The full algorithms and theory of DECOPT can be found in the following references:

• Q. Tran-Dinh and V. Cevher, "Constrained convex minimization via model-based excessive gap". Proceedings of the annual conference on Neural Information Processing Systems Foundation (NIPS), Montreal, Canada, (2014).

Available at: http://infoscience.epfl.ch/record/201842
Abstract: Abstract We introduce a model-based excessive gap technique to analyze first-order primal- dual methods for constrained convex minimization. As a result, we construct first-order primal-dual methods with optimal convergence rates on the primal objec-tive residual and the primal feasibility gap of their iterates separately. Through a dual smoothing and prox- center selection strategy, our framework subsumes the augmented Lagrangian, alternating direction, and dual fast-gradient methods as special cases, where our rates apply.

• Q. Tran-Dinh, and V. Cevher. "A Primal-Dual Algorithmic Framework for Constrained Convex Minimization", LIONS Tech. Report EPFL-REPORT-199844, (2014).

Available at: http://infoscience.epfl.ch/record/199844
Abstract: We present a primal-dual algorithmic framework to obtain approximate solutions to a prototypical constrained convex optimization problem, and rigorously characterize how common structural assumptions affect the numerical efficiency. Our main analysis technique provides a fresh perspective on Nesterov's excessive gap technique in a structured fashion and unifies it with smoothing and primal-dual methods. For instance, through the choices of a dual smoothing strategy and a center point, our framework subsumes decomposition algorithms, augmented Lagrangian as well as the alternating direction method-of-multipliers methods as its special cases, and provides optimal convergence rates on the primal objective residual as well as the primal feasibility gap of the iterates for all.

 

  Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.