CS: Promoting a Larger Set of Compressive Sensing Benchmarks

Following up on Friday's entry on Defining the Current State of the Art, I have already sent an e-mail out to some of you requesting that you share some of your examples or actual datasets directly with the rest of the community. The intent to gather different sets stems from the following:

• The field is getting large by the day and we can't afford to rely only on Lena's image as has been the case in signal processing for the past few decades.
• We can't afford to have too many reconstruction solvers with unchecked claims. By unchecked, I mean that the solvers cannot be judged on one dataset chosen by the solver's authors, we need to know how badly the solvers are doing for certain datasets. or for certain parameters We need to know in which part of the phase space certain solvers are better than others in either the quality of the reconstruction or the time it took to get there.
  
• We have new hardware architectures that can only be revealed to perform well only thanks to new reconstruction solvers. Unlike the old days where the two tasks were disjoint, the quality of a solver can now have a direct consequence on the hardware. We need to have fewer artificial data and more real data to evaluate these new architectures.

Currently, two research teams have responded (I have not sent that many requests out):

• Rich Baraniuk and Marco Duarte at Rice reminded me that their data from the single pixel camera is located here. I look forward to more data in the future from the other declination of the single pixel camera.
  
• Andriyan Suksmono and his group at IT Bandung will be making available data from their data from the Compressive Sensing Ground Penetrating Radar in the coming weeks.

Michael Friedlander told me that he would eventually add those to the current set of benchmark problems in Sparco. But I am sure that if you want to help Michael by coding directly your problem as a Sparco problem, he won't mind integrating it directly into the current list. I have mentioned it before but I have no stakes in Sparco but want to remind everybody of its nice features with regards to the implementation of masks which allows users to focus on other aspect of the CS problems such as the solvers or even enable a parametric play of said encodings.

At some point, we also need to have a nice set of dictionaries accessible to different Operating Systems, when running on my windows box, it seems that I cannot use the curvelets used in the seismic examples.

Initially, the goal of this work is to help newcomers to evaluate solvers more rapidily for specific applications but I also think that it should give us all a sense of where future work should go especially in those cases where reconstruction is still not possible today because the dataset are too large (movies, voice, etc....)

References:
[1] SPARCO: A toolbox for testing sparse reconstruction algorithms

Credit Photo: Rice University, CS Camera.