Nuit Blanche: Sunday Morning Insight: Zero Knowledge Sensor Design / Data Driven Sensor Design

Sunday, July 13, 2014

Sunday Morning Insight: Zero Knowledge Sensor Design / Data Driven Sensor Design

Out of the many things we can tell from the general field of Machine Learning and from the experiment I mentioned this week [1,2,3] are that:

training sets can become large, very large.
if you want to have a better model, you need even more data

This is enabled through several factors, all of them directly related to Moore's law (one of the steamrollers) and distributed architectures.

Computation time and algorithm development (see Randomness and Parsimony )
Our world, while large, is low dimensional in the higher dimensionality afforded by the number of pixels of every camera (CMOS for imaging).
Memory size ( faster than traditional Moore's law )

In effect, we can gather very large training sets in the lab and we can even learn transfer functions from these sets.

Given all this, let us look at the traditional steps involved in sensor design:

build a sensor with a requirement that it should follow a (preferably) linear interpretation
after that sensor is built and before each new measurement in the field, perform a calibration step
obtain a new measurement.

Step 2 is old fashion, nobody wants to talk about calibration, much less fund it. In fact, it's a little bit like sex: beyond not talking about it, there is the question of where and when and most importantly how much time it takes: For some it may take a minute or less, for others, four hours. The parallel obviously stops here as we all yearn for a quickie, eeerrr.... that did not come out right. Anyway, since we don't talk about it, it's a not an issue in polite society .

What is true though is, the more you insist on, say, your sensor being linear, the more time is spent on the calibration process. Is that fair ? Is this a good use of your time ? especially since your sensor will be recording a world with its own statistics, a world which, while it may seem very high dimensional, is, in fact, pretty low dimensional. Should insisting on a particular shape of the transfer function in sensor design be an actual requirement when we can have access to very large training sets ?

Probably not.

Given all this, Zero Knowledge Sensor Design or Data Driven Sensor Design would be the process by which one build sensors and make up/discover their transfer functions thanks to the supervised learning of very large training datasets.

Let me give an example. Let us imagine the experiment we featured this week and imagine that instead of focusing on getting a linear transfer function, the point of the paper, we would produce a very large training dataset and learn the transfer function of that system with the help of learning algorithms found in Machine Learning or elsewhere. That process would be an instance of Zero Knowledge or Data Driven Sensor Design. Some folks will rightly argue that not much would be known about the model error and attendant noise of such sensor and they would be right. What noise are we witnessing: white noise? pink noise? folded noise ? Think of it differently: we currently spend a lot of time getting data from cameras and spent another inordinate amount of time doing image processing/machine learning to make sense of them. Zero Knowledge or Data Driven Sensor Design might provide a shortcut.

Of note, some of these ideas have been expressed in one form or another in blog entries related to manifold signal processing, task-specific imaging, blind deconvolution or calibration but also on entries related to the JIONC meetings. Phase transitions might also play a role here as they are probably a good way of figuring out if a certain transfer functions are admissible or not see [4-10] for attendant discussions.

Blog entries related to Data Driven Sensor Design can be found at:

http://nuit-blanche.blogspot.com/search/label/DataDrivenSensorDesign

P.S: I was initially thinking about "Data Driven Sensor Design". The wording "Zero Knowledge Sensor Design" came from a discussion with Laurent Daudet who remembered another discussion with Remi Gribonval. Laurent tells me that the wording did not originate with him, so we are actively seeking the origin of that term in order to provide adequate reference (don't hesitate to add anything on the matter in the comment section)

References:

[1] Additional Photos and Video to "Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium"

[2] The Post Publication Peer Review of "Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium"

[3] Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium

[4] Sunday Morning Insight: Randomization is not a dirty word

[5] Sunday Morning Insight: Sharp Phase Transitions in Machine Learning ?
[6] Sunday Morning Insight: Exploring Further the Limits of Admissibility
[7] Sunday Morning Insight: The Map Makers
[8] Sunday Morning Insight: Faster Than a Blink of an Eye
[9] Sunday Morning Insight: A Quick Panorama of Sensing from Direct Imaging to Machine Learning
[10] From Direct Imaging to Machine Learning ... a rapid panorama (JIONC 2014)

[11] Predicting the Future: Randomness and Parsimony
[12] Predicting the Future: The Steamrollers
[13] https://sites.google.com/site/imagingwithnature/

Join the CompressiveSensing subreddit or the Google+ Community and post there !