The Visual Microphone: Passive Recovery of Sound from Video

The research that looks at tiny changes in imaging scenes is revealing to be richer in terms of providing information. Here is the latest paper that is making the rounds on the interwebs. Let us note that similar technology used to require interferometry but that, thanks to Moore 's law through the ever redundant supply of more imaging pixels, we can now use that redundant information for what we used to think as technologies that could not exist. Without futher ado:

The Visual Microphone: Passive Recovery of Sound from Video by Abe Davis Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Frédo Durand, William T. Freeman

When sound hits an object, it causes small vibrations of the object’s surface. We show how, using only high-speed video of the object, we can extract those minute vibrations and partially recover the sound that produced them, allowing us to turn everyday objects—a glass of water, a potted plant, a box of tissues, or a bag of chips—into visual microphones. We recover sounds from highspeed footage of a variety of objects with different properties, and use both real and simulated data to examine some of the factors that affect our ability to visually recover sound. We evaluate the quality of recovered sounds using intelligibility and SNR metrics and provide input and recovered audio samples for direct comparison. We also explore how to leverage the rolling shutter in regular consumer cameras to recover audio from standard frame-rate videos, and use the spatial resolution of our method to visualize how sound-related vibrations vary over an object’s surface, which we can use to recover the vibration modes of an object.

The project page is here.


Of related interest:

• Riesz Pyramids for Fast Phase-Based Video Magnification by Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman
• Analysis and Visualization of Temporal Variations in Video, Michael Rubinstein, PhD Thesis, MIT Feb 2014

Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.