Here’s an age old conundrum: If a tree falls in a forest, and no one’s around to hear it, does it make a sound? Well, if a video camera was recording the incident, then theoretically, yes. Thanks to an MIT, Microsoft and Adobe joint venture, a nifty processing algorithm has been developed that can extract audio from pixelated vibrations captured in the frames of video.
No ad to show here.
As the accompanying YouTube snippet explains, researchers point a high-speed camera, positioned behind a pane of soundproof glass at a bag of potato chips while a rather dated oration of “Mary had a Little Lamb” is played from nearby speakers.
The sound emitted by the aurator consists of little vibrations representative through minute movements of the potato chip packet. By analysing the video frame by frame, and running it through the algorithm, researchers can more than faithfully recreate the audio.
Essentially, the algorithm passes the frames through analysis filters that record the differences in horizontal and vertical axes, colour transitions and scales.
The practice is not limited to high speed camera units either. Researchers note that even standard happy-snappy cameras recording at 60 frames per second provides enough visual information to extract the representative audio. It is, however, less accurate.
It sounds like the stuff of fiction (a la “zoom and enhance” movie trope, if you will), but it’s largely because it is. Researchers were inspired by Hollywood lore, as Professor Alexei Efros at University of California Berkeley admits:
We’re scientists, and sometimes we watch these movies, like James Bond, and we think, ‘This is Hollywood theatrics. It’s not possible to do that. This is ridiculous.’ And suddenly, there you have it. This is totally out of some Hollywood thriller. You know that the killer has admitted his guilt because there’s surveillance footage of his potato chip bag vibrating.
There are no apparent uses for such intricate technology as yet, but as a study in understanding sound, it’s largely invaluable.
