Thursday, January 7th, 2010

Amazing Audio Sampling in JavaScript with Firefox

Category: Sound

<p>

Above shows the work of some awesome developers. Alistair MacDonald, David Humphrey, Thomas Saunders and Corban Brook have collectively done some great forward looking work with audio sampling and JavaScript. You have to use a custom build of Firefox to make it work right now, but listen to the story of how this came about. Amazing what can happen when projects are open:

  1. First Thomas Saunders got the ball rolling by posting an audio bug with Mozilla and created a neat demonstration of what could be possible if JavaScript could tap into the audio stream. Thomas’s blog is really interesting, if you get a few minutes in the morning over coffee it is well worth a read.
  2. Then David Humprey, a teacher at Open Seneca began hacking away at Firefox to see if he could expose the audio stream to the terminal. The results did not mean much to Dave, but they looked familiar to me. Dave’s results can be seen here: http://vocamus.net/dave/?p=914
  3. It was then my turn. I created a test-tone in Audacity, from which I grabbed a screen shot. This let us see what the wave form should look like. I passed the “.ogg” test-tone to Thomas who posted back his console values as a tidy JavaScript array. I rebuilt these with JavaScript in the Canvas. The images were identical, proving that Dave was right on track with his code. My results can be seen here:

    Reading the Audio Stream from Firefox 3.7

  4. Dave kept things intense, with regular posts on his progress. On new year’s eve -eve, Dave sacrificed punctuality to help me test out his latest code. This was the first time I saw raw audio data streaming directly into JavaScript, a very exciting moment!! Audio Part III, Audio Part IV, Audio Part V.
  5. Corban Brook, stepped onto the scene and threw some FFT magic into the mix. Corban created a simple spectrum analyser, completely JavaScript based. You can just stare at the beautiful video capture here: Firefox JavaScript <Audio> FFT.

Related Content:

Posted by Dion Almaer at 6:46 am
10 Comments

++++-
4.4 rating from 43 votes

10 Comments »

Comments feed TrackBack URI

I bow down in awe! Another ground breaking web innovation!

Comment by BenGerrissen — January 7, 2010

Amazing! Really hope Mozilla will include the patch it into a future version of FF.

Comment by gabel — January 7, 2010

Hey that is really incredible, great work and like the tune too.

Comment by McDaid — January 7, 2010

Getting spectrum data from audio is actually the only reason I’ve used flash recently. If this makes it into FF (and/or other good browsers) I’ll be looking at many many hours of fun with canvas/audio ;D

Comment by rasmusfl0e — January 7, 2010

This patch does not expose spectrum data but the raw audio data.

While it’s nice to see that it is possible to process an FFT in JS in realtime, I strongly believe that it should be part of the Audio and Video APIs.

Comment by p01 — January 7, 2010

Very glad to see this; Flash has had a form of spectrum/eq data access via SoundMixer.computeSpectrum() in Flash 9, and Flash 10 introduced “raw” waveform access, plus the ability to create sound on the fly from code (and manipulate live playback by feeding data into the sound buffer.)
 
I’m a bit surprised Flash didn’t have this stuff say, eight years ago, but in any event it’s good to see they see the value in it. I’ve been looking forward to native audio in the browser for a long time (so eventually I can drop SoundManager 2, which uses Flash under the hood. :D) Being able to get and/or manipulate the sound data opens tons of creative possibilities when used with canvas, etc.

Comment by Schill — January 7, 2010

@p01:

The demo in the video is running an FFT (implemted in js) on a 2048 sample buffer. I was surprised to see that js could handle this in realtime.

David Humphrey’s latest patch (which we are currently testing) implements the FFT in C++ within Firefox and reveals the spectrum data to js. The hope is that this frees up more cycles for animation/other calculations on the js side.

We are currently discussing which features we would like added to the API. If you would like to join in on the discussion, we hang out in #processing.js / irc.mozilla.org

Comment by corbanbrook — January 7, 2010

I’ve just added dynamically generated audio to JSNES which might be of interest:

http://benfirshman.com/projects/jsnes/

And as a side effect, and audio player:

http://benfirshman.com/projects/jstunes/

As soon as you can write samples to elements, I guess the Flash interface can be swapped out with that.

Comment by bfirsh — January 7, 2010

Truly mind-blowing stuff. Some really incredible work guys.

Comment by iliad — January 8, 2010

This is awesome!!

Comment by rdyson — January 8, 2010

Leave a comment

You must be logged in to post a comment.