Then the obvious question: How do you play back the generated waveform in a browser? A few options exist:
- Use the Mozilla Audio Data API.
- Use the Web Audio API.
- Use the HTML 5 audio element with a fat data URI.
- Support all of the above.
Since option 3 is the only API that’s currently widely supported (and option 4 would make the program larger and more complex) I decided to go with the data URI solution. It basically goes like this:
- Convert the wave data to a string-of-bytes in proper RIFF WAVE format (i.e. prepend a header and ensure little endian byte ordering).
- Convert the WAVE formatted string to base64 encoding using the btoa() function.
- Prepend a data URI header: “data:audio/wav;base64,”
And yes, it works!
What I found out, though, was that the synth ate huge amounts of memory (between 1.5 and 4 GB, depending on browser!). The main reason was that I was using regular ECMAScript arrays for the wave data. There are some differences in how browsers deal with such arrays, but they all have in common that they use more memory per element than the size of the actual data (in ECMAScript, every array element is essentially an object). Add to that the fact that numbers in ECMAScript are treated as doubles (i.e. 8 bytes per number)…
The solution to the problem spells typed arrays. Unfortunately, typed arrays are not that common (yet). The current driving force for typed arrays seems to be WebGL, which is just starting to emerge. But there is one kind of typed array available in browsers that support the canvas element: the CanvasPixelArray.
It may seem like an awkward thing to do, but by using the createImageData(w,h) method (on a 2D canvas context), you get a byte array with w*h*4 elements. This byte array is packed (N array elements cost N bytes of RAM). There were two issues, though:
- I wanted 16-bit sound data, so I had to do manual packing and unpacking of the elements (store one sound sample in two array elements). This required some more code, but on the whole turned out to be faster than regular arrays anyway.
- Some browsers (hrrr-o-mmm-e) don’t support image sizes exceeding 32767 x 32767, meaning that I couldn’t request an N x 1 buffer. The straight forward solution was to use a ceil(sqrt(N)) x ceil(sqrt(N)) buffer. Since the array has linear access, this didn’t introduce any more complexity (just some wasted memory).
You can try out the result for yourself here.
Incredible this thingy is less than 4K. Very well done! I am really impressed! 🙂
MP3 version – http://min.us/mvjoBkQ
had fun with deobfuscation and saving base64 audio data to disk (via PhantomJS and some Ruby code). thanks Marcus and hope to see more articles/source code on synth
Hehe 🙂 Nice job!
This is fantastic work. Would you consider letting others use your ported synthesizer? There does not seem to be anything else out there of this quality yet (especially not with the base64 export support for maximum browser compatibility). Most of the other libs I looked at use the Audio Data API and have serious problems with time sync, popping, etc. I have a project that could really use this, and I would be glad to give you credit if you favor a Creative Commons Attribution license over straight open source.
1) You can implement PADsynth algorithm http://zynaddsubfx.sourceforge.net/doc/PADsynth/PADsynth.htm , which is very simple and it gives you very good sounds
2) You can implement Paulstretch extreme audio stretcher from here http://hypermammut.sourceforge.net/paulstretch/ (or you can see the minimal python implementation here https://github.com/paulnasca/paulstretch_python )
Interesting stuff. Synthesizing in the frequency domain is a really cool technique (you can do almost anything). However, this little demo is more about doing a very minimal synth (in only a couple of hundred lines of code), and I fear that an FFT implementation might be on the heavy side for that. For another project, it might be a good starting point though (I can imagine doing a synth almost completely in the frequency domain).
could we use this technique to create a audio player that protects its source from being simply downloaded ?
Well, you should be able to add some level of “protection”, e.g by using some simple form of encryption/scrambling of e.g and OGG file, send it via XHR to the client, which de-scrambles it and turns it into base64-encoded data URI that is fed to an audio element. It’s not very secure, and it would eat tons of RAM and prevent any form of streaming, but at least you don’t get a pure URL to download. 😉
Amazing stuff! I have been searching for something like this for ages and this is so beautiful. Please release the source code, I’d be really happy to see it.
On the issue of memory and speed on the buffer array. Couldn’t you just use a raw binary data variable and write to it with bitwise shifting? This way you would not need to format your data array for each buffer chunk.
I’d love to see this implemented with more realtime interaction, like a keyboard. But it’s a good song, thx for making this. You rock!
Thanks 🙂 Actually, search around this forum and you’ll find some more updates on the subject. For instance, see this post.
Holy sh17 that’s nice! I’m a big tracker fan.
For something so hacky (by necessity, since js doesn’t have performant arrays yet) your code is really beautiful. Seems so simple once you explain it, but I know I’d be lost if I sat down to port the C code like you did without the benefit of a road map. That’s the definition of good work 😉
To clarify: “good work” is that which makes simple what would seem complicated to *others in general* – not just to *me* 😉
You might want to use Blobs and Object URLs for saving in browsers that support this. I suspect you’ll get less problems with this than with data URLs (URL length/memory). Even if you use data URLs you don’t have to use btoa (it isn’t in any standard). You can use: window.location = “data:application/octet-stream,”+escape(data);
(If data represents text and contains characters > 127 and shall be encoded as UTF-8 use encodeURIComonent instead of escape.)
Simplified it goes like this:
var blob, url, builder = new BlobBuilder();
blob = builder.getBlob(“application/octet-stream”);
url = URL.createObjectURL(blob);
window.open(url, ‘_blank’, ”);
If you like you can do it a bit nicer for Chrome >=14 using an anchor element with an download=”filename” attribute and simulating a click on it (src is the same object URL). This will start a download no matter what mime type the blob has (you don’t need to use “application/octet-stream”) and it won’t flash a new window (what window.open does) and there is no chance of replacing the current document (what window.location = … might do).