iPhone Audio – Part 1, an exercise in patience

This last week or so has seen head firmly buried in audio code on the iPhone. Well, actually the iPad, but it all comes out the same in the wash.

My inspiration, aside from having done little more with the iPhone other than bleeps and bloops, was the Tenori-on. Now I didn’t know anything about this funky little piece of kit until I saw Jonathan Coulton using one rather expertly at a concert in London a couple of years ago. Apparently it’d been around a while before that, unbeknownst to me. For anyone that hasn’t seen one in action, check out a couple of vids here and here.

Rather than spending a cool grand, I hit up xCode to see just how capable the iPad was at replicating what the Tenori-on could do.

My earlier aural experiences with iPhone have focused solely with OpenAL, simply because it’s easy, it works, and getting it into a game using the plethora of example code and documentation is, to be frank, a breeze. We like OpenAL. The drawback to OpenAL is the lack of control you have over your sound playback. It will do what it wants, when it wants, and how it wants, irrespective of your potentially tight requirements. Fine for a game, not so much for a musical instrument. Besides, at some point there’s going to need to be pitch-shifting involved, so there was always going to be a need access to the sound data itself if I wanted to replicated the Tenori-on’s functionality.

Next up, Audio Queues. Now this is where things start to get interesting. Setting up audio streams and filling output buffers with data manually via callbacks is nothing new, my years of using Microsoft’s DirectSound have gotten me comfortable with such mechanisms. The addition of timeStamps meant that I could set a queue playing, knowing that it would only actually trigger when the timeStamp told it to. Kind of fire-and-forget. Quite neat actually.

The drawback to Audio Queues comes when you want to change the pitch of the notes you’re playing. You can do it when you set a queue up, by simply changing the playback frequency, but only when you set the queue up, not whilst it is in use. So fine, set a queue up, play the note, and delete the queue ready for the next, right? Wrong. Because when you delete an Audio Queue it disrupts the DSP in the iPhone, so if you have any other sounds playing at the time, you get a glitch. The more queues you start and stop, the more glitches you get.

The other issue is, the more queues you create, the slower the playback gets. And by “slower” I do mean “slooowwwweeerrrrr”. A six-note piano chord starts sounding like a guitar strum despite the existence of a timeStamp. Add another couple of notes and you have a significant (in musical realms) amount of time between the first note of a chord and the last. So you have a choice: it is possible to have glitch-less playback but suffer from Audio Queue overload, or you can glitch after every note. Each as unsatisfactory as the other, and both suffer from latency issues when playing back large numbers of voices at a time.

Our only solution lies in the lowest of the low level, Audio Units. Now contrary to popular opinion they’re not actually orders of magnitude more complex than Audio Queues; they still have callbacks and you still have to fill sound buffers manually. The biggest issue with Audio Units is the setting up; they are darned finicky and give little-to-no constructive feedback if you get things out of alignment. Actually, on second thoughts, perhaps I’d better reassess; the biggest issue with AUs is not the setting up, no, the biggest issue with AUs is the considerable lack of clear, concise, descriptive documentation and example code. Admittedly if you want to do exactly what is documented in the way that it has been documented, you’re rocking, kinda. But if you want to step off the beaten track even an inch, and bearing in mind the simplicity of what is documented, you can be sure you’re going to want to at some point, you’re pretty much on your own. Hopefully Apple will make amends soon, but until that time, you’ve got to be brave.

To be continued in “iPhone Audio – Part 2, an exercise in bravery”.

Leave a Reply