Archive for September, 2010

iPhone Audio – Part 1, an exercise in patience

Wednesday, September 8th, 2010

This last week or so has seen head firmly buried in audio code on the iPhone. Well, actually the iPad, but it all comes out the same in the wash.

My inspiration, aside from having done little more with the iPhone other than bleeps and bloops, was the Tenori-on. Now I didn’t know anything about this funky little piece of kit until I saw Jonathan Coulton using one rather expertly at a concert in London a couple of years ago. Apparently it’d been around a while before that, unbeknownst to me. For anyone that hasn’t seen one in action, check out a couple of vids here and here.

Rather than spending a cool grand, I hit up xCode to see just how capable the iPad was at replicating what the Tenori-on could do.

My earlier aural experiences with iPhone have focused solely with OpenAL, simply because it’s easy, it works, and getting it into a game using the plethora of example code and documentation is, to be frank, a breeze. We like OpenAL. The drawback to OpenAL is the lack of control you have over your sound playback. It will do what it wants, when it wants, and how it wants, irrespective of your potentially tight requirements. Fine for a game, not so much for a musical instrument. Besides, at some point there’s going to need to be pitch-shifting involved, so there was always going to be a need access to the sound data itself if I wanted to replicated the Tenori-on’s functionality.

Next up, Audio Queues. Now this is where things start to get interesting. Setting up audio streams and filling output buffers with data manually via callbacks is nothing new, my years of using Microsoft’s DirectSound have gotten me comfortable with such mechanisms. The addition of timeStamps meant that I could set a queue playing, knowing that it would only actually trigger when the timeStamp told it to. Kind of fire-and-forget. Quite neat actually.

The drawback to Audio Queues comes when you want to change the pitch of the notes you’re playing. You can do it when you set a queue up, by simply changing the playback frequency, but only when you set the queue up, not whilst it is in use. So fine, set a queue up, play the note, and delete the queue ready for the next, right? Wrong. Because when you delete an Audio Queue it disrupts the DSP in the iPhone, so if you have any other sounds playing at the time, you get a glitch. The more queues you start and stop, the more glitches you get.

The other issue is, the more queues you create, the slower the playback gets. And by “slower” I do mean “slooowwwweeerrrrr”. A six-note piano chord starts sounding like a guitar strum despite the existence of a timeStamp. Add another couple of notes and you have a significant (in musical realms) amount of time between the first note of a chord and the last. So you have a choice: it is possible to have glitch-less playback but suffer from Audio Queue overload, or you can glitch after every note. Each as unsatisfactory as the other, and both suffer from latency issues when playing back large numbers of voices at a time.

Our only solution lies in the lowest of the low level, Audio Units. Now contrary to popular opinion they’re not actually orders of magnitude more complex than Audio Queues; they still have callbacks and you still have to fill sound buffers manually. The biggest issue with Audio Units is the setting up; they are darned finicky and give little-to-no constructive feedback if you get things out of alignment. Actually, on second thoughts, perhaps I’d better reassess; the biggest issue with AUs is not the setting up, no, the biggest issue with AUs is the considerable lack of clear, concise, descriptive documentation and example code. Admittedly if you want to do exactly what is documented in the way that it has been documented, you’re rocking, kinda. But if you want to step off the beaten track even an inch, and bearing in mind the simplicity of what is documented, you can be sure you’re going to want to at some point, you’re pretty much on your own. Hopefully Apple will make amends soon, but until that time, you’ve got to be brave.

To be continued in “iPhone Audio – Part 2, an exercise in bravery”.

Developers as gaming entrepreneurs?

Friday, September 3rd, 2010

There was a question thrown around recently that made me think about my own situation: why do developers seem to be less interested in a career as gaming entrepreneur?

Having spent the last couple of weeks getting Capita onto the iPad, as well as digging my way through the iPhone Audio Queue mire, I think I kinda understand why. The work that we do is, for the most part, utterly absorbing. People talk about “flow” as being something that developers need in order to produce their best work, and when coding, it really is an intoxicating brew. The inevitable downside of flow however is that you can be starting the week reading about buffer sizes, callbacks and latency, and the next time you look up it’s a week next Tuesday. Very bizarre.

And actually particularly problematic when it comes to anything business-related. If you’re running your own show, more often than not you simply can’t afford to be “somewhere else” mentally for that length of time. There are typically too many things that need doing yesterday, be it contracts to chase, accountants to appease, lawyers to pay, and all are eager to destroy the flow that is so carefully cultivated and so ruthlessly defended.

Now I’m reaching the end of both projects I get a chance to sit up and look around, stretch the legs, and put some words onto the screen more of an English bent, a rather pleasant side note to the developing. But the more time goes by, the more I’m starting to think I’m more of an exception than a rule. When caught up in the rush of development, I just don’t think that, for many coders, much else matters, which is why many are only too happy to avoid branching out and away from the work they love.

And the development community is probably all the richer and more productive for it, although I have to say, right now, I think many of us are wondering where on earth the summer went…