Sound and Music

A Better Picture Through Sound

By Jonathan Hoffberg
August 1, 1997
Vol. 1: Issue 3

Several years ago, a study on audience research was done at MIT to determine the effects of sound quality on the perception of picture quality. A high-definition television (HDTV), with approximately 1000 lines of vertical resolution, was placed in a booth, with a low-fidelity sound system. In another booth, a regular television with approximately half the vertical resolution, was outfitted with a CD-quality sound system. Participants in the study were shown the same content on each screen, and asked to identify which picture looked better. The participants chose the NTSC screen, with half the resolution of the HDTV screen, overwhelmingly, over the HDTV screen.

But Games Aren't Movies!

"OK, OK, movies need sound," my producer says to me, "but games are about the gameplay." Absolutely. As an industry, those of us who didn't know that going in learned the hard way. But if sound has this big an effect on a TV, couldn't it still improve the gameplay? A good sound track for a game can do everything that a good sound track for a movie can do, and more. Because sound in a game serves one, even more compelling function. The game sound track provides the player with constant feedback on his actions and the game state.

The Deadly Loop
How many people put their favorite song on repeat on the stereo? Why would we do the same thing with music for a game?

There is an often unchallenged assumption in video game production that there must be music everywhere, all of the time. Given this unspoken directive, the composer goes about creating a series of loops that will blanket the world with sound. Since the game creators don't usually know how often a player will be in any given location, the composer is instructed to create loops for each section which will allow the flexibility to provide players of all abilities with a full sound track until they complete that particular portion of the game.

And so it goes that the best laid tracks are doomed to a silent death. After 3 or 4 loops of a piece of music, a player would just as soon rip the speakers out of his or her computer as have to listen to that infernal tune one more time.

`The Coolest Sound Ever'
Here's a classic sound effects pitfall. You're designing the sound for firing the Zirconium Phase Cannon. At this point in the game, the mood is supposed to be dark, because you're the only sentient being that stands between the Supercharged Blorks and global destruction. You get one sound for when it hits a Supercharged Blork, one sound for a miss, etc.. `Can't wait to get started!

You start with the one where the Cannon hits an invading Blork. After several hours of experimentation, layering, and a recording session with your cat that would make the ASPCA wince, you've finally got it.... the baddest alien death sound ever created.

You send it off to the programmer, who puts it on a Zip disk somewhere where it languishes in silence, until things really get out of hand, and the producer inquires about the sound. Two weeks before the game goes to master, the programmer finally puts in the sound, and the game plays it every eight seconds. What started out as drama becomes comedy, as the same, gut-wrenching scream, played over and over again, settles into some kind of twisted groove from the Depths of Hell.

Once the player rips the speakers out, all is lost.

What Can Be Done?
The first order of business is to make sure that the player doesn't turn off the audio. One way of doing that is to pay real attention to how many repetitions any given noise or piece of music can endure before losing its power..

Generally speaking, the more unusual an effect, the fewer times it can be heard before it loses its dramatic punch and becomes comical. Often, if a sound is more subtle, it will provide the necessary player feedback without making teeth grind. What's even better is if the sound can be altered subtlely each time it's played. Sound spatialization routines can assist in this, but when changes in the game state can modulate sounds, or cause different sounds to play given the same player actions, the effects track gains a whole new dimension.

It's so rare that the sound of a player firing a bullet changes, depending on whether it's the first bullet in his clip or the last. But merely playing back the same tiny sound at slightly different pitches can produce this effect on the fly. The effect is a win-win situation. The gun sounds different every time it's fired, and the player has improved feedback on his battle capability. All without any sacrifice in memory. Several commercially available sound engines allow this basic type of sound manipulation, yet few sound designers and game programmers take full advantage of it.

As for music, why loop? If players are willing to turn the sound off, why not beat them to the punch by giving them a little more peace and quiet. You can get ten times the mileage out of three minutes of music if you break it up into small chunks, and play them back in a variety of different orders, and then fade them in and out. But since this involves close collaboration between the composer and development team, it is an option that is frequently overlooked.

Anticipating the Logjam
All too often, the composer and sound designer never really hear their material integrated in the game until it's too late to do anything about the ensuing din. The sheer quantity of programming tasks is quite often the bottleneck in most game production environments, and the notion of creating new sound intelligence or fancy alternatives to looping at the time that sound is integrated has all the airworthiness of an Egyptian pyramid.

Starting Early
There are now several interactive sound engines on the market that offer more sophisticated methods of sound implementation, and getting one of these libraries integrated into the game engine at the beginning of the development cycle can make things a lot easier.

Once an engine is settled on, and its capabilities are addressed, the composer and sound designer need to become familiar with its tricks. How many sounds can it play at once? Can multiple sounds be triggered at the same time, with enough accuracy to allow multitracking of digital music clips? If the game is a location-based system, or the application is targeted at an audience with high-end sound cards, can you take advantage of the on-board samplers on those sound cards to get the extra flexibility of MIDI with the superior instrumentation of a real sampler?

Also, booking time at regular intervals in the development for effects and music integration allows the composer and sound designer invaluable feedback as to how their material can be modified in order to better fit the game. Incorporating this strategy into the production cycle goes a long way towards a better integrated sound track.

If these issues are ironed out before the crunch sets in, the sound content can be created accordingly, the integration process will be much smoother and more sophisticated, and the result will be an aural experience which informs and engages the player.

Jonathan Hoffberg is a freelance sound track producer based in Berkeley, California. His composition and sound design have appeared in games published by LucasArts, Spectrum-Holobyte, Maxis, and IBM. He can be reached via email at [email protected].