Workflow Evolution

 Web Expanded

Workflow chart from Nick LaMartina, sound designer at Cheyenne Mountain Entertainment

On Richard Garritt''s Tabula Rasa, Tracy Bush, director of audio at NCsoft in Austin, Texas, and team recorded foley-style sound effects using digital recorders running at 96kHz and a Nagra Audio four-track analog deck.

Tracy Bush, director of audio at game developer NCsoft in Austin, Texas, has a trick he uses to get game audio its due, bandwidth-wise, early in the developmental process of titles. “At the beginning of each project, I ask for everything — everything,” he says. “I want speed-of-light sound physics; I want Doppler on everything that moves.” He pauses, then adds, “I never get everything I ask for, but I get enough to make it a decent-sounding game.”

Video games are now bigger than the film business, the industry video-game developers putatively mimic in their workflow: Scripting, preproduction, editing, sound effects, postproduction, and a final layback are milestones in both domains. However, game-audio professionals will point out that games are very much their own animals.

“With movies, you have a locked picture and a linear experience — the viewer experiences that world in exactly the way the director intended it to be viewed,” says Nick LaMartina, sound designer at Cheyenne Mountain Entertainment in Mesa, Ariz. “In games, I compare it to preparing to do a massive live show, and you have to write instructions for an intern about how to react to every possible thing that could happen under any possible circumstance, then flipping a switch and hoping it all goes OK. Game audio has to be ready to deal with any number of possibilities in even the simplest games. That's why a good, logical workflow is important in creating the sound for the game.”

Nick LaMartina, sound designer at Cheyenne Mountain Entertainment in Mesa, Ariz., uses Sony Vegas and Sound Forge audio software for games such as Stargate Worlds (pictured).

LaMartina is currently working on Stargate Worlds, a massively multiplayer online role-playing game (MMOPRG) — which is exponentially more complex than your basic console title, with as many as several thousand players anywhere in the world engaged simultaneously, each confronting hundreds of scenes whose sequence and audio are determined by the course of play. “As a result, the sound image has to be more intricate,” LaMartina says.

In order to achieve that level of detail, the sound workflow starts with the conceptualization of the game itself — another way that game audio differs from film, where audio generally doesn't begin until location recording. The art team developing the visuals will provide the audio team with a list of narrative assets — characters, environments, climates, landscapes, etc. “If it moves or shakes, it's going to need sound,” LaMartina says. His methodology is to poll the other departments involved in the development process and ask them for as much description as they can provide about the assets — especially any movements they will make in the course of play. “From that, you begin to figure out what kind of sounds need to be associated with the assets,” he says.

Marc Schaefgen, worldwide audio director for Midway Home Entertainment and studio audio director for Midway's facility in Austin (the city that has become for game development what Silicon Valley is for computers), says his audio development follows the same track as the overall game structure: a two- to three-month conceptualization period followed by up to six months of prototyping; as many as four months of virtual simulations; then preproduction, production, and post (including authoring) comprising nearly an additional two years.

“It's not like film where the audio comes in closer to the end,” Schaefgen says. “One of the key things that's being worked out in the early stages is: Which type of world is the game and the audio going to live in — realistic or sci-fi? That determines how the sound effects are going to play out.”

The early stages of a game are also the point at which budgets are worked out and when some developers decide if some of the audio will be outsourced, which will also affect workflow in terms of file formats. Music, which is the audio element most often outsourced, has a wide variety of file formats to choose from. Even among Midway Home Entertainment's several facility locations including Austin, Chicago, and Seattle, multiple multitask formats are used — including Steinberg Nuendo, Apple Logic, and MOTU Digital Performer. However, the gold standard remains Digidesign Pro Tools session files, converted to WAV files for mastering and authoring. “We expect our vendors to do their own file-format conversions and deliver Pro Tools files to us,” Schaefgen says.

Ascent Media's game audio division, in Los Angeles, is a prime outsource provider. Sound designer Peter Zinda says that the close and early collaboration that characterizes game development means that files are frequently flowing both ways between Ascent and its clients. “We send a lot of QuickTime movies out that show our sound ideas matched to their pictures,” he says. But most files are Pro Tools sessions, and Zinda says Ascent is transitioning from the FTP transfer mode it has been using to Digidesign's DigiDelivery protocol, which he says is better suited to the Pro Tools files.

The choice of audio platforms is often driven by which functions that audio departments choose to prioritize. Creative Design Network (CDN) NetMix Pro's high degree of compatibility with Pro Tools is what clinched it for Midway, which relies on Pro Tools as a primary digital audio platform; on the other hand, that same criticality of naming tens of thousands of audio tags is why Cheyenne Mountain Entertainment uses Sony Vegas and Sound Forge audio software. “It automates files in a logical manner,” LaMartina says, and it offers other automated functions such as dithering and resampling.

However, live collaboration has to occur at some point in any outsourced scenario, according to Zinda. “Sometimes, a developer will send their audio director to our studios to collaborate with us,” he says. “We have high-end PCs, as well as, for example, Xbox 360 test kits that the audio director can work on. They bring a build of the game with them on a drive and our designers pass sounds to them through our internal network. This allows us to hear sounds in the mix immediately. It speeds up the sound design process, and I think we end up with a higher-quality product. Hearing about how our sounds are working via email is one thing, but actually hearing it in context can be very inspiring.”

Cheyenne Mountain Entertainment uses a combination of Perforce Software''s revision-control system and Firelight Technologies FMOD for audio control.

Large server systems — 4TB and up — are the norm for game developers. While audio will occupy only a portion of that cavernous digital space, it will still represent vast sound libraries as well as being the pool in which departments store and draw assets during development and where those assets are archived. Charles Deenen, senior audio director at the Electronic Arts Black Box development facility in Vancouver, British Columbia, says a single project will take up anywhere from 1TB upwards. “In the last year, with all new content for the next-gen titles, the storage needs were much larger,” he says.

The main workflow issues within the data tanks are management and control. All of Midway's locations use NetMix Pro, a cross-platform sound-library management software that enables users to search, audition, transfer, and manage sound effects and music files across the facilities' shared servers and out into the Internet. “The entire infrastructure between each city is linked by T3 lines, which give us about the same speed as ISDN lines,” says Schaefgen, who says that weapons sounds for both Blacksite: Area 51 and Stranglehold were both shuttled between facilities that way. The searches are keyword-enabled, so Schaefgen says it's critical that a uniform naming protocol for sounds be developed and implemented company-wide.

Both Midway and Cheyenne use Perforce Software's revision-control system, which has emerged as a key element in game-audio workflow. “The ability to mange multiple revisions of the same unit of information is crucial to getting a game done correctly and on time,” LaMartina says. Known generically as version control or source-code management, software such as Perforce registers and manages changes to files that are being used by a team of people by incrementing an associated number or letter code, termed the “revision number,” and associating it historically with the person making the change. Other features of the system include support for notifying users when a file has changed, branching and merging, database checkpoints, and integration with defect-tracking systems. The Perforce system is based on a client/server model with the server managing the collection of source versions in one or more data silos.

“Files can also be checked out like a library book, so only one person has the file at a time, make their edits, then return it to the system, which updates those changes — but, at the same time, never destroys the original,” LaMartina says. “You can always go back to the file you started with.”

Dominating authoring for next-generation game consoles is the multi-platform
Unreal Technology Unreal Engine 3, whose Visual Sound Tool supports all major audio formats.

The asset list that most game-audio developers use as the jumping off point in the quest for matching sounds is a great foundational workflow tool. However, LaMartina says, don't assume it's all you need. “Sound goes well beyond the assets that appear on the screen, and that's where audio really makes its contribution to a game's immersiveness: The ambient sounds and off-screen localization sounds reinforce the world of the game and have a narrative element in that they can foreshadow events to some,” he says.

LaMartina's methodology is to play a section of the game at the level he's designing it. “As if I was walking around the game environment as though it were real, what am I missing that makes it more real?” he says. “You're looking for clues that aren't on the asset list.”

As an example, LaMartina, an avid snowboarder, says he was traversing a virtual winter-world landscape once, and he noticed that the sound that tiny particles of hard snow make when they hit objects was missing. “I added a kind of rustling sound to get that effect, but more importantly, to get that emotional response connection that that sound provides,” he says.

The intensity with which that kind of audio is experienced is now more variable than ever. “Various new middleware applications allow us to let it play back differently according to the circumstances of the game,” LaMartina says. “How hard the snow hits is determined by play, and the resulting sound is affected by filters and level changes. The changes in the audio are behavioral, which greatly enhances the experience. It's much better than just tossing the raw WAV data in there like used to be done.”

Another piece of software that game-audio developers have come to rely on is Firelight Technologies FMOD, a highly compatible cross-platform audio engine, available for the Microsoft Windows, Windows CE, Linux, Apple Macintosh, Nintendo GameCube, Sony PlayStation 2, and Microsoft Xbox platforms. (Part of its allure for the gamer culture is the fact that it's license-free for non-commercial users. And even licenses for commercial applications are quite inexpensive.)

“FMOD gives game developers a much higher level of control over the audio,” LaMartina says, citing its ability to address hundreds of voices simultaneously, creating the kind of matrix of hundreds of variable submixes (which can react to various circumstances in the game) versus the final mix of linear entertainment media.

Once every asset has its sound envelope, and ambient and localization audio has been created, the process moves to authoring, which has lately been dominated by Unreal Technology's Unreal Engine 3, a game-development framework developed for next-generation game consoles — specifically for Microsoft DirectX 9/10 PCs, the Xbox 360, and the PlayStation 3. The engine is multiplatform and comprehensive, covering everything from rendering to animation. Unreal Engine's Visual Sound Tool supports all major audio formats; is 7.1-capable; and can control sound levels, sequencing, looping, filtering, modulation, pitch shift, and randomization.

“Unreal is a scripting environment, and when you combine it with FMOD, the level of control we can achieve over the audio is incredible,” Schaefgen says. “For instance, when a grenade explodes near you, we can automate the ‘flash-bang effect'' [the best example of which is found in the opening moments of Steven Spielberg's Saving Private Ryan, simulating temporary hearing loss and general disorientation]. The sound is low-pass-filtered and a high-pitched whine is introduced. After about 30 seconds, the whine decreases in level and intensity, and the filter ramps off. The combination of FMOD and Unreal Engine 3 lets us do special audio effects like that, and that's really driving how audio is used in games.”

This greater degree of control has brought to the fore some amazing audio nuances. “As a character walks through the world, we attach an audio event to every aspect of its movement — there is a sound for every time its foot hits the ground,” Schaefgen says. “That may have been done before, but now, because the audio is hard-coded into the game instead of being scripted, the audio can change according to what type of surface the foot lands on. The same for bullet impacts — are they hitting hard or soft surfaces, and at what angle are they hitting? The level of nuance is incredible.”

But all this newly attainable realism has an impact on workflow. As Bush had alluded to earlier, audio still has to fight for its share of bandwidth in a game culture that, like film, puts the emphasis on visuals. It also takes more time, and in the process, it can isolate the audio team into its own little bubble. To counter that, LaMartina says Cheyenne uses off-the-shelf tools such as Microsoft SharePoint, a portal-based collaboration and file management platform that can access shared workspaces and files from within a browser. “Every step of the process goes through an approval process between various departments,” he says. “Once each one agrees to a change, they check in through SharePoint, through which every other department can monitor the process, and then it's put into Perforce as a permanent approved document.”

Given how long the game development process is, there are employees at Cheyenne who came on board after a particular title had been started. “This gets them up to speed quickly because they can see the history of the workflow, and the documentation is moving in the same direction as the workflow,” LaMartina says. Another technique to keep everyone on the same page is periodic “milestone tours,” in which producers and department leaders look at each other's work at key points in the continuum.

So game audio is like film sound and then some, and its workflow has to adapt accordingly. Given the plethora of new tools to manipulate audio for games, it has become more complex than a lot of film postproduction, as well. Fortunately, the industry still elicits enormous amounts of passion from its participants, which is probably why Lara Croft moved from the little screen to the big one and not the other way around.

On NCsoft's Dungeon Runners and Richard Garriott's Tabula Rasa titles, much of the sound effects are the equivalent of foley, says Tracy Bush, director of audio. Field recordings are done with both digital recorders running at 96kHz and a Nagra Audio four-track analog deck. “When we're recording gunshots, we'll put a shotgun microphone downrange from the shooter and a kick-drum-type of microphone, like an [Electro-Voice] RE20, at the shooter's feet, and record them to both formats,” he says. “The digital recording gives you tremendous crispness, and the analog recording has a great, fat midrange to it. When you mix them together, it's very dynamic.”
— D.D.

As with linear-based media, sound effects have to make perfect sense to enhance the interactive elements. Sounds easy, right? “Think of a racing game like Need For Speed, where car engine sounds are a major factor,” says Charles Deenen of Electronic Arts. “They have to be loud, big, and proud. So the music has to be crafted carefully around the car engines, and careful frequency banding has to be used in order to make the material work together. You don't want car engines to sound small over music with similar tonality. Same goes for the timing of the elements — music can't get in the way of crashes. A lot of planning goes into what the elements should sound like together versus how they sound individually. It's easy to make smaller-sounding elements work together than making big sounds work well together.”
— D.D.

To comment on this article, email the Digital Content Producer editorial staff at feedback@digitalcontentproducer.com.