AI Armies

Animating Intelligent Characters for the TwoTowers

In Peter Jackson’s The Two Towers, CG soldiers, created usingWETA’s Massive software and rendered in Grunt, react to theirenvironment with patterned responses based on AI technology.

It's a scene that likely exceeds the wildest fantasies of Cecil B.DeMille: 70,000 soldiers marching across the screen in director PeterJackson's The Two Towers, the second installment in New Line'sfilm trilogy The Lord Of The Rings. These aren't cut-and-pasteclips of one photographed regiment used over and over again. They'redigitally animated individuals who run, climb, fight, and die —driven by software based on principles of artificial intelligence(AI).

Crafted at Jackson's WETA Digital facility in Wellington, NewZealand, these CG hordes were wrangled through programs called Massiveand Grunt (General Renderer of Unlimited Numbers of Things). Thissoftware, developed over several years by Steven Regelous and JohnAlitt, has enabled WETA to produce animation of unprecedented scope.“We could tell 10,000 ‘guys’ to stand on one side ofthe battlefield, facing another 10,000 guys wearing red armor,”says The Two Towers visual effects supervisor Jim Rygiel, whowon the 2001 Oscar for supervising the effects in The Fellowship OfThe Ring “We'd then tell the first set of guys, ‘Whenyou see anyone in red armor, run toward them and killthem.’”

That these individuals were programmed to “see” and“hear” makes WETA's approach more powerful than what's beendone before, believes Rygiel. A veteran CG practitioner, Rygiel countsamong his assignments the computer-animated penguins created at BossFilm for 1992's Batman Returns. “Even though that was calledartificial intelligence, what they had was collision detection.Each character ‘knew’ that if its center of gravity wasgetting close to another creature's center of gravity, it would have toturn away. We gave it the random choice of going left or right. Therewere lots of random decisions going on at that point.” Rygiel'swork was an offshoot of the research presented by Craig Reynolds in1986 on flocking birds, which influenced many motion-pictureanimators.

AI-driven characters have advanced greatly since then.

“There's been a thousand-fold leap in this technology from thefirst Lord Of The Rings film to the second one,” saysRygiel. “Before, we mostly showed these characters at a distancebecause they barely would have held up in the video game world. Now thecamera shows almost full head shots of some of these guys.”

One of the most dramatic examples is a battle at Helms Deep, where10,000 soldiers march upon the fortress, throw ladders against it, andscale the walls, all while being pelted by arrows. To achieve suchcomplex animation, the WETA team began by motion-capturing numerousbehaviors, using mo-cap technology set up in Wellington byAtlanta-based Giant Studios.

The team amassed libraries of motion information. “We'd thenteach our AI system what we wanted the characters to do,”explains Rygiel. “For example, the [AI software] brain says to acharacter, ‘You're a climber. When the ladder goes up, you runtoward it and start climbing as fast as you can. When you get to thetop, jump off — using the various motion cycles that we've givenyou.’ There are also variations on this theme. One guy will dohis run-to-ladder cycle B, then go into a climb-ladder cycle A, andthen use jump cycle C. What we can do on top of that is take thesedifferent cycles and add a hyper-random factor, where we can change thelength of their limbs and their gaits. So ten cycles can become acouple of thousand cycles. We can also alter how fast or slow they'regoing, so guys who are running fast can avoid the slower guys and passthem.”

This kind of variability is essential “because if you tookeven 100 cycles and put them over 10,000 guys,” says Rygiel,“they would look exactly the same. The key is gettingextra randomness in there.” The Massive simulation softwarepermits animators to go in and tweak the animation as necessary.“For instance, typical dailies might show 10,000 guys runningacross a field, “says Rygiel. “The first time we'd see it,they'd all be running at exactly the same speed, perfectly placed inrelation to each other. So we'd say, ‘We need more variation inthe groupings. Let's have a cluster of 30 over here, and in thatcluster, half of them are slowpokes.’ Of course, out of 10,000guys, we'd have some brain-dead ones that actually fell off the cliffinstead of avoiding it. That happened all the time. Then we would say,‘Don't have that one show up for work tomorrow.’ We'dliterally kill that character out of the sim!” Usually, Rygielcould see changes by the following day. The ability to render theimages quickly is what makes such simulation workable on afilm-production schedule. “The Grunt renderer for Massive isreally fast, and it's made for doing these duplication simulations. Itallows us to assign different levels of detail, depending upon wherecharacters are in relation to camera.”

WETA typically models its CG with Alias/Wavefront's Maya and renderswith Pixar's RenderMan, using primarily Intel-based computers runningRed Hat Linux. But standard practices wouldn't have worked forefficiently rendering these huge armies. “If we did a sim andrendered it in RenderMan it would take three weeks — if we couldhave rendered it at all,” says Rygiel. “Imagine tellingpeople that you want to render 70,000 things in RenderMan. Their mindswould explode.”

Not surprisingly, word of WETA's approach has the industry talking,especially about where we go from here. “Software that enablescharacters to react to their environment with patterned responses isthe only solution for this kind of work,” says Eric Armstrong,Sony Imageworks' animation director on Stuart Little 2.“You can't have animators keyframe 10,000 characters. It can't bedone.”

Armstrong has a personal appreciation of complex animationchallenges since he previously oversaw ILM's galloping herd of Gallidinosaurs in 1993's Jurassic Park. He has strong opinions aboutintelligent software technology's impact on the industry in years tocome. “Proceduralized characters that think and emote and solveproblems are eventually going to become the norm,” Armstrongsays. “You'll have a system where you'll be able to say, ‘Iwant a character of this height and this body type,’ and it willappear on the screen. Then you may say, ‘Make the waist a littlethinner and the shoulders a little broader, and give him glasses. Nope,make them horn-rimmed glasses.’

“I don't think we're going to replace real actors with digitalactors. It's more a question of whether the Bugs Bunny of 2020 will bekeyframed. I don't think there's any way that it will be.”

When Armstrong sketched this scenario at the 2002 SIGGRAPH panel“Animation's Turning Tide,” he braced himself for a hostilereaction. “I know that some people view this as an evilevolution,” he says. “That it gets rid of the creativity inthe animation process. The truth is, it will make it easier foranimators in the beginning. But eventually it will lead to the pointwhere animators, as we know them today, will become superfluous. Myfeeling — and it's not a statement as to whether I think this isgood or bad — is that this will happen within 20 years. There maybe some people still doing keyframe work for special-case CG projectsor cel animation. But the majority of characters will be controlled bydirectors talking into microphones, giving commands like they would onset. It will be a realtime simulation, rendered in high res.

Digital characters are migrating from games to the film andtelevision industry. BioGraphic Technologies’ new commercialsoftware AI.implant tracks CG characters and gives animators feedbackin realtime.

“I'm not saying all animators will be gone.” saysArmstrong, “But instead of having an army of 25 people lightingone shot at a time, you'll have one guy — like a cinematographer— who'll set up the lights for a shot. Then the director givesthe commands and the simulation runs. If they decide it would lookbetter if they moved the lights, they'll run the simulation again withdifferent lighting. There will be three or four people doing thisinstead of 150.”

Much to Armstrong's surprise, no one at SIGGRAPH challenged thisscenario. His fellow panelists said they offered no rebuttal becausethey didn't really disagree. Armstrong thinks the success of AI-drivencharacters in videogames has opened a lot of minds in the film industryto the possibilities for interactive control of virtual characters.

“Game people are the ones who are driving it. They haveto do this to be competitive. But the live-action industry will embracethis approach like crazy. When they can feed their ideas verbally intoa system and don't have to wait two weeks for animation to come out ofa ‘black box,’ they'll love it.”

At least one AI game creator agrees. “Production studios arereally interested in this technology for television applications, whereefficiency is important,” says Lorne Lanning, co-founder ofOddworld Inhabitants and lead designer of “Abe's Oddysee.”Lanning expects that the idea of running realtime character simulationswill soon become a reality. Lanning, a former Rhythm & Huesemployee who uses Maya for Oddworld's character animation, puts thisevolution in perspective. “The animator of yesterday keyframedevery shot, and occasionally got to reuse cycles,” he says.“What will happen in movie animation — and what's happeningin game animation today — is that animators are not animatingevery frame of what's happening. They're animating the initial cyclesof moves, and then the code is driving how that follows through —knowing when to use different moves and blending them all together.What that means is that during a 20-hour game experience, maybe 5% ofthe time that the characters are animated on screen actually representstime invested by human animators. The rest is recycling through thatworkload to create longer periods of content. In the future, thepercentage of human keyframing work will decrease, but it will still berequired for the initial construction of those moves.”

Lanning concurs with Armstrong's expectations, but only for thesecondary cast. “The bar for featured characters continually getsraised,” Lanning says. “The smartest code in the world isnot going to deliver a good performance. Hey, you've got humanbrains inside thousands of actors in Hollwood and only a handful canreally act! But what can happen is that tasks like pulling a lip bymoving polygons may involve a more library-oriented set of modules,where even a non-animator can ‘slide the anger up’ if thedirector wants, once the character has the embedded anger attributes inhim.”

Both Lanning and Armstrong agree that the arrival of commodityAI-animation tools will drive this trend, moving it beyond the handfulof studios that can afford to develop proprietary code. “You'llbe able to go to places with names like People Builders and buy theirsoftware,” Armstrong predicts.

Commodity AI-animation tool options are already emerging.Montreal-based BioGraphic Technologies now offers AI. implant, whichplugs into both Maya and 3ds Max and runs on Windows and Linuxcomputers. The software tracks each character, chooses the correctanimation cycles, scales and blends them, and gives animators feedbackin realtime.

“AI is the final animation technology to emerge,”observes BioGraphic founder Dr. Paul Kruszewski. “It's beenaround for awhile, but it hasn't been practical on an industrial scale.But game engine technology is getting shoved up into TV and filmproduction. It will not only make animators more productive — itwill also let them do cooler shots.”

Kruszewski admits that the biggest challenges are in the emotionalrealm. “We can't model emotions, though if I know a character isslightly angry and also tired, I can blend two animations,” hesays. “We're far from being able to match great actors. But badactors? We'll get there quickly!”

Armstrong believes it's only a matter of time. “We'llextrapolate from physically based stuff to emotional things,” hesays. “The reality is that this approach will be cheaper andfaster. And as simulations become increasingly elaborate, people willrely on them more and more. All you can do is embrace this newtechnology and learn how you can use it. It's going tohappen.”

To comment on this article, email the Millimeter editorial staff atmmfeedback@primediabusiness.com.