Creating Virtual Performers

Disney’s Human Face Project

The CG figure on the left was generated from the actual actor on theright using a process called performance capture "cross-mapping." Asopposed to conventional motion capture, performance capture focusesmore on overall motion, not the motion of specific coordinates.

Of all the recent developments in computer animation, few hold thepromise of Disney's Human Face Project, spearheaded by one of CGI'sgenuine pioneers, Lance Williams.

Even in a year marked by notable advances in computer visiontechniques — which allowed filmmakers to extract 3D informationfrom 2D photography in order to add 3D-CG elements to live action— the approach developed by Williams' team stands out. When theHuman Face demo was unveiled at Siggraph 2002, its implications forvisual effects, animated films, videogames, and even animatronicscouldn't be denied.

Filmed by Visual Effects Oscar-winner Hoyt Yeatman, the demo simplyshows two men sitting side by side at a gaming table, pleasantlytalking to one another. What's striking is the resemblance betweenthem. Were it not for an obvious difference in their ages, they couldbe twins. But in fact, the older man is the only real actor. Thephoto-real 3D-CG face of his youthful clone was brought to life througha process that Lance Williams characterizes as“cross-mapping.” “We tracked the face of a guy in hissixties onto a guy in his thirties. We abstracted the performance ofthe actor in a way that allowed us to cross-map it to anotherface.”

This cross-mapping technique, developed by Williams' 12-person teamover the past couple of years, was initially prompted by the demands ofa film that ultimately was shelved. Yeatman, slated to supervise thatfilm's effects through Disney's FX arm The Secret Lab, explains thechallenge. “The impetus was to be able to have a famous actorwith a long film history — like a Sean Connery — and have ayounger ‘double’ that would show the actor at about 25years of age. At first they thought, ‘Let's use makeup.’But making someone look young is not a process of adding stuff —it requires removing stuff. Then they also tried 2D effects, but thatlooked like a bad burn victim.”

When Yeatman was brought in, he recalls, “I thought we shouldlook at the idea of doing what I would call digital makeup. We wouldessentially have the mature actor driving the facial performance of hisdouble. There were a number of techniques out there, all of which werereally just motion capture. No one had really conquered the idea ofcapturing performance.”

Conventional mo-cap processes, in which a performer is fitted withreflective markers and then filmed with multiple cameras, really justrecords a shell, Yeatman says. “And that's the problem. If you goto edit it, or apply it to another face with a different shape, youknow nothing of the performance — you just know the physical 3Dcoordinates.”

In contrast, Williams and his collaborator Walt Hyneman came up witha concept not unlike what musicians do when they use MIDI technology totranslate a performance made with one instrument onto another.

Describing how this worked for the Human Face demo, Yeatman says,“We filmed our actor split-screen, sitting at the gaming table.We gathered information on him using just two cameras, almost like aCinerama setup. One was the production camera. In this case it was aVistaVision camera, which offered us higher resolution. We also had a4-perf camera off to one side, not even slaved. That's all that wefilmed, but it gave us a tremendously rich field of data — asopposed to some poor soul having to glue lots of reflective dots ontosomeone's face.”

The resulting footage, when scanned into the computer, was subjectedto software analysis. While Disney understandably considers the detailsof this technique extremely proprietary, in essence what it does istrack pixels. It looks at the photographed image in both views from thetwo cameras as each pixel moves from one view to the other. Every pixelmaking up the face is a target for this optical flow analysis.

In preparation for applying the actor's facial performance to acomputer-generated digital double, the Disney team worked with plasticsurgeons to identify the roughly 84 muscles that comprise the humanface.

Again using a musical analogy, Yeatman likens the movements ofdifferent muscle groups to playing chords. “Usually CG animatorshave sliders that go from happy to sad. But by playing a‘chord’ essentially, you'd get a group of muscles tomove.”

With these sliders in place, the pixel-specific nuances of theactor's facial performance were applied to the digital double's face.The mathematical engine developed by Williams' team looked at those 84individual controls and tried all the different permutations until itfound the right combinations that would drive those 84 sliders to makethe young CG face match the facial expressions of the olderphotographed actor. “Computers are great at comparing,”notes Yeatman.

While this sounds very automated, the results are tweakable,allowing filmmakers to edit them easily by adjusting individualsliders. “Cross-mapping becomes very simple,” says Yeatman,who believes that it's also very malleable. “In this case, webuilt a younger version of a person. But we also did tests where wemapped human expressions onto an insect. As long as your muscle systemsare cross-mapped, just like in MIDI, the data that you've stored isreally the performance, not the motion. You can use it to drive theperformances of characters with unusual shapes. Eerily enough, you cansee the essence of that person in that performance. You could make aworm act like Robin Williams.”

That's a telling analogy, given the great cel animation that wasinspired by Robin's expressive face for the genie character he voicedin Disney's Aladdin. While that was obviously done manually,this new technology represents a logical extension of Disney'stradition of photographing reference material for its animators. AsLance Williams observes, “A lot of Disney animation was wellsupported by shots of actors staging bits of business. What thecomputer offers in this area is a much more flexible way of acquiringmotion.”

In thinking about how the Human Face Project technology might beapplied to future animated characters, Disney Technology Vice PresidentJohn Carey says, “This will allow us even more nuancedperformances that we're able to animate by hand. That's the hope. We'retrying to develop high-level tools for animation. We believe there'sreal production benefit to this technology, enabling us to make eithermore nuanced animation or a different kind of animation. I hesitate tosay ‘more real’ because when we capture a performance wemay cross it with something that couldn't exist in nature.”

The captured performance is scanned and analyzed by software to makesure pixels are tracked correctly.

The prospect of outrageous aliens giving performances driven bygreat actors is enticing. Yeatman says he can imagine an actor likeChristopher Walken creating a creepy creature without undergoing sixhours of makeup every day. “You can apply his performance to a CGcharacter that you can squash and stretch in unusual ways. In additionto expanding the abilities of performers, you can generate charactersquickly and efficiently,” he says.

Another advantage for visual effects filmmakers is that, unlikecapture technologies requiring special stages or infra-red equipment,Disney's approach can be brought on location, and actors can be filmedunder regular stage lights or even sunlight. It's been testedsuccessfully using HD cameras as well.

Carey cautions, however, that Disney has tried not to make claims ofefficiency. “The driving force behind this is having newcapabilities,” he says.

For his part, Williams addresses the question of efficiency byasking: “Do you consider acting a short-cut over animation? Ithink the best near-term way to view this is as a way of opening themedium of animation to the lively arts — like juggling, martialarts, and acting in general. There's no reason that we can'tincorporate the talents of realtime performers as well as thepainstaking work of gifted animators. It may not necessarily be cheaperor faster; it depends on what actor you're trying to shoot. But itcertainly expands the scope of the medium.”

Williams believes that videogame animators are likely to recognizethe advantages of performance capture, especially given theirwidespread reliance on motion capture. And he notes that it could beused to have actors drive the performances of animatronic characters.After all, he remarks, “For its theme parks, Disney developedactual performance capture in the 1950s to drive audioanimation.”

But Williams wonders about resistance among animated moviemakers.“In animated movies, it's amazing how strongly the artists regardtheir traditions. Pixar's John Lasseter emphasizes that his fellowanimators are all actors. “With a technology like ours, they'dhave a chance to prove it — and go head to head with the likes ofJudi Dench.”

During his extensive career, which includes tenures at AppleComputer and DreamWorks, Williams has experimented with facialanimation techniques before. But the strategy of applying computervision techniques to animation has moved his research into a new arena.“I think the time is right for the useful application of thesetechnologies in motion pictures. They may not be ready to performsurgery, but they're safe enough to work on cartoons!”

Williams ultimately sees the Human Face Project as the continuationof a venerable tradition. “At the dawn of motion pictures,Eadweard Muybridge was interested in photography as a tool forcapturing data. It only secondarily was taken up by others as asensational entertainment medium. But it's always advanced the purposesof science, and it will continue to do so until the last producer haschoked on his last cigar!”