Real-Time Optical Motion Capture: Will it Change How You Make TV Animation?

Motion-capture technology has proliferated in the animation world duringthe past year. At the same time, expectations have accelerated even faster,especially for those pioneers who are using the technology in real-lifeproduction settings.

"Our goal is to produce multiple CG-animated shows with this technology,"says Larry Stanton, VP of technology at Netter Digital Entertainment, NorthHollywood. The studio has used motion capture on three animated televisionseries to date. "Clients mainly like motion capture because of its abilityto keep production costs down on such shows. But clients will always wantmore. That's why we're so happy to finally have real-time capabilities."

Indeed, real-time features are the most important recent development inoptical motion capture. That's because realtime is essential to theultimate goal: to simultaneously capture body and facial movements frommultiple actors in realtime without using markers. Pieces of this idealscenario already exist, of course. Real-time capabilities (and lower pricepoints) have long been available via Vermont-based Ascension Technologies'and Polhemus' magnetic systems. But the more costly optical mo-captechnology, which captures more detailed motion data with the smallestmargin for error, could not compete with magnetic systems on the real-timeissue until last year.

Optical Realtime ArrivesAt press time, the Hi-Res Real Time optical system from Santa Rosa,California-based Motion Analysis had been available for several months andis currently in wide use. Netter is a Motion Analysis customer and is usingthe real-time system as a key tool to animate two new CG series: Sony's MaxSteel, which debuts this month on the WB, and 26 episodes of a syndicatedseries the company could not publicly name at press time. At Siggraph '99,Tustin, California-based Vicon demonstrated its real-time optical system,Vicon 8 RT, which the company says is due to hit the market sometime earlythis year.

In addition to these commercially available systems, proprietary real-timeoptical systems are in place at Futurelight, Santa Monica, and GiantStudios, Atlanta. Futurelight's active-marker-based system relies onNorthern Digital Optotrak medical technology; Giant's passive-marker systemcomes from its parent company, Biomechanics, Atlanta. (Biomechanics'technology was formerly licensed to video-game maker Acclaim Entertainment,New York.)

Software SolutionAll of the "real-time" advancements in optical systems have come mainly viasoftware developments. Software now allows optical systems to instantlyinterpret motion data, compensate for any lost or "occluded" data, andinsert that information into computer models for instant viewing on set.This improves the quality of the captured data, reduces time spent in post,and allows for a real-time preview feature-a big advantage for mostdirectors.

Though all of the mo-cap systems benefit from improved software, eachoffers a slightly different solution with its own pluses and minuses. Atpress time, the Motion Analysis and Giant systems were limited to capturingdata from two actors at one time. Futurelight's system currently capturesup to four actors but requires a smaller capture stage. It utilizes active,light-emitting markers, which means that actors have to be tethered or weara transmitter pack similar to magnetic systems. Vicon's system was notready for distribution as of late November.

Still, in each case, new developments speed production through moreefficient mo-cap sessions and reduced post requirements. The"real-timeliness" of the previews is limited only by CPU power-how fast astudio's computers can render the CG model for viewing and how high-res themodel is. The preview is essentially instant for most low- andmid-resolution CG models and generally faster than older systems, even forextremely high-resolution images.

Optical systems have also made significant strides in addressing theproblem of occluded data (data that is not captured because a marker isinvisible due to camera angle or obstruction). With the optical systems,occluded data is compensated for by a combination of multiple cameras,redundant markers, and each system's biomechanically intelligent software.

The software fills in missed data based on the actor's skeletal model. Thismodel is created during a quick scan at the beginning of a mo-cap sessionthat usually takes only a few minutes. Therefore, the preview can proceedeven if specific marker data is missin-with no need for massive cleanup inpost.

Begging the QuestionSo given the improvements to optical motion capture, the obvious questionis: What exactly can you expect in a production environment?

"I'm torn about the issue of realtime for optical mo cap," says Tom Tolles,president of House of Moves, Los Angeles. The studio has used the Vicon 8system on a wide range of service projects and expects to upgrade whenVicon RT becomes available. "On the one hand, clients have heard about itand really want it, so we will eventually have it for them. On the otherhand, realtime doesn't let us capture any data or movement we can't alreadycapture right now, nor does it totally eliminate the need for some type ofpost work later in most cases."

The real-time preview feature, however, is worth a lot when producingbroadcast projects on quick deadlines and low budgets, says John Radulovicof Sony Pictures Imageworks. Radulovic has directed mo-cap sessions usingGiant Studio's Biomechanics system for two freelance TV projects that he isdeveloping. "The ability to drive a computer model in realtime, even ifit's a rudimentary, low-res model, directly helps the creative process,"says Radulovic. "It allows the director to work with the actor as though hewere directing live action, to get the exact performance he wants, and todo it without having to go back for re-shoots or tons of post work."

"With [the Motion Analysis system], if there is a problem, we know itinstantly and not a day or a week later," adds Keith Robinson, co-owner ofModern Uprising, Brooklyn, a Motion Analysis-based studio that recentlyupgraded to the real-time system. "It's true that there is a limit to thenumber of polygons you can use in your model-it can't be too high-resyet-and the system is also limited right now to two actors at a time. Butthe frame delay is just a second or two, so that's essentially realtime.[Motion Analysis says the delay, depending on the model's resolution level,is usually around two video frames.] As CPU power increases and MotionAnalysis improves its software, those problems will go away, and that willbe a huge advantage for studios like ours. In the meantime, the currentsystem saves us post work and it's a fantastic directing tool-a hugecreative weapon and money saver."

Systems in ActionAs of late last year, several TV, video-game, and film-production companieswere finding out firsthand whether optical systems can meet the growingexpectations of users.

Near the end of 1999, for instance, Netter was going into production on twonew TV shows with the Motion Analysis system. Michael Crichton's TimelineStudios in North Carolina hired Giant Studios' system to create a new videogame, and Giant also got a contract from New Zealand animation facilityWeta Ltd. to provide all motion-capture services for New Line's threeupcoming Lord of the Rings movies.

Futurelight, meanwhile, was busy assisting the production of the all-CGRoughnecks: Starship Troopers Chronicles. Foundation Imaging, Valencia, isthe lead animation shop on that show, utilizing its non-real-time Vicon 8system in combination with its Ascension MotionStar magnetic system. HyperImage, Burbank, also produced additional episodes, with Futurelightproviding mo-cap data. As a result, Roughnecks is the first series toincorporate both real-time and non-real-time captured movement.

At Foundation Imaging, computer animators direct the mo-cap sessions fortheir own scenes on an in-house stage using a non-real-time productionmodel. The company has become extremely efficient with this productionmodel. "Right now, the system we use is very good overall," says Dave West,Foundation Imaging's mo-cap supervisor. "We only wait around two minutesbefore we can see a low-res model after shooting the actor. But on a daywhen we are doing 15 captures, that totals 30 minutes of waiting for thecomputer to show us the preview. As I understand it, [Vicon's] real-timesystem will eliminate that delay, as well as the re-shoots and much of thecleanup if data is missed. When you are doing a project with the sheervolume of shots on the tough deadlines this project has, any advantage isgreat. Realtime would obviously be a tool that could help us."

Indeed, Rob Smiley of Hyper Image, who directs mo-cap sessions atFuturelight for Starship Troopers episodes, says he cannot imagineproducing the show without real-time capabilities. "With Futurelight'stechnology, the cleanup was minimal," says Smiley. "By the time we're doneshooting, we have everything we shot on tape already filtered, and we knowthe data is clean, that we won't have to come back and re-shoot. Theproblem with motion capture was always the cleanup, not the actual captureof the data. There is a cost with cleaning data and waiting for that data.On a TV-production schedule, we just couldn't afford that cost. Forepisodic, broadcast stuff, I view this technology as invaluable."

The Creative AdvantageStill, despite all the talk about the financial pros and cons of suchtools, directors who have run real-time, optical sessions in the last yearclaim its biggest advantage is creative.

"As a director, it took a little getting used to, but our project benefitedfrom it," says Bob Griswold of Timeline Studios, who directed sessions atGiant's Atlanta facility last year for the Crichton video game Timeline."To start, I watched the actor and got him moving the way we wanted. Seeingthose moves translated immediately on the computer model helpedtremendously. For video games, especially, the characters won't be thathigh-resolution, but little things like the placement of the feet and theway the actor holds their hands, all that is a big deal. If you only watchthe actor, they can look like they have it right, but then you see on themonitor that it's not right for the character-they are too low or notbalanced right or whatever. The other thing is, what you see in the realworld isn't what will actually be in the game, so the instant preview makesyou more comfortable. We had an actress swimming and we couldn't do it in apool, obviously, so we had her up on a table doing swimming motions. Itlooked goofy on the stage, but on the monitor we were able to direct herinto giving us moves that we knew would look good in the game."

Gary Roberts, co-owner of Performance Capture Studios in Los Angeles, astudio that uses the Motion Analysis real-time system, adds that suchtechnology has lured more live-action directors into his studio. "Before,it was mainly 3D people and animators directing here," he asserts. "It'snot exactly like directing blue or green screen, but there is a similarityin the sense that we are getting second-unit people in here and somefirst-unit directors, as well."

Roberts adds that the system helps actors. "We recently had a clientcapturing the moves of a famous actor for placement into the body of athree-foot, non-human creature," he says. "The character was obviously notthe same size as the actor, so the guy had to work on his walk and subtlemovements. By seeing the little guy on the screen, he was better able toget into character. He and the director could see instantly if he needed toshorten his stride or whatever. That put him at ease because he had a frameof reference for his performance."

Motion-capture technology has helped animators bring life-like human motionto animated characters. Most animated characters, however, are not humanbut animals, aliens, or other creatures. In the last year, mo-cap studioshave advanced the fledgling art of animal motion capture to address thisproblem.

Animal motion capture is a daunting task, considering the "actors" don'ttake direction very well and probably don't like having optical sensorsstuck to their hides. Plus, there are a host of safety issues involved whenworking with animals, and all sessions are subject to supervision by animaltrainers.

Late last year, two Texas studios- Dallas-based Computed AnimationTechnology and Wimberly-based Locomotion-combined their Vicon 8 opticalsystems to capture horse movement for the CG-animated Fox special The Nightof the Headless Horseman, a project directed by CAT's creative director,Shane Williams.

"The big problem was how to attach the markers to the horses," Williamsexplains. "We talked about using spirit gum and glue and shaving thehorse's hair, but the trainers weren't keen on that idea because these wereexpensive Arabian trick horses. We finally decided to sew together spandexcapture suits for the horses, essentially the same as what a human wouldwear but fitted for a horse. The horses were used to wearing outfits forshows, so they got used to it quickly."

The next challenge was how to configure the 18 Vicon cameras for maximumefficiency. "We had to do it in a corral because the tricks required softdirt," says Williams. "So we suspended the cameras in the air over thecorral, connecting them to ceiling scaffolds, and ran wires back downthrough the air to connect them to a Vicon workstation processing the data.That was the only way to get the range of motion we wanted while keepingthe camera cables off the track. It looked like a spider's web."

Performance Capture Studios of Los Angeles, a Motion Analysis-based studio,also conducted sessions with an African elephant and a lynx last year foran unnamed test film. Company co-owner Gary Roberts admits that attachingmarkers to the two-ton elephant was "a royal pain."

"Working with the trainer, we decided to put markers onto the elephant'sbristly skin," he says. "We attached them with veterinary bandage tapedirectly onto his skin in some places and onto flexible veterinary wrapbandages for his legs, trunk, and tail. We got about 17 markers on him andcaptured the motion the client wanted, but it took several hours. That wasmore because the elephant was mischievous and hungry all the time. Atfirst, he pulled markers off with his trunk and fired them across thestudio, but he finally agreed to leave them on when the trainer beganrewarding him with treats."