Tracking the Beast

Tracking tools are the key to integrating computer graphics and video. Here are a few that can help you tame the CG beast in your next project.

Ever since 1914 when Winsor McCay thrilled vaudeville audiences with Gertie the Dinosaur, a silent film that featured McCay - both in person and as an animation - interacting with his hand-drawn title character, the combination of live action and artificially created graphics has been a mainstay for creative professionals. Today, we see a mix of computer-generated 3D wizardry and real images in everything from sci-fi blockbusters to primetime TV commercials. The onscreen results often blur the line between fact and fantasy.

There are a few different ways to combine live action and 3D modeling. The most common method is to track a virtual camera along live-action video inside a graphics computer workstation and composite it with the movements of 3D animated figures or XYZ-axis rotating models. Several companies make tools for extracting this camera path and multi-dimensional tracking data.

SynaPix's SynaMatch, currently in version 1.7, is a Windows NT-based software package that can recover 3D camera paths from what have traditionally been difficult clips to track. Once a computer graphic has been created in a software animation package - Discreet 3D Studio Max, Softimage|3D and XSI, Alias|Wavefront Maya, NewTek LightWave 3D, to name a few - SynaMatch can recover a 3D camera path on unstable handheld shots, long image sequences, archival footage, and sequences with objects entering and leaving the scene.

"You select a series of keyframes in the live-action video and have those correspond to points at a later time in the scene," says Mark Cajolet, director of product development at SynaPix. "What makes SynaMatch unique is that you can identify points in the first and last frames in a sequence and ask the system to automatically track all the other points in between."

The SynaMatch user interface presents the scene in full color in the left window, and in the right window displays a 3D graphic representation of the camera path that can be played back over time to verify the accuracy of the recovered 3D points. Linear bars at the bottom of the display resemble timelines with a horizontal band tracking each point against its position moving through the shot. An area on the left of each band gives immediate feedback about the confidence the system has in the accuracy of the intermediate points, and labels their reliability with a color indicator ranging from green (highly reliable) to yellow (needs user input) to red (cannot be tracked).

Realviz, on the other hand, utilizes technology derived from years of research at the famous INRIA laboratory in France. Realviz has just released version 2.0 of its MatchMover camera tracking software application for the Windows NT and Irix platforms. "Realviz was born in 1998 as a result of a meeting of research scientists and people from the post-production industry," says Emmanuel Javal, general manager of Realviz. "[The company] uses what is called `computer vision' technology developed for robotic implementations by extracting a 3D camera path from designated 2D points in live-action sequences. For example, the Mars Sojourner rover used computer vision technology to determine its distance from objects on the surface of the red planet."

Computer vision is a form of the parallax vectoring paradigm, by which the geometry of 3D space can be extrapolated from two points of view. Realviz signed an exclusive technology transfer agreement with INRIA to develop this technology for the integration of computer models with live video. New features in MatchMover 2.0 include a Graph Editor that allows the user to edit camera parameters as curves on a graph, and a Status Tracking feature that permits tracking and analyzing a project's progress and results. Version 2.0 can track 2D processes up to eight times faster than the previous version. Specifications like these may have prompted Discreet to chose MatchMover as the 3D tracking engine for its Inferno effects system.

Diabolina Productions, a digital visual effects house in Montreal, Canada, that specializes in creating photo-real elements and integrating them into live-action footage, uses Softimage in concert with Realviz's Image Processing Factory tools, including MatchMover. Akie Prapas, co-founder of Diabolina, used this combo to create and track the gargoyle image on pages 30 and 31. "In creating the gargoyle shot, we had to extract the camera information from the footage and import that into Softimage to allow the virtual set to follow the real camera motion. This was done in MatchMover," he says. "Once the footage was transferred to the computer, the initial test did not take long at all. There were over 350 frames on the clip that we decided were usable and interesting. Match-Mover did an amazing job at the test level as well as the final solve of the camera."

2d3, a member of the Oxford Metrics Group of companies, has come up with a different approach to determining camera path data. Called Boujou (pronounced "boo-zhoo"), 2d3's innovative system for Windows NT and 2000 is the industry's first fully automated camera calibration and tracking system that frees the operator from manually identifying target pixels. "The hardest aspect of inserting a computer-generated object into live action is matching the natural movements of the cameraperson," says Chris Steele, CEO of 2d3. "Unless that can look convincing in the final composite, the viewer's eye will immediately reject the effect. Boujou can analyze a video stream and automatically produce data points that reference to where the camera was and at what it was pointing. It even figures out the focal length. That lets you drop 3D objects into the scene immediately without the user having to plot the necessary tracking points."

2d3's Boujou system was recently put to the test by Vizix Digital Studio, Austin, Texas, to prove that a proposed condominium development would not have a negative impact on Austin's skyline. Cal Rodgers, executive producer at Vizix, says his architectural visualization company produced a series of 3D animations re-creating the downtown area where the controversial condominiums were to be located. But the city's design commission didn't trust the completely computer-generated visualization. "By utilizing the Boujou software, we were able to shoot live video of the actual construction site and insert 3D models of the buildings themselves," Rodgers says.

The final shot is from the perspective of a Ford Explorer approaching the virtual city, and Rodgers' team was even able to matte out passing lampposts and parked cars that occluded the computer-generated building.

Tracking Master PThe only commercially available software environment that provides support for real-time hardware camera tracking is Kaydara FilmBox. As a result, the FilmBox application suite offers content developers a fully scalable, turnkey solution for the creation, management, and broadcast of 3D digital animation, all in real time.

Dan Kraus, director of products at Kaydara, says FilmBox can be used in combination with a number of packages, such as Softimage| XSI or Maya, to create animation quickly and output to a number of formats for many applications. The goal is to enable high-volume character animation and repurpose its delivery for any release format. "We are sort of the Switzerland of 3D packages in the sense we don't have a modeling system of our own," Kraus says. "We depend on importing animation content from other vendors. Our customers bring in data from those packages and use FilmBox's real-time rendering to position the images in the context of live video backgrounds and use that either for direct broadcast or output to a storage medium, such as tape or disk."

FilmBox incorporates drivers for any hardware camera tracking system and streams the positional coordinates into its software. Since it runs in real time, the real camera and virtual camera always remain synchronized. The video input from the real camera can be used as a background for the virtual camera, and located behind a 3D object to composite a computer-generated, multidimensional environment.

Crispin Broadhurst, senior animator at Computed Animation Technology, Dallas, recently used FilmBox on a music video starring rapper Master P. The video, titled "Soljas," is comprised solely of computer-generated images. "We used the optical Vicon 8 motion-capture system to track individual figures and needed to generate dozens of soldiers and hip-hop characters dancing across the screen," Broadhurst says. "We were even able to capture a model of Master P throwing his hat at the camera with an excellent 3D effect. We were able to create all the animation in the Kaydara FilmBox and export it to NewTek's LightWave software for rendering. The Character Tool in FilmBox let us fit all the motion-capture data onto the people, and its NLE gave us the ability to combine motions."

Other ApproachesThere are other methods for integrating 3D images in live-action scenes besides computer-originated camera tracking. In traditional digital post production, editors have long yearned for the ability to position 3D computer-generated images within an edited sequence. Quantel Editbox, an uncompressed nonlinear editing and effects system, allows users to manipulate 3D graphics within its own editing software. Two years ago, Quantel began integrating features from Alias|Wavefront's Maya 3D graphics software into Editbox to bridge the gap between 2D and 3D effects creation. The company originally attempted to accomplish this by including a complete Java interface to Maya, but this approach met with only limited success. Then at last year's NAB the company debuted an external Windows NT platform called the Quantel Open Render Engine, which has successfully linked the two operating environments.

Dan Germain, technology specialist at Quantel, says instead of importing 3D graphics into Editbox, a computer graphics artist can create a model as individual layers on their own Maya system and then export the layers into Editbox as a pack. "To maximize the workflow, each layer is rendered clean with hard edges so that the Quantel editor can use Editbox's real-time effects capabilities to add blur and shadows," he says. "The editor is also given handles to the 3D graphic for final positioning to fit the image into the space of a sequence's live-action video, and then all the tracking is accomplished in Editbox using uncompressed video."

When the X, Y, and Z axes for the model have been determined based on its position in the live video in Editbox, that information can be transferred back to the Maya system so the graphics artist can apply it to the animation. That way, interactive aspects, such as reflections, can be made more realistic. The ultimate result is to divide the creation and positioning of the graphic model and its relationship to the live-action video between the Maya and Quantel systems so that each can contribute aspects based on their strengths.

Another company, Inter-Sense, has found a way to economically generate tracking information from the camera head itself. Previously, this had involved prohibitively expensive technology suitable only for major motion pictures. Now,for half the cost of other camera tracking systems, InterSense has produced the compact IS-900 SCT system, which provides accurate position and orientation tracking based on the company's own award-winning Constellation technology.

Dean Wormell, Inter-Sense's manager of strategic partnerships, says the technology was originally developed for military training simulations. "Back in 1997, our founder Eric Foxlin had the idea he could record the movements of a soldier's head in virtual environments better than the existing motion-tracking technology that depended on expensive gyroscopes and accelerometers," Wormell says. "He configured a navigation system using inexpensive components and made it small enough to fit on the body. Now we have adapted it to record the position and orientation of a camera in relation to the geometry of a computer-generated virtual set."

The core component of the IS-900 SCT employs both inertial and acoustic tracking technologies. Its Constellation architecture uses advanced SensorFu-sion algorithms to deliver wide-area tracking performance. This lets the system choose the best measurement from any sensor at any given time to get the best result.

"This is all real-time tracking information," Wormell adds. "And the data from the IS-900 SCT can be exported to film or video post-production systems. It can also help digitize single images on the set to add or subtract elements that might get in the way. The result is a tracking system that resides on top of the studio camera at a $30,000 price level that makes it affordable to a much wider selection of producers."

Finally, there is a potentially exciting technology called Zcam on the horizon from 3DV Systems that has been demonstrated at video tradeshows in the past two years but has not yet emerged as a full-fledged product, although the company is targeting the first quarter of 2001. Zcam uses the depth measurement, or Z axis, of an object before the lens to insert computer-generated elements.

Ori J. Braun, president and CEO of 3DV, says Zcam is actually a chipset made up of three components: a pulsed infrared illumination source, an image sensor, and a digital signal processor that controls the logic of the system. In simplistic terms, the Zcam chips work like a radar system integrated into a traditional video capture device. 3DV's proprietary Parallel Range Sensing technology ensures that all pixels are read in parallel for real-time performance.

We should see Zcam technology demonstrated on rental camera systems in the next few months, and it may be available for purchase by the end of the year. Then, news anchors will be seen sitting in front of any locale without a bluescreen back plate, even though they are stuck in a studio. 3DV even hopes to put Zcam chips into inexpensive consumer cam-corders, turning them into "object cameras." Imagine teleconferencing from your home and using Zcam to make it look as if you're sitting in a penthouse office. By integrating computer-generated backdrops with your own live video, Zcam will be able to take you anywhere.

2d3Oxford, UK+44 1 865 261830www.2d3.com

3DV SystemsYokneam, Israel+972 4 959 9599www.3dvsystems.com

InterSenseBurlington, MA781-270-0900www.isense.com

KaydaraMontreal, Quebec514-842-8446www.kaydara.com

RealvizSophia Antipolis, France+33 49 238 8460www.realviz.com

SynaPixLowell, MA978-970-5300www.synapix.com