Driving Force

The Sci-Tech Award-winning compositing tool Nuke, which was developed at Digital Domain, is now marketed by The Foundry. The studio used it most recently on The Mummy: Tomb of the Dragon Emperor.

Software spawned by the needs of production studios is a venerable tradition in the animation and visual-effects industries. Pixar's 20-year-old RenderMan software originally grew from the computer-graphics department at Lucasfilm, and Autodesk Maya has roots in Wavefront — which evolved from the '80s effects studio Robert Abel and Associates. And this tradition has continued. Weta Digital nurtured the creation of the crowd-simulation software Massive, while the compositing tool Nuke came from Digital Domain and is now marketed by The Foundry. Yet despite the proliferation of commodity tools for creating high-end CGI, many production studios are still developing proprietary code for key parts of their pipelines. Software that is not for sale at any price speaks volumes about where the leading edge lies.

At longstanding studios such as Industrial Light & Magic (ILM), Rhythm & Hues, and PDI/DreamWorks, inhouse legacy tools remain core technologies. And although Pixar sells RenderMan and its render-queue software Alfred, as well as the Maya-to-RenderMan code RenderMan Studio, its animation tools remain closely held. Conversely, while Blue Sky Studios buys Autodesk Maya for animation, it has spent 20 years perfecting its inhouse raytracing software CGI Studio to give its films a signature look. Established shops such as Sony Pictures Imageworks (SPI) and Digital Domain (DD) rely on commodity tools, yet devote considerable resources to developing proprietary code. Even studios that have ascended during the past decade, such as Double Negative (DNeg), are committed to writing new tools.

The conventional wisdom is that custom code can give a digital production shop a competitive edge, allowing it to deliver unique-looking shots. This has certainly been the case with ILM, which never licenses its proprietary tools. But other factors are at work, too, according to Ray Feeney of RFX, a veteran software reseller. “All the major successful companies have R&D departments of consequence,” he says. “They create a certain amount of their own pipelines themselves, probably for different reasons. For the early companies, when commercial tools cost $20,000 per license, the equations that determined whether you could compete by building your own tools were different than they are now. RenderMan and Maya were hatched when there wasn't the pressure to keep the cost less than $2,000 per license. I don't think people will ever again be able to charge tens of thousands for a piece of software, no matter how good it is.

“There's also a crossover point once a studio is above a certain size. When you reach a certain size, maybe it's cheaper to do your own thing, or at least be competitive.”

The ability to maintain a proprietary system over time is affected by larger forces, Feeney says, citing the example of how PDI's animation system survived that studio's purchase by DreamWorks. “All sorts of things could have happened — and if not for Shrek, all sorts of things would have happened,” he says.

Debbie Denise, SPI executive VP of production infrastructure, agrees that timing is crucial. “It's really dependent upon when a company sets their pipeline in motion,” she says. “So much is dependent on when a certain type of software is robust enough for use in a production environment, because you don't want to be testing new software when you're trying to deliver 1,000 shots. There's also the question of continuing support and development. If you purchase software, there may be 50 people supporting it. That can be good or bad — depending on whether you're waiting in line to get your specific needs addressed."

Sony Pictures Imageworks developed Arnold, a global illumination renderer for characters and environments. Shown is a progression of shots for Eagle Eye, for which the tunnel is completely digitally constructed. The truck and car were shot at an exterior street location and then inserted into the digital tunnel. The debris, fire, and wreckage was from the stunt plate, but interactive lighting was added digitally to the CG tunnel to integrate the explosion.

Companies also have to weigh training issues when they consider
building versus buying tools. “If you're using something that is off
the shelf, it's easier to bring in new people without having to train
them on your internal software,” says SPI Chief Technology Officer
George Joblove. “But if you'll need 500 licenses of something, it can
get really expensive. So we're pulled in conflicting directions.”

Imageworks has elected to build software in
several areas of its pipeline. The studio has developed software for
doing facial animation of mo-capped actors, using it on both Beowulf and I Am Legend.
And the studio has two proprietary rendering packages in active
development, according to Rob Bredow, Imageworks creative technology
supervisor and visual-effects supervisor on Surf's Up. “There
is Svea, which is a fast volume renderer for clouds and atmospheric
effects,” he says. “We have lots of people trained on it. The other
area, which is even larger, is that we're looking to do all of our
primary rendering with a new raytracer called Arnold. It's a global
illumination renderer for characters and environments that was first
used on Monster House, and then for the effects in Eagle Eye. Cloudy With A Chance Of Meatballs is being rendered exclusively in Arnold, and we're actually able to light interactively at the desktop.”

While Imageworks has the resources for
ambitious R&D programs, smaller shops also have come up with
proprietary rendering solutions for specialized needs. London-based
Double Negative has been rendering impressive atmospheric effects using
its dnb volumetric renderer, a 64-bit voxel tool that was originally
written to create steam effects for Batman Begins. “Since then,
dnb has undergone four major revisions, and has been used to render all
sorts of fluid and particle simulations in Hellboy 2, Harry Potter and the Order of the Phoenix, and The Dark Knight,”
says DNeg's Head of Technology David Scott. “Having our own inhouse
renderer means we have access to all the inner workings that you would
not usually have access to with a commercial renderer.”

The rendering of atmospheric effects is a
crucial focus for all companies that work on visual-effects pictures,
and Digital Domain has developed its own volumetric renderer, Storm, a
2004 Sci-Tech Academy Award honoree. “We won't write any of our own
software unless it gives us an edge,” says DD's 15-year Creative
Director of Software Doug Roble. “We've always had the philosophy that
we don't need to compete with the big software companies. Very early
on, we decided not to write our own renderer, though we've often
second-guessed ourselves on that because rendering is not a solved
problem. Sometimes I'm a little jealous of companies that have their
own renderer, because if you haven't written the renderer, you can't
modify it. But on the other hand, I look at companies that are
developing an inhouse renderer and they have five or even 10 people
working on it just to keep pace with new versions of RenderMan.”

Roble says that the pressing challenge for proprietary development is in simulation, which is needed to create believable water, hair, cloth, and other complex effects. “With rendering, you render one frame and the next frame doesn't care what goes on in the first frame,” says Roble, who won 2007 Sci-Tech honors for DD's fluid-simulation software. “So you can render all the frames at the same time if you have enough computers. With simulation, that's not the case. You have to know what happens on frame 10 before you can start on frame 11. So there's a lot of work being done now on how to do this faster.

“Our system is highly integrated into [Side Effects Software] Houdini. We didn't have to write our own animation package to give ourselves an edge.”

Imageworks, which also uses Houdini for effects animation, has broken off its cloth and hair teams as a proprietary group. “We have an inhouse cloth solver called Tango,” Bredow says. “We also have proprietary fluid solvers that work in both Maya and Houdini. It's a combination of techniques.”

Double Negative also developed a fluid solver called Squirt for Ink-heart, and it has added capabilities for simulating large bodies of water in the upcoming Harry Potter and the Half-Blood Prince. “After a year and a half of development, nearly all our fluid-based effects go through Squirt,” Scott says.

The point in the pipeline when houses put their final stamp on a shot before it goes out the door is in compositing, and that's an area that's remained ripe for proprietary development. This is especially true since Apple stopped developing Shake, according to Feeney. “Shake is never going to migrate to a 64-bit application, and as people revamp their pipelines to 64-bit, The Foundry's Nuke software will come into its own,” he says.

This makes Roble smile, since the Sci-Tech Award-winning Nuke was developed at Digital Domain and the company continues to refine its version. “It's a nice advantage when people come here having some Nuke experience,” he says. “Compositing isn't a solved problem, and there's lots of room for development. People are doing so much during compositing that — even here — many of our software requests are for development on Nuke.”

Notably, DNeg has modified its own 64-bit version of Shake, while at SPI, a huge effort is underway to develop a next-generation compositing system that will integrate 3D lighting and 2D compositing. “It's tied to our inhouse lighting tool Katana,” Bredow says. “It enables us to defer some choices to the composite stage but still have that information fed back into 3D. Compositing used to always be about 2D, but today's compositing packages all deal with issues of 3D space. So integrating 2D and 3D means we only have to set stuff up once. You get the benefit of efficiency if you do something custom.”

Efficiency may be a big bottom-line reason why proprietary software continues to thrive. But unpredictable creative urges may also drive studios to develop new code, according to Roble. “One of the great things about developing software in a production company is that you have artists coming in all the time saying, ‘Wouldn't it be cool if…?''” he says.