Summer of the Supervisors

Web-Expanded Sidebar

Seven Films: Visual Effects Stats

I, Robot used the EncodaCam camera visualization system, among other tools, to previsualize all effects shots in the film. (Photo courtesy 20th Century Fox)

According to John Dykstra, the biggest change in feature film visual effects is change itself. Dykstra, the veteran Sony Pictures Imageworks visual effects designer who recently completed effects for Spider-Man 2, simply means that visual effects teams, thanks to advances in technology, technique, and experience, can not only “produce magic,” but do it at a faster pace, continually incorporating changes up until the eleventh hour and beyond.

“We now have the ability to let the director change his mind with more regularity,” says Dykstra. “He or she can use their intuition or the spontaneity that occurs with photographing live action in conjunction with visual effects to try different things. We aren't hamstrung anymore with doing it all separately, as multiple tasks. We aren't stuck, for instance, with motion-control cameras on-stage and being locked into certain camera moves. This is a huge change in visual effects.”

In many ways, this change has been prominently displayed on cinema screens this summer, which have featured an unusual glut of big budget, effects-driven motion pictures. Between May and September, at least 10 prominent effects movies were scheduled to debut. These films, most of which were in production around the same time, often pushed the effects envelope on the tightest of deadlines. But did they move the visual effects industry forward in any significant way?

“It's not so much that anything truly new or earthshattering was developed,” explains Matt Johnson, visual effects supervisor on King Arthur. “It's more about the fact that we've reached a place where we really can do whatever the story calls for in an affordable way. I predict this will free up a lot of scripts that have been lying around on shelves because they weren't affordable before.”

To delve deeper into what legacy this summer's visual effects will yield, Millimeter surveyed Dykstra, Johnson, and five other visual effects supervisors on big summer films: Karen Goulekas of The Day After Tomorrow; Roger Guyett, co-supervisor, along with Tim Burke, on Harry Potter and the Prisoner of Azkaban; John Nelson of I, Robot; Scott Squires, co-supervisor, along with Ben Snow, of Van Helsing; and Nick Davis of Troy. Following are their thoughts on their films and the current direction of their industry.

Up Next: What is the major visual effects advancement that you are most proud of on your particular film?

Millimeter: What is the major visual effects advancement that you are most proud of on your particular film?

Spider-Man 2, according to John Dykstra, posed great challenges in creating believable tentacles for the film's villain, Doctor Octopus. (Photo courtesy Columbia TriStar)

John Dykstra: “We learned a good lesson from the first movie [Spider-Man] about lighting the CG character when he is swinging through the city, for instance. It's much more advanced this time around. In this film, the character and his opponent, Doctor Octopus, are seen in CG with their faces exposed — Spider-Man's face isn't always covered by the mask anymore. That means we had to improve our CG facial stuff a great deal. We wanted a system to simulate the 180 muscles that comprise a person's face, and to track and record and play back in simulations each of those muscles in concert with the actor's performance to create emotional content. In reality, it's impractical to do it that way, so instead, our animators took a shortcut that still accomplishes the same overall goal — they simulated the extremes of emotional content as portrayed by the actor and blended them together. You mix an ironic expression with angry, for instance, you get angry irony, I guess you could say.”

Karen Goulekas: “On The Day After Tomorrow it was our environmental work. Unlike movies like Twister or The Perfect Storm, this movie wasn't about doing one thing very well. We had storms, huge tides and waves, super freezes, twisters — all that stuff in the same film. In terms of making all that photoreal, I like to think we kind of raised the bar, at least a bit.” (For a detailed look at creating the “big freeze” that engulfs New York City, see the Step by Step column in the May issue of Millimeter.)

Roger Guyett: “Our challenge [on Harry Potter] was the need to have a more organic look to the effects, particularly the creatures. We didn't want too many ‘magic wand effects’ this time, or a fantasy look to the effects. We wanted the creatures believable. The hippogriff Buckbeak [a half-bird, half-horse creature], for instance, gets extended screen time, and we show him eating, even going to the bathroom.

Harry Potter and the Prisoner of Azkaban included a realistic-looking creature called a hippogriff, created at Framestore CFC, which did extensive R&D work on feather dynamics for the project. (Photo © Warner Brothers)

“I'm very proud of the work on that creature. His body is covered with both feathers and hair, which have to move dynamically with his animation.

“Another equally challenging task was the Dementors [wraith-like creatures that fly through the air]. Our cloth simulation work really improved to make the Dementors believable. We initially did tests inside water tanks, where we clothed real puppets and moved them underwater. We loved the tests, but we eventually used a CG solution to give us more control over the movement of the cloth. In particular, we wanted to reverse the physics of some of the cloth motion.”

Nick Davis: “The thing I'm proudest about in Troy is the seamless blending of real and virtual elements for armies. We shot most of the plates in full daylight with harsh lighting, which shows all flaws, and yet we blended them seamlessly together with the CG elements. This is a wonderful advance, in my opinion. Our armies are large, but not as large as in Lord of the Rings, which means that flaws are easier to spot.

“In our case, we wrote our own code from the ground up. [Director] Wolfgang Petersen wanted a sort of balancing style with the armies — epic style for wide shots, as the two armies approached each other, sort of a Lawrence of Arabia approach, and then a closer, handheld, close-quarters feel once the two armies begin to clash.

“The biggest challenge was probably the Battle of Arrows sequence. We only had 600 to 800 extras for filming, and that doesn't go far in terms for creating size and scope. But we agreed not to use any blue- or greenscreens. We wanted to maintain the natural canvas of our location as the background — no digital environment. So when we were tight, those were real actors. As soon as we moved the camera back, CG would take over, and we created about 70,000 digital soldiers to go with our real actors. There was a ton of rotoscope work for this, performed at Moving Picture Company [London].”

Scott Squires: “[Van Helsing] was a very challenging project, mainly in terms of the variety of effects. Each creature behaves in a totally different way, so we had to use different types of animation and technical details. We have fur, ripping skin, Dracula's flying brides, and so on. We have CG creatures and environments, 3D digital matte paintings, and other things.

“One of the big challenges was creating believable ripping skin for the werewolf sequences — an entirely new approach to visualizing onscreen the transformation from man to werewolf. Our art department developed the look of the transformation, the ripping skin. But we were actually ripping both skin and fur.”

Matt Johnson: “Environmental effects were the main accomplishment on King Arthur. The entire movie was shot in the middle of summer in the green valleys of Ireland, yet we had to make it take place in a cold, snowy winter. We did lots of work taking green mountains and other footage and digitally painting in snow and ice to create a sort of mythological landscape.

The frozen lake sequence in King Arthur was shot in the summer in Ireland, and artists at Cinesite then added computer generated snow, ice, rocks, flurries, and other elements. (Photos © Touchstone)

“The toughest sequence in this regard was what we call the ‘frozen lake’ sequence. That one sequence has about 275 digital shots. Our heroes are pursued by bad guys across a frozen lake. There obviously was no frozen lake in Ireland in the middle of August, so [director Antoine Fuqua] had us design a scene where they are entering a narrow, rocky gorge closed on all sides, making it clear there is only one direction to escape into. Everywhere in that scene, the actors are standing on CG snow and ice, the rocks and mountains surrounding them have been added digitally, and there are CG snow flurries — everything created in the computer. As the scene progresses, the ice begins to crack and we get a POV underneath the surface of the lake, looking up. We then had to create full CG shots from the underwater environment, looking through the ice, to give a sense of the thickness of the ice, the reflections of light, and so on. We even had CG people walking across the ice, and CG leaves floating underwater.”

John Nelson: “One of the key breakthroughs on I, Robot was our extensive use of the EncodaCam [camera visualization system created by General Lift, El Segundo, Calif.] during the shoot, which allowed us to do a form of on-set compositing for the entire production. In essence, we previzzed the entire movie this way, and that helped tremendously.

“The breakthrough is that the encoding equipment is precise and fast enough to capture every component of camera movement, encode it, put it into a computer and composite it with rough, pre-existing digital sets in realtime — the lag time is less than a thirtieth of a second. I believe this to be the first time this system was used extensively on a feature film. I really think this is a whole new world for shooting effects. Lots of people have tried different things for various sequences on various films, but this was the first time we could encode all camera data for the entire movie — everything, even moving dollies and spinning, special effects camera rigs that we built, all that stuff. It was a big step for us, and faster on-set and more user-friendly to the first unit than motion control would have been. In particular, it helped [director Alex Proyas] compose and choreograph shots from the real set to the completely digital set in realtime.

“The scope of I, Robot was so huge we had to previz the entire movie. In addition to using the EncodaCam, we had extensive previz work done for us by Image Engine in Vancouver [British Columbia] and Pixel Liberation Front in [Venice, Calif.].”

Up Next: Which of the other films made pre-visualization strides?

Millimeter: Which of the other films made pre-visualization strides?

The Day After Tomorrow required extensive environmental work and the development of dozens of proprietary tools at several facilities for sequences like this one, showing tornadoes devastating Los Angeles. (Photo © 2004 Twentieth Century Fox)

Karen Goulekas: “Our early previsualization work might have been the most important thing that we did on [The Day After Tomorrow]. I originally hired a team of nine people — one Final Cut Pro editor and eight previz artists using their own computers and set them up in [director] Roland Emmerich's office. We bought a model of New York City and sections of Los Angeles from Urban Data Solutions, which gave us accurate building and floor heights of the existing buildings. This enabled us to create previz camera moves that were relative to the real-world environments we were working within.

“Eventually, Joshua Kolden, who led the previz effort, formed his own boutique operation, Crack Creative [Los Angeles], and utilized that same group of previz artists. We ended up using them for previz well into postproduction, and also hired them for monitor composites, as well.”

Millimeter: What special tools were developed for these projects, or existing technologies that you improved or used more extensively than in the past?

Matt Johnson: “Cinesite's [London] inhouse development team was crucial for us. They gave us lots of new shaders running through RenderMan and Maya to create the ice and the proper geometry. Those tools used by our artists gave a proper sense of depth to scatter the light through the surface of the ice.

“Cinesite also developed a new AI crowd simulation software for our battle sequences, called REACT. It was built specifically to work within Cinesite's Maya and RenderMan pipeline. What was particularly nice about the software is that it is fluid — you can make changes and see a render of those changes quickly. It doesn't take days anymore.”

Karen Goulekas: “Almost all of our shops developed proprietary tools for effects like water, twisters, super cells, and so on. For example, The Orphanage [San Francisco] used a voxel-based 3D volume rendering technique to make the super cells appear to be sucking in things and pushing down the atmospheric cloud layers surrounding them.

“For the ice-shelf sequence, Hydraulx [Santa Monica] also did a great job developing shaders and using subsurface scattering techniques for the ice shelf and icebergs they created. That is an approach usually associated with human flesh rendering and the way light passes through the areas of translucency. The same need applies to ice since it has so many layers comprised of various opacities.

“Our use of LIDAR laser scanning technology to collect precise data for creating a virtual New York City was also pretty impressive. Paul Maurice and his team from Lidar Services [UK] scanned 13 blocks of New York over the course of three months. At the same time, we had three teams of photographers shooting over 50,000 photos of buildings to be used as textures.”

John Nelson: “We had all sorts of special-purpose tools for building the lead robot in I, Robot that Digital Domain developed using RenderMan and their inhouse compositing program Nuke. Rendering the robot for sub-surface scattering and compositing him so that he had a translucent quality also was a difficult task that Digital Domain did a great job with. There are thousands of robots in this film, though, so for AI work, we relied on Massive software from Weta Digital, our other major vendor, which developed that technology originally for Lord of the Rings.”

Visual effects artists created digital armies and a digital Greek armada for Troy and blended them with plates shot on location in daylight during the compositing phase. (Photo © Warner Brothers)

Nick Davis: “Moving Picture Company's ALICE software played a big role on Troy. They tweaked the software to govern the movement of soldiers using motion-capture clips. It's essentially a mo-cap blending tool that lets us blend frames from any motion-capture clip in realtime to seamlessly create motion for individual soldiers. It then lets us render it out in a realistic amount of time, which was key because we had a huge amount of shots and about 70,000-plus soldiers to control. We needed them to follow paths, conform terrain, collide, and fight with other soldiers. I think they took about 10 months to write and develop that software from the ground up.”

Roger Guyett: “Tools for creating the Buckbeak creature basically had to be created from scratch, especially the system that controlled the dynamics of the feather movement. There was an incredible amount of R&D that went into that [at Framestore CFC, London]. They have close-ups of the animal flying, wind traveling through the feathers, and so on, so they had to both design the feathers to be realistic, with some broken or longer or shorter than others, but also they had to design the system to have the feathers react to the dynamics of wind and the environment. Every feather has to push and react to all the feathers around it. That's a very difficult computer problem to solve. There were over 50 shots of the creature in the movie, and that is work I'm very pleased with.

“For the Dementors, we developed a lot of tools to allow us to control the way the cloth moves since the cloth is meant to be almost an extension of the creature. We actually built a new cloth simulation program around the program Industrial Light & Magic already had, which gave the animators the ability to put character and personality into the way the tendrils of cloth moved over the skeleton of the creature.”

Scott Squires: “The werewolf transformations in Van Helsing required our creature team to do a huge amount of R&D. Our starting point was actually the work we did for The Hulk to show that creature's ripping clothes as he transforms. We basically re-engineered it for our particular needs. Our team wrote a specific tool for ripping to occur naturally — as a body part changed, it would rip in its own unique way, as compared to other body parts. It was very complex.

“The scene showing the attack of Dracula's brides was also very complex.” (See the Step by Step column in the June issue of Millimeter for a detailed look at this sequence.)

Up Next: How has the effects industry evolved or changed around the globe during the time you were making these movies?

Millimeter: How has the effects industry evolved or changed around the globe during the time you were making these movies?

Nick Davis: “One thing I've noticed is London's growth as a mature visual effects center. All the facilities in London have grown enormously. Framestore CFC has [approximately 150 artists], Moving Picture Company has about 300, and Cinesite is really growing also, and there are some newer boutiques on the scene. This means an enormous amount of work can all be done in London simultaneously as it is done in Los Angeles and elsewhere. All these facilities were working, for instance, on Harry Potter and other films at the same time they were working on Troy.

“They are actually attracting talent back to town. There was a period where there was a drain on British vendors with artists going to Hollywood and Weta in New Zealand during the Lord of the Rings period. Now, they are coming back and bringing with them tremendous experience from working at places like Weta and ILM. That's one reason I think you are seeing the better productions coming here now — the British vendors are now on par with anyone in the world.”

Karen Goulekas: “One thing I learned on The Day After Tomorrow is that the fall in the price of computer software and hardware has made it possible to go with smaller groups to do specific jobs. We originally had the entire film at Digital Domain, but eventually realized the work would be best served if we spread it all around to various vendors.

“When we assembled the previz team, for instance, we brought people in who had their own computers. We even offered two different salaries — one wage if you brought your own machine and another if we had to buy you one. Now people are setting up four walls and putting together facilities. Most of them can afford software and have talented people to R&D it, and they can be effective, like Crack Creative was for us on this show.”

Millimeter: What other interesting trends are you seeing around the industry?

Roger Guyett: “Certainly, the advent of digital color timing for entire movies is changing things, including for us visual effects people. I certainly think it won't be long before all movies are digitally timed. And once that happens, the line of responsibility between color timing and visual effects will become more blurry. The difference between ‘real’ effects work and what they do in the digital intermediate as a matter of course will change things. The DI people are already doing wire removals routinely now. That stuff might seem trivial by nature, but on the scale of these big effects movies, it's an important part of the process.”

The I, Robot camera team films Will Smith on a greenscreen stage during production. (Photo courtesy 20th Century Fox)

John Nelson: “Entirely digital lead characters interacting with real humans is something our movie, and a couple others, have proven is possible. It's complicated stuff, but techniques have been developed to shoot clean plates without motion control and using a proxy actor to stand in for the digital character, and they work very well. I think you'll see this happening more frequently in the near future.” (See Step by Step on I, Robot for an in-depth look at creating the CG robot Sonny in I, Robot.)

Scott Squires: “Creature work overall has certainly gotten much better rather rapidly. Skin rendering technology is much better, and hair remains difficult, but has made leaps and bounds. It depends, of course, on how humanoid, or non-humanoid, the creature is in terms of developing a successful technique, but there are now several different ways to do it.

“For Mr. Hyde in Van Helsing, for instance, that character is all CG. But we learned from past experience to use a proxy actor, so we had Shuler Hensley, who played Frankenstein in the movie, come in and give us a reference performance on-set to capture plates for the CG version. We had him pantomiming and things while we held a Hyde head over his head to simulate the creature's height. We were able to choreograph the animation better this way, and we got good reference data for lighting the skin and so forth. Using a visual guide seems obvious now, but just a couple years ago this was much more difficult to do in a realistic way.”

Matt Johnson: “Long-distance collaboration is now far easier. We used a system from Media.net (see next month's issue of Millimeter for more on this system's use on King Arthur), and it was extremely useful on this job. We were able to work, in realtime, with producers at Jerry Bruckheimer Films in Santa Monica to discuss shots. The director and editor could look on their Avid while we were looking on ours and making notes, all in realtime, while videoconferencing with them. Among other things, this saved me a lot of jetlag. I didn't have to make nearly as many trips to Santa Monica as I normally do on such shows. This makes trans-Atlantic filmmaking a lot easier because it preserves that important element of human interaction.”