Mummy Reboot

Director Rob Cohen wanted a different feel to his Mummy movie, The Mummy: Tomb of the Dragon Emperor, so he brought in an entirely new creative team.

Coming off of The Mummy: Tomb of the Dragon Emperor — the biggest, most expensive, and most complicated project of his career — director Rob Cohen marvels at the changes in visual effects that have occurred in the 12 years since he directed his first effects effort, 1996's Dragonheart.

“Back then, we were dealing with full roto — ILM[Industrial Light & Magic] would have been happy if I never moved the camera,” he says. “We were trying things that weren't done much in those days — shooting a creature in daytime rather than hiding him at night, putting more reflective surfaces on him, giving him the personality of an actor [Sean Connery], and so on. Interactivity was very difficult. The dragon — if we wanted him to touch something, we had to rig it. If he cut a tree with his tail, we had to have a separate [physical] tree-separating gag for perfect timing on the set. There was no flexibility.

“Now, that's all changed,” Cohen says. “In [The Mummy: Tomb of the Dragon Emperor], we do any kind of interaction — with water, wood chips, clay shards. We have martial-arts contact between spears and [CG] terra-cotta warriors. The amount of contact in that sequence alone was impossible in 1996. In 12 years, it really has become a whole other digital universe.”

In addition to directing the film, Cohen served as chief for the visual-effects effort for the third movie in the Mummy franchise — work done primarily by two visual-effects houses, Digital Domain and Rhythm & Hues (R&H). One requirement of that work, and for the movie's visuals generally, was to get far away from the look of the first two Mummy movies — directed by Stephen Sommers, who serves as a producer on this project — and move in a different direction.

“That's not to say I didn't like those films — I have total love and respect for Stephen,” Cohen says. “I'm proud to say I think what we achieved builds on what he did in the first two films in sense of the entertainment value, but it shows a franchise can have new muscularity and strike off in new directions and doesn't have to be the same thing that came before. I told them when they asked me to direct the movie that I was going to do my own movie, not connected to Stephen's work — his technique or sense of humor or visuals — or the effects houses, or anything. It's a reboot of the franchise. I wanted it to be more textured and historical and edgy and intense.”

Extensive sets were built on backlots in China in an effort to minimize CG sets, with a gold-hued and red-hued palette emphasized.

Thus, the movie pushes the franchise away from Egypt and into China for an entirely new visual palette and story direction. In keeping with that move, Cohen hired a new creative team for the franchise. He brought in cinematographer Simon Duggan to shoot the piece, Kelly Matsumoto and Joel Negron to edit the movie, and Nigel Phelps to handle production design. He then asked Digital Domain's Joel Hynek and Matthew Butler and Rhythm & Hues' Derek Spears to head up visual effects, working strategically to link the two facilities for the duration of the project while they crafted the bulk of almost 1,000 digital effects shots for the movie.

“Digital Domain was my preferred effects house before this project, and because of the sheer volume of material, we knew we had to pair them with someone who would be a good fit,” Cohen says. “Rhythm & Hues and Digital Domain are physically close to each other [in Venice, Calif.], and [R&H] was the other logical independent company to think of, considering their experience with creatures. I asked the two to commit to work together, including the sharing of proprietary software and data, and that's what they did.”

Cohen and Phelps designed a basic Chinese palette for the film, according to the director. “A lot more gold, a lot more red, a lot more warmth,” Cohen says. “A lot more texture. Nigel and our Chinesedesign team gave us an art department that I really think gave us more detail than you've seen in a movie in a long time. It was almost like going back to the days of the studio system, when you had all these artisans on hand. That's what we had — it was pretty amazing.”

During a production cycle that took almost a year in various locations in China and the Montreal area, Dugganshot the film using two lightweight Arricam Lites, an Arriflex 235, and an Arriflex 435 Xtreme high-speed camera, shooting Super 35 (2.35:1) with a set of Cooke Optics S4/i prime lenses and a full set of Angenieux Optimo zoom lenses. In desert settings and on massive sets built on various backlots in China, the shoot was a particularly grueling one for the production team, according to the DP.

“For a brief moment in the pre-production stage, we looked into shooting with a digital medium, but decided to go with film,” Duggan says. “In hindsight, we made the right decision as we were often shooting with very high-contrast exterior lighting conditions, such as backlit snow and desert sands. The cameras were also subjected to the most physically arduous conditions during our shoot. The snow-covered Himalayan sets were created using tons of Epsom salts, and snowflakes were created with detergent. Some of the visual-effects team's computer equipment would break down in this corrosive atmosphere. And in China, the equipment was sandblasted by windstorms a few times.

“Rob wanted very dynamic visualsfor the film, and the cameras were mostly in handheld mode or flying around on the end of a Technocrane. We also imported a remote-controlled Ultimate Arm [camera crane] and attached it to a Mercedes four-wheel-drive tracking vehicle that a specialized crew of four drove around in China. We used that system extensively on all of our [chase scenes through the streets of Shanghai], when the heroes are chasing after the Emperor's chariot. We got some amazing shots not possible with a standard crane-mounted vehicle.”

Visual-effects house Rhythm & Hues, which teamed with Digital Domain on the project, was largely responsible for the film's creature work, including Yetis and a three-headed dragon.

CG sets were largely avoided, and under sometimes grueling conditions, Duggan's team spent much of its time capturing effects plates during principal photography without the benefit of motion control and, often, greenscreens.

Indeed, the DP and the visual-effects team all say that one of the great leaps forward in the state of the visual-effects industry is the ability to capture workable plates through the natural flow of the photographic process, thanks to great confidence in advances in tracking software and a growing range of post options.

“We tended to use as little greenscreen as possible, and when we used it, it was to enable set or location extensions or add additional background elements such as CGI characters behind our actors,” Duggan says. “At times, we were shooting in gale-force winds in a sandy desert location outside of Beijing.Setting up greenscreens there was sometimes impossible, and our visual-effects crew had to make the decision, at times, to rotoscope our characters when scenes required the addition of background elements behind them. At times, complete CGI backgrounds were created from the still photographic references taken by the effects crew at those locations.”

“Long gone is the day of motion control and screen work for this kind of shoot,” Hynek says.

“Here we go to the desert [in China],put together a package of what we need to shoot for this or that, and they say, ‘Don't bother bringing those big greenscreens,''” Butler says. “‘You'll have to use techniques to extract and composite layers.'' That means we let Rob do what he wanted and we pulled [mattes] off that. In our case, we improved on techniques we have used for a while — combining [The Foundry's] Nuke compositing package, which allows us to incorporate rotoscope splines in the composite, combined with optical flow techniques for perspective. Then you can post-apply motion-blur attributes to those profiles to create a matte that is very similar to a scene-derived matte, and it ends up looking very good.”

Cohen had Digital Domain and Rhythm & Hues split up most of the effects shots based on each company's specialty. Thus, most of the creature work — most prominently, a three-headed dragon, Yeti creatures, and a Nian beast that is essentially a combination of a giant dog and a lion — was handled by Rhythm & Hues. Digital Domain focused on major battles between rival armies of undead warriors and development of the terra-cotta-styled warriors and the digital, Jet Li-inspired, shape-shifting, Chinese emperor mummy who serves as the movie's villain. Many times, however, the two facilities shared shots, with R&H creatures sewn into Digital Domain battle shots or Digital Domain's emperor mummy brought into contact with the creatures created at R&H.

Spears, visual-effects supervisor on the project from Rythm & Hues and a seasoned veteran of big CG pictures, says even by the standards he's used to, The Mummy: Tomb of the Dragon Emperor is particularly notable for the scale and volume of the visual-effects requirements.

“The sheer complexity and number of shots probably makes this one of our tougher shows,” he says. “The biggest problem for us was the scale of what is going on, and the schedule for getting it all done. There is a lot of interaction [between CG characters] and the real world in this film, for instance.”

Li is seen in human form only briefly at the start and end of the picture; he is a synthetic character for most of the movie. Developing his mummy proved to be one of the key challenges involved with designing the entire movie.

“In his mummy form, he and his whole army are terra cotta — meaning they are supposed to be [clay-like] inanimate objects that come to life,” Butler says.

“Originally, the Jet Li mummy was supposed to be terra cotta the whole way through, and that made him not all that expressive,” Hynek says. “The idea of the terra cotta moving and cracking meant it limited how expressive the character could be. So Rob thought that might be too distracting for the entire movie, and asked us to try something to differentiate him a bit. So we created the ‘under-mummy'' idea — the notion that underneath the terra cotta is the real emperor character, all burned and desiccated, so eventually he breaks out and we see him like that.”

Massive software was used to help animate the epic battle scenes featuring the stylized terra-cotta warriors featured in The Mummy: Tomb of the Dragon Emperor. Filmmakers went through months of research to build the "brain" for the program—including the use of video reference, motion capture, and behavioral data from some of the film's actors.

Both companies animated the emperor mummy at various times, each facing the challenge of how to evoke the familiar and distinctive movements of Li. This challenge was further complicated by the fact that Li's availability to the production was severely limited due to a scheduling conflict, thus limiting the filmmakers' ability to run him through detailed motion-
capturesessions.

“We had him scanned and all that, but the character's movement is largely from pure imagination,” Cohen says. “The original idea was that the clay would have this basic blankness that true terra-cotta [statues] have. In other words, the eyes are not articulated. But we had to rig the face to have total expressiveness. So the scan and Jet Li's vocal performance [and video reference] were key to getting us the reference we needed.

“To be honest, [the actor] was only available for one scene in which he played a terra-cotta warrior. But I felt it was better to make him a CG character for the middle part of the movie anyway, because no matter how good makeup might be, he wouldn't have the same power for what we had in mind that a true CG creation would have,” Cohen says. “So that is how the idea that he was two characters got developed. He had a clay terra-cotta warrior shell and the real burned and disfigured character underneath.”

Among other techniques, for close-up shots of Li's emperor mummy, Digital Domain used the so-called “witness-cam” technique. Essentially, the company used two HD witness cameras to film his face during principal photography while Duggan's unit was recording him with a film camera. This gave filmmakers Li's facial performance from two wide angles (almost 45 degrees apart). Company artists were then able to take the HD footage and sync it together with the primary film camera's footage to produce three distinct views of his facial movements.

“Then, using our proprietary [Sci-Tech Award-winning] tracking software [dubbed ‘Track''], we were able to identify and track unique features through each of these cameras,” Butler says. “For example, we could locate Jet Li's right eye as one point, visible through all three cameras, and then use triangulation to give us an accurate 3D location for that point in space relative to the main camera. After locating several of these points through each camera, we were able to accurately constrain a model of Jet Li's head to the main camera image. The addition of two extra views of the action allows us to ensure that the 3D position of the CG Jet Li was accurate on [scenes such as] when mud is flowing from his eyes and nose.”

The texture and general look for the terra-cotta mummies, according to Cohen, was meant to evoke “a liquid-solid approach in which the terra cotta cracks at certain stress vectors that would happen when force was applied, and then there is a sort of molten generation of terra cotta that reseals it. That infinite amount of cracking and re-sealing creates the terra-cotta look. So the emperor and the terra-cotta army work on a slightly different set of aesthetics,” Cohen says.

It's not only the terra-cotta army that awakens from the dead, but also their horses, which crack out of their own bronze shells at one point. According to Rhythm & Hues' Spears, the presence of different kinds of shells presented filmmakers with the question of exactly how their fragile exteriors should crack.

“There's two types of cracking really,” Spears says. “The terra cotta has a different type of crack than how metal would crack. Terra cotta has more of a clay fracture, resulting in ragged edges. That work was largely done by Digital Domain, and then we re-implemented it with our own internal systems. Then, we had a different type of cracking for [horses]. Both approaches are meant to relieve stress, so that they wouldn't have stretching textures or look like they were made of rubber. But the horses are made of bronze, and therefore have a crisper crack. It's a more subtle differentiation, but we built our systems to allow a little give and take between stretching and cracking, so that it wouldn't look like a collection of spare parts running around and would retain some shape coherency. Our internal system essentially does collision detecting and then cracking based on stress factors. Those are actually brand-new tools we developed on this show — an extension of some of our cloth and physical systems that are built into our animation software.”

Digital Domain and R&H usedAutodesk Maya on the project. Digital Domain used Side Effects Houdini for procedural-based, natural-phenomenon-type effects and Nuke, the compositing package originally created at the company, as well as its Track software and its Storm volumetric rendering package. Rhythm & Hues, meanwhile, also relied on its own suite of proprietary tools, including its Ren internal rendering technology and its Icy compositing system.

Another familiar tool also played a key role during the climactic battle between rival armies of dead Chinese warriors. That tool, of course, is the famous Massive AI software that is used for animating large crowds in an organic way. That technology, first made famous on the Lord of the Rings trilogy, is still doing its thing quite nicely for applications such as the battle scene in The Mummy: Tomb of the Dragon Emperor, Hynek says. Digital Domain was able to create “quite a sophisticated brain” for manipulating the warring armies.

“It was my first foray into Massive, actually,” Hynek says. “But it was astounding to see scenes come alive with it. The thing about it is you don't know what it will look like when you put the parameters in there. The actions it came up with for both armies were quite startling, largely because it is so unpredictable. The hard part is building the brain — that took us many months. But, once you do that, they really start rolling nicely off the conveyer belt.”

Building that brain, however, is no simple task, Butler says.

“You have to design the options, and decide what to go out and capture and add [to the artificial intelligence],” Butler says. “We did research; we did video shoots; we went to Montreal where they were shooting and got behavioral characteristics from some of our actors. From all that, we built a familiar tree of points that the brain could feed from. Then, you find out what the director wants for a particular scene when you get there, and you might augment things with some very specific mo-cap animation or hand animation if you need to, and then you can achieve all those shots. It's pretty powerful what it can do.”

With the production taking place in Canada and China; the editorial happening (incorporating Avid Media Composer Adrenaline HD's DNxHD 36 codec) in Los Angeles; film being transferred at Technicolor in Burbank, Calif.; the visual-effects process ongoing at the two Los Angeles-based facilities; and dailies and the digital-intermediate process happening at Company 3,Santa Monica, Calif., there were — understandably — lots of IT issues to grapple with in moving data around. But just as important, the filmmakers say, was having a common approach for viewing dailies imagery at these disparate stops along the workflow highway. Duggan, in consultation with Cohen, took charge of distributing a common template for the basic color palette as a first step in this process.

“During the shoot, I took hundreds of digital reference stills of each scene, simply grading them in Photoshop and distributing them each day to Rob Cohen, the dailies colorists, and the visual-effects teams,” Duggan says. “Rob was happy with how it looked, so this gave everyone a fairly clear visual direction. In Montreal, the crew watched projected HD dailies, and the quality was great for technical checks such as focus and makeup. In China, we were unable to turn dailies around fast enough, so Company 3 set up a password-protected link to their server so we could access any of our dailies at any time.”

The look-up table (LUT) component to make sure that what was being viewed at different locations remained consistent on the various 2K projectors used at the different facilities was Filmlight's TrueLight color management and calibration system.

“That was important — both effects companies and Company 3 got together to work out a common format and platform so that we all had a common LUT and viewing environment to see how our digital images will look on film,” Butler says.

Cohen collaborated with colorist Stefan Sonnenfeld at Company 3 to establish the final color of the piece, because Duggan was off shooting another film during the DI. For the most part, the color template for sequencesranging from the snow-coveredHimalayas to warm, desert battlefields to the streets of Shanghai were well-established during the dailies process.

But Cohen, like many of his contemporaries, says he was thrilled with the DI experience on this project, and ingeneral.

“The idea that we used to sit in a screening room with a timer, and as the film unspoiled, we yelled ‘quarter point cyan,'' or ‘I don't want the green,'' and by the time we did one note, six to eight shots had gone by — I mean, it was primitive,” he says. “And I was doing it that way as recently as The Fast and the Furious [in 2001]. I did my first DI with xXx [2002]. Now, we can define a wall, take off some rim light, open up someone's face, shift that person more toward gold — you can make infinite moves on the image; you can virtually relight it. So, I don't miss smelling those chemicals — I have no nostalgia for it whatsoever.”