No Clones: Unexpected Adventures on Episode II

Click
here to read about the pipeline for Episode I

A major change on this film was ILM’s switch to the Linux operating system for animation of CG characters like Yoda.

Episode II in the Star Wars franchise might be titled Attack of the
Clones, but for Industrial Light + Magic it was hardly a
“copycat” production.

ILM, the image tech arm of Lucas Digital, developed a host of new
techniques for director George Lucas' first HD feature project.
Executing this breakthrough production required innovations nearly
every step of the way — from software that helped the HD shoot
run smoothly to server hardware that enabled ILM's visual effects
artists to work at a level of precision they'd never had before. It was
a process that brought some surprises, and that will undoubtedly
influence how ILM creates images for projects to come.

The R&D behind this picture began more than six years ago,
according to ILM's principal engineer, Fred Meyers. Yet, right up until
the week before shooting started, the team was blazing new ground with
prototype equipment. “We were building it, or squeezing the last
bit of function out of it as we started to shoot,” says Meyers.
Meyers, who served as Lucas' HD supervisor, was the project's
“technology guru,” observes Michael Cooper, ILM's director
of film and editorial services. Cooper's video engineering team worked
with Meyers to come up with solutions that just weren't available when
the project started.

“When we didn't have vendor-supplied solutions, we needed to
make sure that what we were building was online before we needed
it,” Cooper recalls. “A few times it was mighty close. I
think the production cameras arrived from Sony a day or two before
George started rolling. We had tested a number of cameras at ILM,
thinking they were the production cameras, but when first unit arrived
on location in Australia they got a new set of cameras.” Although
getting improved gear was welcome, it demonstrated that in some
respects, the engineers were aiming at moving target.

Although known for post-production effects expertise, ILM played a
key role in facilitating the HD capture process by creating a graphical
camera interface that they called the Camera GUI.

“It was essentially a software program written by David
Nahman-Ramos that allowed us to set the camera up,” Cooper
explains. “We could know, for example, where the black level was
set, just in case something got tweaked a little — because it
would get tweaked. Instead of chasing our tails wondering why one image
was not falling in with the others, we could see how the camera was set
up and make adjustments.”

ILM's system was operated discreetly during shooting, hidden inside
Fred Meyers' control tent. “We built a portable system that was
kept slightly off set,” recalls Meyers, “but it had all the
equipment needed to calibrate the color monitors, to put them in an
environment where set lighting was not a problem, and also to have
control and repeatability. The camera control software allowed us to
attach a connector to the camera and know we had all the settings. We
were capturing data almost in the same way that we capture data from a
film scan. That did require quite a bit of technology. The software
controlled the Sony cameras and basically took a snapshot of every
single setting. The camera operators didn't have to look at pop-up
menus to make sure things were dialed in. They focused their cameras,
set their stops, and proceeded as they would on a film-style shoot. The
strategy all along was that we would simply replace film technology
with HD and proceed as we normally would. If anything, we probably sped
up the process, doing 36 setups a day. We didn't have camera problems
that slowed us down.”

For scenes where actors rode CG creatures, ILM created digital doubles that replaced all or part of actors’ bodies. New clothing animation software was required to achieve this.

Throughout the first unit shoot, ILM visual effects supervisor John
Knoll made close observations of the unique characteristics of the HD
images because so many of them would require effects later. Knoll,
along with Pablo Helman, Dennis Muren, and Ben Snow, supervised ILM's
2,000-plus effects shots.

“We had more depth of field with the HD. From the production
standpoint it was an advantage to shoot in HD because the lighting
package could be smaller,” Knoll observes. “But there were
surprises. Because the film isn't jumping around in the gate and you
don't have all that flicker and grain, the images are super-steady and
vibrant. Sometimes we'd be looking at a close-up of an actor and his
makeup looked too obvious, or we could see how a set was constructed.
Film mushes things together in a way that HD doesn't.”

That lesson was reiterated when ILM's miniature and model
photography got underway back in Northern California.

“All of us who do this work have gotten used to how things
look on film and the level of polish you need to put into something for
it to work on screen,” Knoll admits. “It's a bit different
in HD. When we were working on our first miniatures, it was a bit of a
shock. We had to really clean them up! But since we were seeing them
live while we were shooting, we could see what wasn't working. We
didn't have to shoot wedges because we were looking at the live feed
from the monitor. There were never any questions like ‘Are we
going to carry depth of field on that?’ As soon as we wrapped we
could take a look. If it was OK, we could strike the set. It was
instant gratification.”

Since the HD camera didn't have a time-lapse mode to accumulate long exposures, ILM had to light models a little hotter to shoot in real time. That required building brigher lighting into some models.

Although this capability improved efficiency, there was a downside.
“We had to light our miniatures at a level to shoot, essentially,
realtime,” Knoll explains. “The HD camera didn't have a
time-lapse mode to accumulate a longer exposure, so we had to light a
little hotter. That created a problem for any models with internal
lighting because we had to build brighter lighting into them. In some
cases, we couldn't do the equivalent of stopping down to f/32 to carry
depth of field. Even if we could, we couldn't put enough light on the
model without burning it. In some cases we'd shoot multiple passes
— a foreground focus and a background focus and split them
together. It wasn't the absolute ideal system, but I think the
advantages outweighed the disadvantages.”

To help with the process, ILM's engineers developed post-processing
software that could simulate long exposures. Meyers explains,
“Since the cameras have an electronic shutter, in turning off
that shutter we could simulate what we called a 360-degree shutter.
That allowed us to blend multiple frames together to build exposure,
reduce noise, and get shots that could be shot at 12fps or 3fps. Some
of the software techniques for manipulating multiple frames had already
been well tested. We were able to put those in, although it wasn't
simple. We did have to build some processes with the way the tape
material gets loaded onto a disk system and then gets pre-processed
before it goes into our CG environment. That was all streamlined
specifically to handle all the motion-control work. For example, if
they decided to do a camera move at 3fps, we had a formula that we
would type in, and when that material was loaded it would pre-process
that automatically. The other thing that had been done was that we
synchronized the HD camera to the motion control system so that we
could lock-step the exposure with the camera move.”

To the delight of ILM's effects team, images from the model stage
could be projected live in the company's screening room, which was
equipped with a Texas Instruments DLP projector from Christie connected
to ILM's own custom-built server.

“We implemented systems such that you could be in the theater
seeing the image up on the big screen while it was actually being lit
on stage,” Meyers says. “The signal coming out of the
camera is your image; there's no telecine, no transformation that needs
to happen. We took advantage of the fact that digital projection was to
the point that we could review material digitally projected without
having to go out to film.”

The HD cameras captured so many small details that ILM's models had to be finished to a very precise degree.

Knoll recalls, “One day we were shooting a miniature on stage
when George came to the screening room to review some stuff. So we
patched a feed from the stage to the big digital projection in the
theater and showed it to him. It was a model of a room that had a big
scanner in it. George asked, ‘Does that scanner turn?’ So
one of coordinators called up the stage, and all of a sudden we saw
shadows of people moving in the background. Then the scanner started
up. It was hilarious because it was live. We're so unused to that
experience. Usually when you're in a theater looking at an image, it's
permanent and unchangeable — something that happened long ago
instead of happening at that very moment.”

The ability to call up a shot and look at it in on screen with no
turnaround time was perhaps the most eye-opening aspect of this project
for the creative teams from both ILM and Lucasfilm. “If George's
editors wanted to see things in full resolution in a theater they would
come here,” Cooper recalls. “They utilized ILM almost as an
arm of editorial.”

The technology that enabled this approach was, to a significant
degree, built by ILM's engineers, including the HD conformer and the
video servers. ILM also built the databases to track every frame of the
movie and sync it up to Lucas' Avid cut.

“What Lucasfilm had in their cutting room,” Meyers
explains, “were the down-converts that we had done either in
principal or back at ILM, and it was all tied via the time code from
the tapes.” ILM's engineers had extended their tools for
accepting time code, which allowed them to put a tape into a machine
and have it cue up and load whatever shots they wanted to see right
onto the server. Whenever ILM added model shots and CG elements to the
picture, Meyers notes, “We could tie back into Lucasfilm's
editorial and give them the updated takes with the combined elements.
We maintained sync between editorial and ILM's elements all the way
from start to delivery.”

Handling all this data required huge servers. “We ended up
with 14TB in editorial alone,” Cooper admits. “We kept tons
of images in full HD online and accessible to our artists. Our video
engineering people built an incredible database so that everybody, no
matter what platform they were on, could pull up thumbnails of any shot
they wanted. When someone made a request, we could pull it off the HD
server and deliver it in full resolution. This allowed the visual
effects supervisors to see all of their shots in sequence with
surrounding shots.”

Animation Director Rob Coleman (foreground) examines the digital version of Yoda with CG Model Supervisor Geoff Campbell.

For Knoll, “It was a great experience to be able to see a shot
full-res in dailies, be able to scrub through it and spot all kinds of
things that probably would have burned us later if we didn't spot them
early on. Our usual process — running low-res for a while then
later doing hi-res crops, and then finally running the shot full-res
— that process has changed. We started running things high-res
much earlier on. Being able see the work at full resolution in the
theater has me totally spoiled now. I don't think I can ever go
back.”

That's a sentiment echoed by ILM animation director Rob Coleman, who
was responsible for the cast of CG characters — including an
all-digital Yoda — who populate almost 70 minutes of animation in
Episode II. “Having HD servers that can play back high-resolution
images at speed has spoiled me too. I was able to see, sooner in the
process, what my animators' subtle bits of acting looked like. The
twinge of a muscle under an eye or the smallest glance can tell you a
lot about a character. But that doesn't show up on the resolution of a
27-inch computer monitor viewed from 18 inches away, and I found I was
asking animators to go over the top with their animation. You can
create an eye movement that's a couple of pixels on a monitor, but when
you blow it up it's a huge eye movement, and you end up with a stylized
performance. I was more confident that I was making the correct
decisions by seeing it in high resolution. I could make decisions
before it was too late.”

Coleman had to make some fine discriminations on the CG in this
picture because many of the digital actors were wearing complex,
multi-layered costume — often while standing beside real actors.
“We needed R&D to rethink how we do clothing. Our clothing
simulation team spent months figuring out how to cut and drape digital
garments.” The result of this effort was a kind of tweakable
simulation software, which Knoll describes as “a simulation
engine that could go in and out of preset poses.” Because the
movie makes extensive use of digital doubles for stunt shots, he notes,
“We needed to be able to cut from a live actor to a CG character
and back. So the way a robe draped was constrained at the beginning of
some shots by the live-action photography. We had to sculpt that shape
and begin our simulation with that — and have it not look
awkward. We didn't want to see any seams.”

ILM's improved clothing software was crucial for shots that combined
real actors with CG body extensions in the same frame. “We have
live-action actors riding CG creatures,” says Knoll. “When
we were shooting the bluescreen elements, I had the actors riding a
blue shape that was roughly the contours of the creature we would later
animate. Rob Coleman and I did the best we could shooting these
elements, but it's really, really hard to animate a CG creature that's
constrained in its movement — and make it look natural. It's also
hard to make the actor look like he's really sitting on the creature
and not sliding around. In some cases, it was easier to cut the actor
at the waist and replace him with computer graphics from the waist
down. Then we could do really tight interactions between his CG legs
and the CG creature. In a couple of places the split line moved up
higher and ended up being just the actor's head with the rest of the
body CG. We used digital doubles a lot, which I hadn't anticipated in
advance.”

Both Knoll and Coleman approached Episode II hoping they would be
able to build upon what they'd achieved in Episode I. Both got pleasant
surprises, from very different quarters. For Coleman, it was the
arrival of computers running the Linux operating system, which
presented a departure from ILM's longstanding use of Unix-based
computers. “They asked if they could put low-cost Linux machines
[Dell P4s] on our animators' desks,” recalls Coleman. “The
animators still had their traditional O2s running Unix versions of
Softimage and Maya. They could work on either computer. We found that
Softimage ran on a Linux machine eight to 10 times faster. So I could
see more iterations of each shot because animators could turn things
around quicker. I was able to get this movie done faster because of
Linux.”

Unexpected for Knoll was the latitude that HD offered Lucas to do
one of his favorite things — to select a portion of a frame that
he liked, blow it up and reposition it to create a new image.

“One of the things that we wanted to nail down up front was
what the limits were in HD of blowing something up,” Knoll
recounts. “On Episode I, we found out that for something shot on
anamorphic 4-perf we could do about a 17% blowup. For stuff shot on
VistaVision we could go up to 30% because of the bigger negative. When
we tested HD, we were stunned to find out that almost universally you
could do a 30% to 40% blowup. That went completely against my
intuition. I thought it would end up being less because of the JPEG
compression. I don't have a good explanation for why it held up better
than film. I think the noise characteristics are different from film
grain. You have to be shooting on a pretty fine grain film to get the
noise equivalent of HD.”

Any material that could be captured in HD was, but that didn't
include pyro elements. As Coleman observes, “Typically you shoot
miniature pyro at a very high frame rate to create a sense of scale
because you're comping it to be an enormous pyrotechnic event.”
Creating digitally simulated pyro wasn't really an option because, as
Coleman observes, “that's still time-consuming to do in computer
graphics. John Knoll felt that he needed a real element to achieve the
realism that he wanted.” The only alternative, Knoll reports,
“was to shoot on film when I needed to run at 300fps. I didn't
have an HD camera that could do that.”

Lucas hopes that Episode II will be shown digitally in as many
theatres as possible, but the overwhelming majority of moviegoers will
see it projected on film. One of ILM's jobs was to carry out Lucas'
mandate that all of those film prints reflect the best quality
possible.

To achieve that, explains John Ellis, ILM's manager of imaging
services, “We're taking a digital master and creating multiple
negatives, and from those the lab will make first-generation prints.
Filmmakers typically strike an IP and then, after most prints are made,
they'll make maybe six prints off the original negative. This film will
have thousands of prints off original negatives. If theaters can't go
directly to digital projection, it's the highest quality they can
get.”

To create those multiple negatives, ILM took a unique approach. As
Knoll recounts, “It occurred to David Tattersall, our DP, that if
we were going to take the entire picture through our system, and it all
had to start with a digital master and get filmed out, that should we
consider doing the conform digitally and then filming out entire reels
at once. By doing that multiple times we would make our master printing
negatives directly. We all thought ‘Wow, we could do that.’
We'd need to have some new systems in place to do it, but it would be
great because we'd save two generations. We started looking into how
practical that would be. We'd have to have a method for conforming the
whole movie at full HD resolution and do color timing in-house. Then
we'd have to do these extremely lengthy film-outs, to ensure there was
no color drift. We tested this idea with the movie trailers —
every trailer that was seen in a theater was a first-generation
print.”

“There are so many advantages to this,” Knoll continues,
“that I think it's certainly going to catch on. One of the nice
things about conforming it and running the whole thing out ourselves
was that we got to control the color timings. I've worked on projects
where you make your best effort to get the shots looking the way the
director wants, but then later in timing it ends up a stop brighter or
redder than you intended. That's heartbreaking because then it's too
late. In this case, because the timing was happening here and it was
happening so much earlier in the process, if something fell apart there
was time to fix it. We ended up with much more consistent color, and
more quality control.”

The challenge of filming out entire 2,000-foot reels becomes evident
as Ellis explains. “The 30,000 frames we're filming out all have
to be perfect. We can't drop a frame because if we do that, the roll
isn't viable anymore. We're even filming out the leader so that the lab
won't be splicing anything on to these reels. They'll go through
printers smoother and faster — it's just a whole new
way.”

Filming out more than 1.5 million frames for Episode II puts an
exclamation point on what has been an unprecedented production —
even by ILM's standards. The impact this approach will have on future
projects is yet to be seen of course, but Michael Cooper has some
projections. “People may think that this pipeline is only good
for HD, but it's simply a digital pipeline. Even if we get a show that
requires only 400 visual effects shots, we can scan in the balance of
the picture and get it into our full-res conformer — as opposed
to filming out the finals and scanning those back in because you need
to cut negative. You think ‘Wait, we had a digital final. Why did
we go to film with it?’ We could cut it in digitally at full
resolution, do the color correction, and then film back out — and
give our clients a digital master as well. Even if DPs and directors
think digital capture is a bad idea, they like digital
intermediates.”

While it will undoubtedly take awhile for other filmmakers to follow
George Lucas' lead, Cooper predicts, “They'll quickly get used to
having these tools, especially when they realize they will have more
control.”