Blue Sky Bluescreen

Sky Captain was originally planned as a black-and-white movie. As that plan changed, shots were drained of all color from the bluescreen shoot, composited in black and white with CG elements, and stylized bits of color and sepia tones were reintroduced. Movie photos courtesy of Paramount Pictures. Behind-the-scenes photos courtesy of Darin Hollings.

For digital filmmaking aficionados, Sky Captain and the World of Tomorrow's path to the big screen has to be considered the ultimate feel-good story. It goes something like this: In the early '90s, a film-school graduate dreams of making a stylized, independent film combining live actors with digital environments entirely in a computer. He labors for years to make a six-minute rough version of his idea on an early Macintosh laptop in his apartment, using actors from a one-day shoot, still photos for backgrounds, and early versions of what later became After Effects. He lucks out one day and gets to show that six-minute short to a major movie producer (Jon Avnet). The producer falls in love with the stylized look of the short, options it, hires major movie stars to appear in it, and launches the movie into production. Later, the producer gets a major studio (Paramount) to add financing and distribute the movie.

And best of all, that young filmmaker — Kerry Conran — is hired to write and direct the movie, mainly because of his vision about how to make it on a modest budget.

“Going back 10 years, I had just gotten out of film school at Cal Arts,” he recalls. “It occurred to me that if you took the premise of an animation stand and shot an actor against a bluescreen, you create the equivalent of the cell animation portion of traditional animation. Then you could paint in the background any way you wanted: traditional animation, painting, live action, or whatever.

“Soon, the first version of After Effects [then called Egg, before Adobe purchased it] began appearing. It clicked with me that this was the computer equivalent of an optical printer and an animation stand — a way to treat animation as if it were live-action, or live-action as animation. That led me to create the short film, which took about four years. That project seemed to have promise and looked, potentially, like a good production model for doing this on a larger scale, and it ultimately served as the template for this movie. But never in my wildest dreams did I conceive of it growing to the scale [almost a $60 million budget] that it did.”

Thus, Sky Captain, regardless of critical or audience reception, is an historically important movie: the first studio-backed, wide-released feature film with big movie stars (Jude Law, Gwyneth Paltrow, Angelina Jolie) shot entirely in high-definition in front of a bluescreen, with all backgrounds, environments, creatures, and machines created digitally using off-the-shelf tools.

Sky Captain, however, isn't simply a case of matching live actors to digital backgrounds. What helped sell the movie to Avnet and Paramount was the unique look of the piece. The goal was to somehow make everything fit into Conran's original vision of a stylized, film noir-ish, Flash Gordon-type serial look, originally intended to be black and white. That vision evolved due to a variety of factors — it's no longer a black-and-white film, for instance. But it isn't exactly color either. Most of the movie features dark, vintage-looking, sepia and gray tones, with rich colors appearing briefly.

A typical scene being shot bluescreen using Sony HDW-F900 cameras. As the photo shows, only props that came in direct contact with actors were real--everything else was created later in computers.

How to create this look on what was then an extremely low budget, while still attending to the myriad details involved in making a digital movie that ended up featuring more than 2,000 visual effects shots, became the central challenge. The answer required extensive planning to formulate a template and assembly line production approach.

That basic template began with artists storyboarding the entire movie based on designs from Conran's brother Kevin, the film's production designer. It then included the construction of sophisticated 3D animatics created in Maya; a practice run-through of the looming bluescreen shoot to manufacture data used to create a detailed grid map showing positioning of every element on the bluescreen set for each shot in the movie; the actual bluescreen shoot over the course of 26 days, relying on the animatics and grid map to keep to a strict schedule; and the creation of an inhouse effects facility to build 3D models, composite live actors with those models, color correct the material, and edit and conform the movie, all under one roof.

This basic approach was expected to continue at that inhouse facility — housed in a Van Nuys, Calif., industrial park and dubbed World of Tomorrow (WOT) — until the then-independent movie was completed. Producers hoped to find enough financing to then finish it with a digital intermediate and sell the picture for distribution.

But, of course, things turned out to be far more complicated than that.

According to visual effects supervisor Darin Hollings, the movie's starting point was Conran's original six-minute short. “I call that a trailer for a movie that didn't exist yet, and it helped us develop our method all the way through,” says Hollings.

He adds that, prior to the movie being sold and earning distribution, filmmakers were assuming they would have to do this for essentially nothing and shoot it in about four weeks. “Once we knew there would eventually be a [blue-screen] shoot, the first thing we had to do was put together a facility where the entire movie would be made, and that became World of Tomorrow,” Hollings says. “We began having artists storyboard the movie. We cut those storyboards together in Final Cut Pro, and then Kerry went to London and conducted a table read-through of the script with the actors. That gave us an audio track to cut together with the storyboards. Eventually, we replaced those storyboards with 3D animatics built in Maya.”

The next step, according to Hollings, was to figure out a system to conduct the bluescreen shoot efficiently because actor availability was limited and principal photography would have to take place on a compressed schedule.

“Once we had the animatic and staged the run-through [using stand-in actors] on our Van Nuys stage, we took each shot in the movie and created a top-down version of it on the grid map by aligning our Maya virtual camera to exactly mimic our Sony [HDW-F900] camera and lens settings [DP Eric Adkins used Fujinon HD Cine Style HA10×5B-10 zoom lenses],” he explains. “The map had letters and numbers indicating what items and people would be positioned where. We knew we were shooting HD, that Kerry had never directed a feature film before, and that we had little money, so we needed a process to help us stay on schedule. For a couple of months in Van Nuys, we shot the run-through and refined the process. With the animatic and our grid map for every shot in hand, we eventually went off to [London's Elstree Studios] to shoot on three large bluescreen stages. The next challenge became scheduling precisely enough to stay on track.”

Writer/director Kerry Conran (wearing cap) supervises preparations for capturing another sequence on the bluescreen stage in London.

According to Hollings, staying on track meant filmmakers had to capture a shot every 12 1/2 minutes — an average of 37 shots a day for 26 days — on three bluescreen stages at Elstree. The burden to keep each day's shoot to this schedule fell to Hollings, DP Adkins, and production supervisor Matthew Feitshans. That trio held scheduling meetings every weekend to plan the coming week's schedule, and every production evening they would plan the entire next day in detail. Hollings says the trio would often examine the week's upcoming shots virtually through the Maya animatic.

“Eric looked at it from the DP's perspective and needs, I looked at it from the effects POV, and Matthew handled it from the POV of the actors' needs,” Hollings explains. “We decided then and there if we needed a dolly track or anything else. I would bring up the first shot in Maya, and we would all watch the camera run through the action. Then, I'd bring up the second shot, and see how it related to the first. Is the dolly track in the shot? Is there some piece of equipment or prop that might cause a delay or problem? Can we leave the camera in the same spot? Doing this helped us catch all sorts of problems before they could happen. I don't think this has ever been done to this extent before.”

Adkins says it was obvious in late 2002 when preparations were underway for the London shoot that he would have to use the Sony HDW-F900/3 camera. (Three HDW-F900s were used during the bluescreen shoot, recording to three Sony HDW-F500 HDCAM recorders.)

“Sony's HDW-F950 wasn't available yet at that point,” he says. “The [Thomson Grass Valley Viper FilmStream camera] was still in its experimental stage back then, so the old reliable 900 was the camera we used. The HDW-F900/3 camera does not use the same RGB 4:4:4 color space as those cameras do, but this was less important to us at the time than the practicality of a reliable data capturing system. We decided long before that we would throw out all the down-sampled color in compositing anyway, and we were expecting to do a black-and-white film or to add a subdued color element back in later if required, as it eventually was.

“I sat down with Stephen Lawes, our compositing supervisor, during preproduction, and we examined test footage and decided how far to take highlights, how far into the blacks we could go without things being crushed and irretrievable,” continues Adkins. “We wanted to keep clean edges, since the whole thing was shot bluescreen, and so we knew it was possible to add diffusion to bloom the highlights as required later.”

Because of that fact, Adkins was relieved of the usual concern over in-camera diffusion. He adds that it was determined early that the production would use bluescreen, rather than greenscreen, in order to produce the cleanest possible mattes.

“With greenscreen, if we accidentally overexposed and blew out the actors' hair, we would sometimes get bright, red edges around things,” he says. “That would make for difficult key pulling. With bluescreen, the processing software we were using [After Effects keyer plug-in Primatte] was kinder pulling keys on the hair of our blonde actors [Paltrow and Law].”

Adkins adds that the previsualization process, particularly the grid map, was central to making his job manageable. Adkins compares the grid approach to playing Battleship: “We analyzed every single shot. We had reference for every shot — information on tilt, angle, camera height, lens information, angles, lots of things. Each day, we printed out a fact sheet for every shot, and handed that paper to the operators, assistants, dolly grips, everybody who needed it. So it was a well-choreographed endeavor. On one hand, I was doing a traditional DP job, but at the same time, I was far more involved in scheduling and logistics than normal.”

The London bluescreen shoot, however, was not the only shoot conducted for Sky Captain. Adkins and Hollings conducted an even more unorthodox still-photo shoot at New York City's Radio City Music Hall in order to capture background imagery and textures for a crucial early scene. Those stills were used extensively as nodal tiles, stitched together to represent the Radio City interior. Adkins and Hollings took two, one-week trips to NYC, shooting stills with a couple of Nikon D1x digital cameras.

“We brought storyboards and used them to give us reference of the types of background plates we needed,” says Adkins. “We realized, with some manipulation, we could do panning shots and zooming shots and tilting shots when we stitched them all together in post, and that's what we did.”

Visual effects supervisor Darin Hollings (center) consults with Conran about capturing clean mattes during production.

Processing all this material, producing detailed digital environments, characters, and props, and compositing the elements together — building the movie in computers, in other words — remained the primary challenge. Construction of CG models in Maya (version 4.5) began in earnest at WOT, while the testing and run-through phases of the live-action shoot continued simultaneously. At that point in time, the film had not yet been picked up, so the plan was to construct the entire movie at WOT, with teams set up by Hollings to operate under supervision from modeling supervisor Zack Petroc, animation supervisor Steve Yamamoto, CG supervisor David Santiago, CG lighting director Michael Sean Foley, compositing supervisor Stephen Lawes, tracking supervisor Candida Nunez, and lead texture artist Larry Wu.

Conran says WOT adopted a hybrid approach between traditional live-action and animation production, using a computer to marry them together. “For instance, we would approach the creation of a virtual set like a film shoot — we didn't look at it like we were dealing with a single shot in the computer. So we figured out how to light the scene with a virtual camera and lighting rigs within the Maya software set up to permit us to trigger 10 or more shots in a single session. We pre-rigged and pre-designated rigs to capture particular groups of shots,” he says.

“David Santiago created MEL scripts [directed by Yamamoto and Foley] that we used to animate low-res versions of the models, and then more MEL scripts to send those images to our lighting department, where they could easily cut and paste it into Maya, light it, hit another button, render it, and so forth,” Hollings adds. “This gave us an easy RenderMan pipeline.”

On the compositing side, the plan changed somewhat as more producers joined the project and decided that Sky Captain would not be a black-and-white picture after all. Because of this, Hollings and Lawes split the compositing department into two separate units — black and white and color.

“Once we started to nail down the approach to lighting bluescreens, to coordinate how the CG should look, we suddenly had to figure out an approach to color,” says Lawes. “No one had viewed this project from a color standpoint since the plan was to make it a black-and-white movie, so we had to come up with something completely new to us. Our original thought was to outsource all the color work to a company that does traditional frame-by-frame hand painting. But we had such a subjective palette on this project that we realized it would require a whole staff just to oversee the outside company. That was going to create more work on our end anyway, so we decided it made more sense to do it as part of the compositing process. Our first attempt involved re-introducing the original color of the footage as a whitewash overlay on top of the finished black-and-white shots, and instead of using the original colors of the backgrounds and CG elements, we used broader washes of color, which was more of a comic book approach.

“But it didn't look quite right — it was kind of a weird concoction of semi-photographic, semi-CG, semi-comic book type looks, without committing to any one of those. It was a particularly useful failure, however, because it got us going in the right direction. We next decided to commit to a more photographic POV, and wash those colors on top of our black-and-white shots. That started to look better, but it still looked like the color was swimming on top of the black and white, rather than being baked into the shots. So we backed off a bit and decided to reduce the amount of contrast and diffusion we were applying in the black-and-white phase, and then apply some of that back in during color compositing, baking colors into it and sandwiching the layers together. This looked better, but it became obvious we would need a whole team for this, so we came up with the two-team compositing approach.”

Still, it quickly became apparent that as well as WOT was performing, the facility would not have time to finish the movie by Paramount's looming deadline without outside help. This caused the production to bring in senior visual effects producer Scott Anderson to oversee the hiring of 13 facilities around the world to finish shots for the second half of the movie. According to Conran, the first half of the movie — about 1,100 shots — was completed entirely at WOT, while another 1,000 shots were split up among the 13 facilities. WOT did all the CG modeling and much of the initial animation for those shots, while the outside facilities then tweaked, colored, and composited those sequences.

Visual Effects supervisor Darin Hollings studies the film''s animatic during production.

“I had a lot of concern [about farming out shots], but it was inevitable to get this movie made on time,” says Conran. “Still, this was a place where, once again, our pre-planning really helped. These companies mainly were used to render, texture, composite, and perform minor animation and modeling chores. We took every precaution to make sure everything would be seamless.”

A key precaution involved Lawes' efforts to create guide frames of composites for each facility to follow, to keep shots as close to the WOT approach as possible.

“This was a tricky issue because these are all very good facilities, but they all use different tools and different approaches,” says Lawes. “For instance, we did most of our compositing work in After Effects with a bit of Shake. Other companies used different tools, yet I had a set way of doing the color compositing as each shot was built up layer-by-layer.

“As each vendor came in, I sat down with them and went over our overall approach. Then, as we got into specific sequences, I would take particular shots and create look frames for them. That gave them a great idea of the direction we wanted to go. Then, when the shot came back to me, I would give more detailed direction, and we'd talk about whether they had enough diffusion, or not enough, and if there were too many shadows, heightening the contrast, and so forth.”

This approach maintained the overall, stylized color plan for the movie, but it was impossible for every shot from every facility to look identical to imagery from the WOT facility. This is why the conform and DI stages loomed even more important than usual.

The first change necessitated by Paramount's deadline was a major upgrade of WOT's SAN because not only the plethora of visual effects and compositing, but also the film's conform, would be running through the facility's servers.

Hollings recruited director of technology Brian Chacon to upgrade the WOT machine room with racks of Linux PCs for 3D rendering and a host of professional-grade storage devices. “Remember, we started as an independent film with almost no money, so we first tried to be a sleepy facility using off-the-shelf consumer boxes, but we ended up needing powerhouse storage, and Brian was central in helping us get our SAN up to speed,” Hollings says. “We added about 30 TB of .NET App storage, while using our original AttaBoy disk space for near-line storage, as well as completely overhauling our network and adding uninterruptable power supplies and other things.”

The movie, on its way to EFilm, Hollywood, for the color correction portion of the DI process, first had to be assembled into its final form by freelance digital conform artist Raymond Yeung at the WOT facility. Yeung relied on the EDL created by the film's editor, Sabrina Plisco, using Final Cut Pro 4. He didn't have to worry about color control issues because that chore would be dealt with at EFilm, but Yeung did have a massive data management challenge on his hands.

A stylized and color-manipulated version of New York was created using CG and still-photo textural elements to evoke influences from the classic film serials of the 1930s.

Working with data in the form of HD resolution, 16-bit SGI files from the WOT shared server and the offline EDL, Yeung used a Nucoda Data Conform system to replicate the original edit piece-by-piece, constantly patching in missing shots in place of chunks of the animatic.

“The Nucoda software is pretty advanced — it knows where the data lives on the network,” Yeung explains. “It will map the data folder on our local storage to particular clips based on shot names listed in the EDL. But it also has a very strong viewing capability. So basically, I can ingest the data, shot-by-shot, and quickly turn it into a viewable, continuous stream, building continuous media for the conformed reels. Once we have that media available, we can go in, and based on the reference guides, I then go, cut by cut, and match it all up until I have built an entire reel. I then sent the reels over to EFilm on LaCie Big Disc 467GB-capacity FireWire drives in the same HD (16-bit) format. The entire movie, after assembly, took up about 1.5TB of storage, so it fit on about three drives. A custom format cut list, generated automatically from the original EDL, was also supplied to EFilm for their editorial line-up and proper handling of individual cuts.”

Yeung adds that the most challenging aspect of the conform was keeping track of the shots as they arrived daily, including the different versions. “Versioning is a big issue with this kind of production,” he says.

The other challenge was the movie's looming deadline. Yeung therefore had to send some reels to EFilm without final versions of certain shots, and the final conform on those last chunks was handled at EFilm.

“It really went down to the wire,” adds Yeung. “We were able to patch in on the final weekend of the conform over 200 shots. Basically, we had a precise numbering scheme. We knew where the shots would fit into the reel, and we just patched them in as little data packages with the file's proper name and precise frame range.”

Michael Kanfer, visual effects imaging supervisor, was responsible for maintaining calibration and quality control between the production and all of the outsource vendors. He also supervised the DI at EFilm for WOT.

“We did months of testing with EFilm,” says Kanfer. “We gave them 16-bit [1920×1080] SGI files and had them converted to 10-bit, Cineon log files. [Colorist] Steve Bowen then color-timed the movie using the proprietary EFilm DI system. EFilm recorded the color-timed reels out to film on Arri Laser recorders on 5242 stock. We did a variety of different release print tests at Deluxe Labs to make sure that our footage would look great when it finally made it to the theaters. Our material had an extremely desaturated look to it, which made it very sensitive to even minor color corrections and lab variations. Steve Bowen was able to cleverly manipulate the data and keep the integrity of our images.

“We worked with EFilm and Deluxe to keep everything within plus or minus one-half of a printer point, which is generally unheard of on a theatrical release, but it was absolutely necessary on our type of content. [The usual tolerance is ± two or three printer points.] In addition to selective sharpening and diffusion touch-ups, EFilm was also able to fix camera noise and other imaging artifacts that popped up now and then.”

Kanfer adds that the viewing environment at EFilm was an issue to be dealt with because filmmakers were used to viewing Sky Captain clips on HD monitors, while EFilm was used to projecting film gamut imagery for most projects at 2K on a 20ft. screen.

“Originally, Kerry Conran wanted the reproduction of the show on film to match the look of his HD monitor,” adds Kanfer. “But there were issues with that since the color temperature, color gamut, and dynamic range of HD is different than that of film. By using the log-converted, 16-bit, SGI files at EFilm for the DI, we had the ability to get even more information onto film than was originally available on the 8-bit HD QuickTime version. This gave Kerry quite a bit more latitude with the dynamic range, and the softer roll-off in highlights resembled more of a film look.”

Elizabeth Cotter, color-imaging director at EFilm, says the company projected the movie in the DI suite using the Barco 2K projectors it uses for all projects. “We utilized look-up tables that were set up for what [the filmmakers] were used to seeing. The equipment was the same, in other words, but the handshake was a little different. We came up with a good mapping method to better emulate the HD environment.”

Despite the film's unique color approach, Bowen says his job did not require intense color decisions during the DI because those were handled during compositing. Instead, his job was more about “interpreting the filmmakers' formula to make it all flow from one area to the next” since material came from different vendors.

“In that sense, this DI was more difficult than many because we had to adhere completely to a stylized, pre-established look,” he adds. “This limited some options in terms of range for diffusion, highlights, etc. I had to sharpen some shots, diffuse others, and that was pretty challenging. In terms of color, what I had to do was very limited. I'll take credit for fixing some things, but not for creating this look. Since they took all color out of the images and then reintroduced it, color went mainly back into the actors since they were the only live elements shot on-stage.”

At the end of the process, Sky Captain made it onto celluloid about a decade after Conran first had the idea. For him, the movie's arrival is a personal triumph, but more importantly, he appreciates the project's historical niche.

“I know there are critics of this idea of doing without locations and sets — doing it all digitally,” he says. “But for this film, I never would have gotten the opportunities I eventually got any other way. I had to take advantage of what these tools offered and make the best of them. Obviously, every film should not be made this way. But for this particular project, this approach allowed me to shoot the movie with principal cast, including major actors, in just 26 days.

“The method of making this movie in front of a bluescreen was, therefore, one of the most significant reasons the actors would agree to take a chance on this project. Maybe, soon, you might see more films that have the scope, scale, and ambition of a studio-financed film on a lower budget. If nothing else, this project demonstrates such things are possible.”