Making Mega Matrix

Sidebars
The Producer's Perspective
DP's Double Duty

Massive Effects Pipeline Evolves for Sequels

The so-called "burly brawl" (top) and the freeway sequence (middleand bottom) were two of the most complicated scenes produced forMatrix Reloaded..

Four years ago, a unique film reached out and visually electrocutedsci-fi fans with a darker, sleeker, sexier, more mature, and surrealapproach to an effects-laden movie — an approach debatably unseensince Blade Runner. That movie was 1999's The Matrix,direct out of the minds of co-directors/brothers Andy and LarryWachowski. What many people didn't realize at the time was that TheMatrix was but the first step in a planned trilogy, coming tofruition this year with the unprecedented, back-to-back releases ofMatrix Reloaded (May) and Matrix Revolutions (November)from Warner Bros.

Back in the late '90s, when they were trying to get TheMatrix made, the Wachowski brothers — visual bibles in hand,filled with creature concepts and storyboards by Geof Darrow and SteveSkroce — showed their proposal to veteran visual effectssupervisor John Gaeta. That meeting inadvertently set wheels in motionto develop not only a surprise box-office hit, but also an AcademyAward-winning visual effects team that has been very busy the last twoyears on two new Matrix movies.

The effects' work in Matrix Reloaded and MatrixRevolutions attempts to continue the legacy of the franchise'saward-winning original film. Ghostly twins materialize from flesh withdecreased opacity, emulating a cybernetic form. Characters fly throughcars to poised attack positions. Wide camera shots show the city ofZion with squid-like robotic machines digging deep into the Earth whileattacking the human fortress. And hundreds of replicated versions ofAgent Smith (Hugo Weaving) furiously battle Neo (Keanu Reeves) inReloaded's centerpiece — a CG/martial arts extravaganzadubbed “the burly brawl.”

To understand the massive digital effects effort (approximately2,500 shots between the two new movies on an effects budget surpassingsix figures, according to producer Joel Silver), Gaeta insists it isnecessary to understand the nature of the team behind the job. Althoughthe two films rely on work from several different facilities, Gaetasays that the core group responsible for effects on all threeMatrix films has been together since they met while he stillworked for industry legend Douglas Trumbull in the early '90s. Thatgroup collaborated at now-defunct Manex Visual Effects, which producedthe first movie's effects, and many of them are now continuing thecollaboration at ESC Visual Effects, Alameda, Calif., lead facility onthe new films.

During the early Trumbull phase of his career, Gaeta says he beganto connect with like-minded, talented people on similar tracks. Thesewere people who, like him, had become obsessed with the notion ofseamlessly integrating digital and real-world images.

“[At Trumbull's company, Mass.Illusions], I got connected to afellow named Joel Hynek [Predator, Oscar-winner for WhatDreams May Come],” Gaeta says. “I worked with him onthat project for a while, but left to work on Matrix. Before allthat, when I hooked up with Joel at Mass.Illusions, the pair of usdecided to steal the CAD department away from Doug's art department andrepurpose it to do 3D visualization for shot planning and for exportingNURBS that we made in the 3D animation environment into motion-controlsystems, so that we could get precisely what we were designing in 3D inadvance. That was our real beginning of being basically connectedbetween the three-dimensional computer graphics rules and the rules ofthe real world as it pertains to photography. We began speculatingabout how you define physical space so that you can integrate computergraphics.”

Around this time, Gaeta also encountered other key influencesresponsible for helping to develop the famous “bullet-time”effects that won the first Matrix movie an Oscar.

“During What Dreams and into the beginning ofMatrix, I became very interested in meeting people who weredoing similar types of experimentation of photogrammetry and shapeextraction,” he says. “[During this period], I eventuallyhooked up with Kim Libreri, George Borshukov, and DanPiponi.”

Gaeta credits that trio for its groundbreaking, develop-mental workon the principals that later allowed them and other colleagues toformulate the “bullet-time” effect. At ESC, they are tryingto surpass that accomplishment with work on the new films. Gaeta refersto Libreri, currently ESC's visual effects supervisor, as“probably the most knowledgeable person in the entire industrywhen it comes to image quality and understanding thresholds ofresolution enhancement, understanding the properties of film, and howto simulate them.”

Meanwhile, Gaeta calls Piponi, ESC's current R&D chief, “abrilliant R&D innovator,” and he credits Borshukov, ESC'seffects' technical supervisor, with pioneering some of the originalalgorithms, which have emerged as the foundation of contemporaryimage-space rendering methods.

“[This has led to the development of] the most realisticrenderer that has ever been completed for computer graphics,”Gaeta says. “It was basically the beginning of projectingtextures onto forms and extracting the forms by way of multiple cameraviews of those forms — suddenly getting this enhanced 3D thatpeople had not seen before, photographic-based texturing.”

Actors in this year's two Matrix releases appear in bothlive-action and animated form throughout the two films.

Gaeta says this team is now evolving into the world of true“virtual cinematography.”

“[That group] embodied what [has now become] this whole newrealm of what we call ‘reality capture,’” he says.“Other terms we've used are ‘virtual cinema’ and‘virtual cinematography.’ That was the beginning of thatphase of doing ‘bullet-time’ stuff — trying torepresent the concept of working with a virtual camera and a percentageof it with still cameras, all planned in 3D in advance. We are usinglocations, floating interests points, and things that real cameras do,as opposed to fixed interest points in frozen time. Instead, we aredoing floating interest points and dynamic time, because we couldfigure it out by way of 3D visualization, and then we were doing someof the very first virtual backgrounds ever for featurefilms.”

Gaeta hastens to add, however, that the new films should not becomeknown for any one visual effect approach. “They are two of themost diverse films that you will ever find in terms of effect styles,looks, and themes. We have everything from creatures to proceduralanimation to procedural destruction that looks really cool and stylizedto simulated phenomenon like plain water, weather — all thatstuff — done in really interesting ways. There are totallypsychedelic, full-on, head-banging, tripping, freaked-out visuals thatare there because the directors wanted to pursue a surreal flavor ofaction.”

As the sequels evolved, Gaeta had to form a much larger team tohandle the load. He spent months hand-picking the talent — amixture of longtime colleagues, well-established supervisors, emergingleaders in the effects world, and young, newly discoveredcode-crankers, all collaborating globally between a mix of well-knowneffects facilities and ESC.

To put all this in perspective, keep in mind that this team wasresponsible not only for the simultaneous effects work on the twofilms, but also the parallel video game release and material for theanime-styled, animated prequels that are now being released on DVD (see“Final Fantasy Live On,” Millimeter, March 2003).This volume of work on multiple releases from the same franchise isalmost unprecedented. Only the Lord of the Rings trilogy and,years ago, the second and third installments of the Back to theFuture franchise have ever attempted a production scheduleapproaching this scope. In the end, the team Gaeta formed consisted ofmore than 500 postproduction artists at eight facilities on threecontinents, not to mention on-set visual effects unit crews and othercrews from related departments, all working simultaneously onReloaded and Revolutions — creating, tracking, andwrangling thousands of digital assets.

“I am the effects designer/director and senior visual effectssupervisor, working for [production company EON Enter-tainment], thecreative interface between Larry and Andy and all the other vendors onthis job,” Gaeta says. “I have a staff and other small flexdepartments that go along with that. I have two associates, [visualeffects supervisors] Dan Glass and John Des Jardin. These guys arebasically my right and left arms for all manner of logisticsmanagement, image quality, and trying to basically create a continuityof communications once I have found a basic directive/path, helping mesort it all out with either Larry or Andy, or other designers, like[production designer] Owen Paterson or [DP] Bill Pope.”

Gaeta says Glass and Des Jardin, along with visual effects producerTerry Clotiaux, were key players in helping him keep track of more than2,500 shots — “all manner of scanning and layout andquality control, while I am working on the animation direction and lookand concept of the shots.”

Beyond his army of animators and compositors, Gaeta also had datawranglers, a LIDAR (Light Detection and Ranging) laser scanning team,an aerial unit, a live-action motion control unit, a pyro unit, and aminiatures unit collaborating with him. Gaeta also had a mo-cap unitcalled “universal capture,” which was a proprietaryperformance-capture system that Libreri developed at ESC.

“We also built a huge motion-control stage,” Gaeta adds,“to do a lot of the stuff that we had to do running in parallelwith first-unit photography on the same stages with all the primaryactors.”

Gaeta says the many effects vendors came next in that structure. Heemphasizes, in particular, ESC's work as a lead vendor, withapproximately 300 people working on the project. “That's afacility funded specifically for Matrix,” he says.“[I consider] ESC a nexus of talents from the bestcompanies.”

That group was responsible, among other things, for a variety ofvirtual humans, car-chase effects, and what Gaeta calls “stylizeddestruction and stylized phenomenon,” as well as extensivecreature-animation work under the supervision of former ILMer GeorgeMurphy, much of which won't be seen until Revolutionsdebuts.

Gaeta also brought Sony Pictures Imageworks into the mix, givingthat facility “a large package of shots that are quiteinvolved.” In particular: The City of Zion, a labyrinth oftunnels in both films, sewer crafts, and the squid-like creature sceneswere all handled at SPI.

Simultaneously, in Paris, BUF Compagnie toiled away on stylized,perceptual effects under the supervision of Stephane Ceretti and PierreBuffin, according to Gaeta.

Other key vendors on the project included Giant Killer Robots (SanFrancisco) which had supervisor Mike Schmidt oversee the creation ofwhat Gaeta calls “scenes beneath the Earth, new spaces wevisit.” Under Lynn Cartwright's supervision, Animal Logic(Sydney, Australia) created the twin dreadlocked ghosts effect seen intrailers for Reloaded.

The freeway scene was shot at the beginning of the mammoth project,on a small stretch of specially built freeway in Alameda, Calif.

Tippett Studio (Berkeley, Calif.) added important creatureenvironment shots on Revolutions, under Craig Hayes'supervision, but was not involved on Reloaded. (Other shotsbegan at now-defunct Centropolis Visual Effects, but were transferredto SPI when Centropolis closed during production.)

Gaeta is particularly interested in promoting both the quality andquantity of the work done by these companies, pointing out that despitethe huge combined budget for the two movies, the 2,500-plus shots mightbe considered a bargain, when one does the math.

“We're working multiple miracles in parallel here because ofthe amount of content and the quality that is insisted upon,” hesays. “People like to think, ‘oh yeah, they get to spend alot of money.’ But if you look at how much content the Wachowskibrothers are creating, and you actually run the numbers, an argumentcould be made that we have even less money per shot than the firstmovie, and what we have is a huge amount of content to accomplish, andvery complex at that. We had to be very smart in amortizing the R&Dand how we use it.”

Gaeta and his team of supervisors worked two years on evolving aprocess that began with primordial concepts during a February 2000meeting with the Wachowski brothers and other key participants, andgrew through a pseudo-Darwinian path to its full realization a week anda half before delivery in the first quarter of 2003. From those earlypreviz meetings, the team grew from a small, tight group of artists andthe two directors to meetings that eventually included close to 50people breaking down shots in detail.

But Gaeta has an almost religious devotion to that early CG previzstage, something he believes essential to the success of a project withsuch a large scope. All effects scenes were laid out both instoryboards and 3D form to lock down timing, angles, action, and soforth. Entire sequences were completed and cut together in previz form,allowing Gaeta's team to troubleshoot thoroughly before spendingmillions on any particular shot.

Some previz versions of the elements, created by Pixel LiberationFront (PLF), Venice, Calif., working as an EON subcontractor, were thenused as references on set during shooting to help the directors executelive-action scenes. Those digital pieces were also integrated into anon-set compositing system, allowing the Wachowski brothers to get goodcompositional previews of final effects shots long before they werefinal.

Plates were then shot, scanned, and supplied to whichever vendor wasin charge of that sequence. Vendors also received the 3D previzelements, allowing them to create early rough scene comps. Each vendorthen puzzled these various pieces together with their correspondinganimated elements, replacing over time the previz proxies withhigher-resolution versions. Each stage of development added more detailto the scenes, with Gaeta's team often changing parameters and addinglevels of complexity as they went along. “We were nevercompletely out of the previsualization stage until final post,”says Gaeta.

During production, Gaeta's team had a cornucopia of tools at itsdisposal to make sure the production could figure out real-worldparameters for the many 3D elements.

Most of the facilities involved used Maya-based pipelines, withrendering approaches varying between RenderMan, Mental Ray for itsglobal illumination rendering capabilities, and a handful of customsolutions. Shake was widely used for compositing, with both filmsfeaturing some Inferno work as well. ESC used its proprietary,photographically based pipeline for acquiring textures for synthetichumans and virtual backgrounds, while other facilities also used Maya3D StudioPaint for texture work. Filmbox processed motion-capture data,and LightWave performed some modeling work. ESC and most of the otherfacilities involved also relied heavily on a wide range of proprietaryplug-ins.

The toolbox also included laser scanning systems, photogrammetry,tracking markers, tracking cameras, observation cameras, opticaltrackers, and 3D lasers. All of these provided data to permit Gaeta andhis colleagues to create a viable relationship between the 3D world andthe physical world. Also, the production extensively used miniatures,optical tracking, a complete package of coders, and what Gaeta claimsare probably today's largest blue- and greenscreens.

Glass and Des Jardin were in charge of the complex bluescreen andgreenscreen setups. Glass supervised bluescreens for all the green-huedsubjects photographed inside the Matrix set and on the film'smotorcycle stunt course, while Des Jardin handled greenscreens for allthe blue-hued subjects filmed in real-world locations.

Glass's team created, among other setups, a 50'×80' bluescreenand another completely immersive bluescreen environment called the“blue egg,” essentially a hollowed-out, egg-shaped rigbuilt on Stage 1 at Sydney's Fox Studios, painted blue on the inside.The unit had arms on each side, to which actors could be attached androtated while being filmed for what visual effects coordinator DennisCooper calls “crazy camera moves.”

Des Jardin, meanwhile, supervised the largest greenscreen on theproduction, and maybe the planet — a 131'×279' soundstagecovered entirely in greenscreen.

“These were mostly custom screens imported from Thailand andcovering approximately the entire width and length of the stage,”says Des Jardin. “We also had perimeter screens that were [about]39ft. to 46ft. tall, depending on certain walls or ceilingobstructions, and you add to that the additional smaller greenscreens[20'×20'] scattered around to take care of custom edges on thesehuge sets, and we're talking about quite a lot of greenscreen. Most ofthe sets were pretty much surrounded by green to allow [the Wachowskibrothers] to move the camera all around their subjects, with plenty ofcoverage for future background replacements.”

The effects team also utilized a digital daily system at ESC, whichGaeta calls “one of the best digital daily screening systems everdevised.”

“We played the images off hard drives, using an IridasFrameCycler [a PC-based, uncompressed video playback system designedfor digital daily use by German manufacturer Iridas],” explainsCooper. “We used in-house software to create playlists on the flyto generate dailies, allowing us to put them into the editorial cuts wegot from production, trim the shots, and compare new versions of shotsto older ones by easily searching and playing off hard drives. Weviewed the clips through a JVC DLA-QX1G projector in our main theater,and we also watched dailies in breakout rooms using two JVC DLA-G150CLUprojectors. It was an impressive setup because we were able to seedifferent shots, comparing different versions, just sitting there inthe theater, without significant delay.”

It wasn't easy keeping tabs of 2,500-plus visual effects shots. EONengineers therefore devised a proprietary media management andrelational database to keep track of digital assets, data, records, andmundane details. That system, dubbed “the Zion Mainframe,”stores virtually all of the digital assets pertaining to theMatrix movies. Plates, 3D models, textures, previz animation,work in progress, and virtually every single asset that has been drawn,painted, modeled, and rendered dwells inside the Zion Mainframe.

The system's proprietary software features a node for eachdepartment under the artistic umbrella, and those departments lookingfor specific data can each query the system by department, artist, shotnumber, vendor, scene, and keywords. The project spanned threecontinents — North America, Australia, and Europe — and theZion Mainframe was designed to be accessible from EON offices inSydney, Los Angeles, and Alameda.

The movies, though, aren't all about CG creatures and environments.They also have a human side, with well-known characters returning,played by Keanu Reeves, Lawrence Fishburne, Carrie Ann Moss, HugoWeaving, and others. Producers made those actors available to theeffects team during preproduction for a series of facial-capturesessions, using the proprietary Universal Facial Capture System.Libreri, Borshukov, and Piponi wrote the system's optical flow code,and ESC CTO Paul Ryan wrote the actual capture software. That, in turn,permitted the effects unit to apply facial performances directly to 3Dversions of the characters.

The system used five Sony HDW-900 HD cameras, taping facialperformances from actors sitting in chairs. The cameras were modifiedto offload the images as uncompressed data directly to a disk array forlater use. That storage system consisted of a series of arrays fromHuge Systems, Agoura Hills, Calif. The mo-cap unit ran three capturesystems simultaneously in Australia, with each relying on two 1TB disksfor storage, giving a total of 6TB of storage at any given moment.

“The way it was set up, we could take those six disks offlineto a [Sony] Tape Robot system, which pulled data off the disks ontotape, and replace them with six other disks,” says Cooper.“We also had a backup capture station always running, and acouple of extra disks, so we had about 20TB of storage on stage duringthe capture shoot.”

That data was later fed into computer systems running proprietaryalgorithms designed to analyze facial movement and calculate 3Dinformation, triangulating points on the face from the positions of thefive cameras. This allowed Gaeta's artists to view the animated facefrom nearly any angle.

Reloaded features an out-of-control, over-the-top chase downa busy freeway, shot at an abandoned U.S. Naval base in Alameda. Torealize what had been previsualized for the complicated sequence,producers gave the production permission to build a quarter-mile pieceof real highway in Alameda so that full control could be maintained forindefinite periods of time. Gaeta says that investment paid off becauseit would have been even more costly to build the full highway in CG.There are, according to Gaeta, “hundreds and hundreds ofshots” in the sequence.

The sequence itself features a potpourri of CG cars, live-actionstunts, real backgrounds, and CG backgrounds, typifying the project'ssophisticated effects approach. In particular, the freeway sequencefeatures a CG version of Agent Smith leaping from car to car. Thesequence mixes-and-matches two complete CG shots, extensive live-actionplates, CG cars, CG overpasses, and much more.

Likewise, the “burly brawl” mixes-and-matches freelybefore climaxing with an all-CG shot that epitomizes Gaeta's beloved“virtual cinematography” by replicating the real actors andlocation doing seemingly impossible things. The sequence includeshundreds of CG replicants of Weaving, his face placed onto the bodiesof martial artists (all choreographed by Hong Kong legend YuenWoo-Ping). As they attack Neo, a virtual camera flies in, around,above, and through the scene.

“The ‘burly brawl’ is three different things: somelive action, some head and face replacements, and some pieces that arecompletely virtual,” says Cooper. “The martial arts movesin the CG shots, though, are still Woo-Ping's choreography. Theymo-capped, at the most, nine guys at any one time, doing martial artsmoves, and then built all that up. The cool thing is, you end up withsomething you can't possibly film in real life, yet it is still avision of what the directors wanted in their photography — anextension of their photography. So while it is synthetic at somepoints, the idea is to achieve the vision of what they were trying toaccomplish on set.”

Jennifer Champagne is a producer and founding partner of specialeffects company Max Ink Café and production company Max InkProductions. She can be reached via email at champagne@maxinkcafe.com. Michael Goldman alsocontributed to this story.

The Producer's Perspective

By Michael Goldman

Because the directors of the Matrix franchise — thebrothers Wachowski — have elected not to discuss their work withthe media, it's left to producer Joel Silver to explain this year's twosequels. The first thing Silver wants to make clear about MatrixReloaded and Matrix Revolutions: the two films are notback-to-back sequels at all.

“They're one giant movie that we are cutting in half to showeach half separately,” Silver explains. “We shot for 270days in Australia and California, making not only this giant movie, butalso shooting footage and creating effects for the videogame and theMatrix prequel animation [coming soon on DVD], as well as thewebsite, the sound track, all that stuff. We spent well over $100million, much of it on visual effects. The difference between thesesequels and, say, when I produced the Die Hard sequels, is thatmost sequels, including Die Hard, get made because the first oneis a box-office hit. In this case, this was a complete, huge story inthe heads of the [Wachowski] brothers, and they agreed to do all thiswork, writing and directing the three films, if I would work to getWarner Brothers to commit to all three. These two films were alwaysnecessary to complete the story.”

From a producer's point of view, Silver makes no bones about whymost of that story was produced in Australia: Runaway production is anecessary evil these days, in his view.

“It's just not financially feasible to shoot something of thisscope in the U.S. right now,” he says. “It's not acoincidence we went to Australia. The fact is down there, the cost isless, and the government helps us with subsidies. It's that simple. Wedidn't want to go there, but the truth is people here in Californiadon't seem to care. The aircraft industry is gone, and I guess theywant us gone, too. Our state and national governments don't seem tocare about keeping productions in the U.S., even though entertainmentis the single biggest export this country has.”

In any case, the entire project has been a “freight train, andwe've just managed to stay in front of it,” says Silver.“The key was a cohesive team — one editor, one visualeffects supervisor, one costume designer, great crews in Australia, andall that. But it has definitely been the most complicated project I'veever worked on.”

And finally, why won't the Wachowski brothers talk publicly aboutwhat has, in essence, become their life's work?

“They don't want to explain it,” says Silver.“They want people to talk about it and figure it out forthemselves. They figure that if they comment, that gives finalresolution to many questions, and they don't want those questionsresolved. I tried to get them to talk, but they wouldn't do it. That'sthe deal they made with me, so I said OK.”

DP's Double Duty

By Michael Goldman

DP Bill Pope is evasive when asked if he would ever again agree toshoot two epic films simultaneously as he did for the two Matrixsequels.

“I'd rather not answer that,” he says. “But I willsay this is easily the most grueling project I've ever done. It wasspiritually and physically difficult.”

From a DP's point of view, the shoot Pope conducted in Australia andAlameda, Calif., over the course of 18 months (with a break in betweenlocations) is a testament to the art of balancing preparation withflexibility. The frantic pace, however, was necessary for MatrixReloaded and Matrix Revolutions to premiere this year.

“In a sense, we had plenty of prep, and in a sense, we had noprep at all,” he says. “The preproduction time was largelyspent preparing to film two sequences in Alameda — the freewaysequence and the [‘burly brawl’] fight between Keanu Reevesand Hugo Weaving. Those two sequences alone took 65 days of first-unitwork and almost the same number of days by the second unit. That camefirst, so our prep focused on that. Then, we had a month break totravel to Australia and set up. By the time we got there, I thought,‘We are under-prepared for the rest of the shoot.’ In thatsense, we needed a lot of day-to-day planning, so we set up a processto shoot and plan at the same time.”

That process came in the form of what Pope calls “a shadowgovernment” that set things up for the first unit as productionprogressed.

“The shadow government had a grip, a gaffer, and a fullproduction team from every relevant department that shadowed us,”he explains. “The first unit would talk with them at some pointduring the day, and then they would go off and prep the next thing we'dhave to shoot. It was exhausting because I had to leave the set, watchdailies, and then go talk to them about lighting and things, but it wasthe only way to get it all done.”

Also important was finding the time to attend daily meetings to planfuture shots, which, as Pope points out, might have been just a day ortwo away. “In my particular case,” he says, “my keygrip, Ray Brown, and my gaffer, Reg Garside, got me through it. Theywould often prep for me while I was dealing with otherthings.”

Pope emphasizes that the job shooting effects plates was a big partof the load because there were more than 2,500 digital effects shotscreated for the two films. “Most of those shots had dozens oflayers, so you might have 2,500 shots, but it probably took around7,500 plates to create those shots,” he says. “It waspretty intense.”

The cinematography team also had to contend with maintaining theMatrix look pioneered in the first movie while at the same timemoving beyond it.

“The first movie dictated the notion that there is the‘real world’ and there is the ‘Matrix,’”he says. “That automatically divided the films into differentcolors, with the real world being cold and blue because it is a worldwith no sun, and the Matrix being green and sunless, because it ismanufactured by a machine. That held true for these two movies —you use wider lenses for the Matrix scenes because you want to befurther away from the actors. In the real world, you want to be closerbecause, there, they are real people. At the same time, we have newcharacters and locations and different themes in these two films. Thatmeans we present some new colors, and we light some scenes differently.There is also a difference between the two sequels. The second moviegets darker, and we move outdoors more. I can't give away the story,but things get ‘worse’ in the second movie, before theyever might get better later on, so the lighting and color schemesreflect that.

“Most of the two movies was shot on Kodak [Vision] 5279,except the effects plates, for which I mainly used Vistavision. Thefreeway scene, though, I used 5274. The reason I used a different stockthere was for continuity — that sequence was shot over 40-somedays, and I knew we'd have to scan the whole thing to digitally treatthe sky and sunlight to match it all up since the weather changedduring that period. I figured 5279 would give me a finer grainstructure since we had to scan it and treat it anyway.”