Broadcast Science

As its name implies, Fight Science, an upcoming National Geographic documentary special slated for national broadcast in the spring of 2006, will showcase the science of martial arts. The “science” aspect involves documenting a complicated analytical process that scientists formulate as they examine the force, accuracy, speed, and range of movement produced by many of the world's leading martial artists.

The way the Fight Science crew put the show together, however, has significance beyond its upcoming broadcast. Its production demonstrates innovative usage of existing tools and techniques, combined and applied in new ways that can inspire more efficient workflows — especially for broadcast projects on modest budgets. Those innovations evolved directly from overcoming challenges inherent in the project's creative needs.

Fight Science shows martial artists practicing and performing routines solo, in choreographed combat, and with weapons and props, and striking dummies and pads while scientists record, analyze, and discuss the force and impact of their blows. The question facing producers from BASE Productions, Washington, D.C., however, was how to produce the documentary in a way that allowed meaningful scientific analysis in a controlled environment on camera — while remaining entertaining to viewers. The solution involved an intimate marriage between production and post techniques (HD production, motion capture, and CG animation) and scientific processes and tools, including ultra-high-speed cameras. This marriage relied so heavily on motion capture that all aspects of the documentary — live-action, mo-cap data sessions, and background plate photography — were filmed on a specially designed and configured mo-cap stage in Playa del Rey, Calif.

“[BASE Productions] had done a show a couple of years ago in which we integrated CG and motion capture to study martial artists, but at the time, the technology was not far enough along to analyze things by applying motion-capture data on a live stage as part of the show,” says Mickey Stern, co-executive producer of the documentary. “This time, we wanted to take the highest level performers in various disciplines and record their live-action performances in one space, while at the same time collecting motion-capture data. We wanted the mo-cap data to drive the CG imagery we were creating to show how the bones and muscles and nerves move during these routines. But we also handed [the mo-cap data] directly to [the scientists featured in the program] to let them analyze it as part of the show, along with other data they were collecting, in order to reach conclusions on things that have never been studied this accurately before. So we built sort of a super mechanical testing ground where aspects of martial arts could be viewed and tested simultaneously by the scientists.”

A combo mo-cap/live-action set called the Dojo was used to tape all aspects of Fight Science, including live action using Panasonic''s AJ-HDC27F 720 Varicam.

To achieve this goal, BASE Productions hired mo-cap studio House of Moves (HOM), Venice, Calif., to spearhead construction of a combo live-action set/mo-cap stage in a nearby warehouse. With input from producers and DP Tim Gordon, production designer Tom Buderwitz designed the set. They came up with a main fighting stage, referred to as “the dojo,” and two annexes — called the “black limbo” stages — where demonstrations and scientific analysis were shot.

Built into the dojo set and prominently featured in the documentary's live-action scenes was HOM's mo-cap setup for the production — 32 Vicon MX-40 four-megapixel cameras attached directly to the top of the set's 15ft.-high walls at 4ft. intervals. An additional six MX-40 cameras were configured on ground level around the perimeter of the set. Following live-action sessions, the crew brought additional Vicon cameras onto the set on three rollaway towers that were used for the separate mo-cap sessions.

During production, live-action segments were filmed daily on the set. Then, for about two hours each evening for eight consecutive days, mo-cap sessions were conducted with the same martial artists/performers in mo-cap suits as they reprised their performances from earlier in the day.

“Originally, we discussed having live-action and mo-cap integrated into one shot, but the fact is, with the technology right now, we need the mo-cap suits to fit very tightly to make sure we record body motion and not clothing motion,” says Scott Gagain, executive producer on the project for HOM. “Therefore, we had to split it up between live-action and mo-cap sessions with the same actors on the same stage each day.”

Motion-capture data from martial artists and their weapons was used both for animation and to provide data for scientists during the show.

Steve Olson, HOM's studio technical director, set up the mo-cap cameras, and explains that the job was more complicated than usual, given that the mo-cap technology was meant to be part of the set itself.

“We would normally place cameras onto a set, obviously, not into the set,” Olson explains. “Our big concern was that the walls wouldn't be as stable as our normal setup, or that someone could knock a camera down. But, thankfully, that never happened. Each day, we also wheeled in the towers after they were done with the live-action shoot, and we placed them in about the same location they had put the live-action camera crane. They would take out the crane, and our towers became the fourth wall of [mo-cap] cameras. We also had the six ground-level cameras built into the set to capture some wrestling-type moves when fighters were locked arm-in-arm or entangled with their legs. It turned out to be a remarkably stable setup.”

During the course of production, HOM also rigged weapons, props, and crash-test dummies with mo-cap markers to help scientists measure the impact of blows, throws, and force impacting the person or thing receiving that motion.

“The scientists operated in the same station where we were working,” explains Gagain. “The way it's designed, our cameras are part of the live set, and then, during the scientific segments, a wall of the dojo opens and our technicians are part of the team in a little control room where the scientific study of the biomechanics of martial artists takes place. We were capturing skeletal data for our purposes, but in realtime during the shoot, the scientists were also interested in our marker data. They were mainly focused on the pressure of each punch, things like that. But we also did speed tests where they could see from our data how hard the performers were hitting pads and then compare that to high-speed camera data they captured themselves to further their understanding of the mechanics involved when the body moves in this way. A lot of this was more similar to the type of life-science things the other side of our company normally does, except that stuff is not normally filmed for broadcast.”

Each evening during the shoot, the live-action crew taped mo-cap sessions as part of the documentary. Sensors on 32 Vicon MX-40 cameras required DP Tim Gordon to film the sessions with very little light.

Although live-action and mo-cap sessions were split day/night during production, DP Tim Gordon had to perform double duty because, among other things, his job included filming behind-the-scenes footage of the mo-cap sessions for incorporation into of the documentary. Gordon shot the entire documentary using Panasonic's AJ-HDC27F 720p variable frame rate camcorder. He faced numerous production challenges due to the limited size of the dojo set, for example, but taping the mo-cap sessions for broadcast was among his biggest hurdles. During those sequences, Gordon was severely light-challenged.

“They can't have any direct light coming anywhere near the radius of the Vicon cameras, which was, in this case, nearly 360 degrees because of the design of the set,” Gordon says. “So all we could do was shoot with a little top light and a little background set light, and nothing more. The infrared camera sensors on the mo-cap cameras would cause occasional cycle problems here or there where, in a rare camera angle or frame rate, we would see random flickering and other anomalies in post, but that turned out to be pretty rare. One of the great things about shooting high-def was the fact that we could see and catch any such problems on the [14in. Sony BVM-D14 HSU] HD monitors on set and re-shoot, if necessary. By the time we got to post, we had dealt with most of that already.”

Overall, Gordon says the production benefited greatly from using the Varicam system. “The Varicam was essential for this type of production,” he says. “I have shot choreographed fights in the past with John Brenkus [creator/co-producer/director of the show for BASE Productions], and he really prefers stylized fighting. We shot some of that stuff 35mm in the past, but this show needed more flexibility than a 35mm shoot could provide since the schedule was very compact, and also because we were doing a variety of other things besides the fights — weapons demos, the documentary portions, the science stuff, and so forth — all on the same set and packed into the same schedule.

“The good thing about the Varicam was that the adjustable frame rate covered all our fighting needs and gave us all kinds of shuttered angles for the choreographed portions. Then, for the rest of the show, our operators could use the same system handheld to cover the other segments. Plus, we had to cover the mo-cap under extreme low-light conditions. In other words, we were serving a lot of masters and had limited ability to change our approach for each one. This format was the best one for that purpose.”

The crew taped sophisticated crash-test dummies receiving what producers call “perfect kicks and punches” in order to examine the force of killer blows from world-class martial artists.

Gordon explains that choreographed fights particularly benefited from his ability to alter frame rates, because it allowed him to capture “specific moments in time.”

“For each beat of a fight sequence, there were certain punches, kicks, etc., that were points of emphasis, either to relate to the mo-cap animation or to illustrate the science,” says Gordon. “In order to capture these specific moments in time between two or more martial artists, overcranking was a must to clearly see these moments. To see a spinning kick at 60fps goes a long way in illustrating the complexity and level of difficulty; whereas, at 24fps, the moment is so fast that you can't understand the athletic ability, timing, and speed these athletes need to perform at this level. Sometimes, we shot 6fps or 12fps just because we liked the motion, and it added a little flare to fight sequences.

“As for shutter angles, this system was valuable for understanding the weapons themselves, and how they are used either in attacking or defending. Using different shutter angles allows the eye to understand the path the weapon takes, and why that path makes it an effective weapon. It also gives the fight a hyper-real aspect, which, in the end, creates a fight with more impact than shooting a standard frame rate and shutter speed. If you watch how fast these guys use nunchakus, for instance, and you shoot different shutter angles, you can actually see the pattern left by the chuk, like a tracer in the air. That gave us an excellent way to illustrate the movement.”

Gordon shot the documentary using Fujinon 18×7.6 zoom lenses and one wide-angle Fujinon 13×4.5 lens. “We discussed using primes, but there was not enough time in the shooting day to keep switching, and we didn't have the luxury of three camera crews and two assistants to help get it all done,” he says. “From a production standpoint, we were tailored to the documentary approach, and then we adapted and went on dollies for the fights themselves. But John Brenkus emphasized to me that we had to be flexible and fast, so we stayed with the zoom lenses.”

Gordon also found himself aiding scientists featured in the documentary as they sought to capture ultra high-speed imagery using scientific cameras — footage that was later incorporated as insert shots. Primarily, Gordon collaborated with the scientists on lighting those sequences.

Mo-cap (top) and the Varicam (bottom) helped producers to study movement of martial artists as part of the documentary''s content and illustrate that movement efficiently.

“The scientists operated the high-speed cameras themselves, while we were normally shooting between 24fps and 60fps [on the Varicam],” says Gordon. “For particular insert shots, like shots of cinderblocks breaking in the course of a microsecond after being struck, they needed to shoot the blocks breaking at [1,000fps], and so, I had to light correctly to give them the stop they needed for that kind of work. Those were specialty shots, and they required such a difference in lighting that we had to light each of them individually.”

Primarily, for ultra-fast camera work, the scientists used the MotionXtra HG-100k camera from Redlake Imaging, San Diego. The camera is capable of shooting up to 1,000fps at high-quality resolution and 100,000fps at reduced resolutions. These cameras find routine use in ballistics and other scientific endeavors. For Fight Science, scientists used the MotionXtra camera extensively to illustrate blink-of-an-eye moments, such as the smashing of a cinderblock.

“In typical HD footage, you would see a stack of 10 cinderblocks all breaking in a single frame, and it happens so fast that not even the Varicam can capture it any better than that,” says Brenkus. “Using mo-cap, we are able to capture at 120fps exactly how the body moves when the fighter strikes the block, including movement of his joints and the origination of the impact and how that force ripples through his elbow and the rest of his arm. But our problem was how to accurately show the breaking of the blocks. The high-speed camera was the only way to do that, and what it taught us was that the blocks do not solely break down the center, one block to another. They break off-center through a process called progressive collapse. If an object breaks on top, and the force of the split moves into the next block and the one after that, scientists typically thought they broke down the center, in order. But here, we showed that the fighter snaps a whole column of cinderblocks using his fist and elbow in two places and then the force of his arm drives the center column through the entire set of blocks. It is far more than a simple domino effect. Things like that were proven in this documentary.”

Stern adds that the uniqueness — and importance — of the documentary is the fact that it gave scientists what he calls “perfect conditions” to examine theories about such things.

“We wanted to capture the perfect kick, the perfect punch, and so forth,” he says. “That is why we brought state-of-the-art [Hybrid III] crash-test dummies [provided by Denton of Detroit, which typically serves the automotive industry]. We wanted the martial artists to strike harder than they normally are able to do in combat, because to strike that hard would kill somebody. Here, the point was to measure and examine the full force of their blows — their full strength. Scientists have rarely been able to measure that sort of thing because fighters can't normally use full strength. Some things we filmed, like leveraged joint locks during grappling sequences, had never been filmed before, in fact.”

In addition to high-speed cameras and crash-test dummies, scientists in the show also used such devices as load cells and pressure sensors inside the dummies, designed to measure forces exerted on the dummies generally; rotary potentiometers, which are devices inserted into the dummies' chests to measure chest deflection when struck; accelerometers, which measure velocities and acceleration and are typically used in crash tests to measure air bag deployment velocity; and Tekscan's F-Scan bipedal in-shoe plantar pressure/force-measuring system, which maps and measures balance, weight distribution, and center of gravity on athletes.

Scientific tools used in production included Tekscan''s F-Scan bipedal in-shoe plantar pressure/force-measuring systems, which maps and measures balance, weight distribution, and center of gravity.

Even with all the live-action imagery and analysis captured during production, however, much of the documentary is devoted to what happens inside the body of martial artists when they strike or are struck. Therefore, combining the mo-cap data with sophisticated CG models was crucial. CG design took place at Sight Effects, Venice, Calif., primarily using Maya (v. 7), with additional models created at BrainZoo Studios and Entity FX, both based in Los Angeles. Brenkus charged CG artists with building what he calls “very complicated, three-layered” models of each performer's body, which were then animated using mo-cap data to illustrate the internal impact of various movements.

“We built scaled models in proportion to each individual fighter we were animating and moved the mo-cap data into those models,” Brenkus explains. “Usually, scientific computer models simply have veins and muscles and things painted on top of a core model. Here, we built models with three separate layers to them — skeletal layer, muscular layer, and the nervous system layer. They are nested, and with the mo-cap data, move biomechanically. In other words, our models move each visible portion of their bodies to help us better illustrate what scientists call ‘bio-fidelity,’ meaning ‘true to life’ movements, impacts, and effects on the human body of various forces.

“This approach let us, at any point, strip away nerves and muscles, or strip away bones and look only at muscles. Or we can have all three layers interacting at any given time. On a balance sequence, for instance, you have to look at a complex interaction of all three layers to see what is going on during an incredible feat of balance. The bones are girders; muscles are cables; and the nervous system provides feedback millions of times per second. With these complex models and motion-capture data, we were able to illustrate all of that.”

As production proceeded, while BASE Productions edited the documentary in Avid Xpress Pro 4.81 at 15:1 resolution, the company sent sequential TIF files of plates, background elements, transitions, and so forth to the three effects facilities. The facilities then added CG elements before returning the shots as QuickTime movies for BASE artists to down-rez and composite into the evolving cut.

CG work on Fight Science included human models with three separate layers—skeletal, muscular, and nervous system. These layers are nested and, combined with the mo-cap data, move biomechanically.

At press time, the offline process was just ending, and producers were getting ready to finish the documentary under the watchful eye of postproduction manager Richard McNealy. The finishing workflow, McNealy says, was greatly dependent on the Pro Import FCP plug-in from Automatic Duck, Seattle — a tool that permits elements of the project to travel seamlessly between the Avid, Final Cut Pro, and After Effects worlds at various stages.

After offlining in Avid Xpress Pro, McNealy says producers were planning to online the documentary in Final Cut Studio (v. 5) in the camera tape's native DVCPRO HD resolution. In addition, BASE was expecting to perform substantial 2D work in After Effects (6.5.1) throughout postproduction. Therefore, he says that Automatic Duck's workflow solution was central to this plan.

“Automatic Duck lets us deal with timing changes or speed effects, any nesting of layers, and those sorts of things, transferring the sequences and data seamlessly between the Avid and Final Cut Pro worlds,” says McNealy. “Basically, we are using Automatic Duck to go from Avid to Final Cut Pro and from Final Cut to After Effects far more efficiently than we would be able to do otherwise. We just go to Final Cut Pro and re-render, back into After Effects, and then send the final rendered QuickTimes back to Final Cut Pro and cut them into the final version of the show — mastering everything to D5 tape. Eventually, we'll take that tape and audio OMF files and combine them in a 5.1 mix and then take the whole thing to [Henninger Media Services, Arlington, Va.] for a final color correction session on their Da Vinci 2K system.”

McNealy emphasizes that while using Automatic Duck in this kind of workflow is hardly new, it is progressive in the sense that it illustrates how a job of this size can efficiently be put together on a modest budget without forcing producers to choose a single platform or format.

“We are able to synthesize a lot of different tools, which is always nice to do, but only if you can still have an efficient workflow,” he says. “Early on, for this project and others that we do, we decided that it made the most sense to use both Avid and Final Cut Pro, rather than committing to one or the other. The reason for Avid was that most of our editors were extremely Avid friendly, and we wanted them to work in the way that was most comfortable for them. The reason for Final Cut Pro was the fact that this was a Varicam project, and it is capable of working in the native camera format with no quality loss. So, rather than make a choice between those two advantages, we wanted to get both, and this workflow lets us do that.”