Realtime Redux

Realtime capabilities continue to be a hot issue at both the highand low ends of the NLE market.

Apple has taken its own approach to realtime with Final Cut Pro 3.Realtime performance will vary, and portions of the timeline willautomatically render, depending upon how FCP evaluates the CPU/OSperformance limits of the system and the “performance cost”of each effect on the timeline.

In 2001, when I wrote “Real Needs, Real Questions,Realtime” (see the August 2001 issue of Video Systems), Idid not expect to write a second time about the topic. However, at boththe high and low ends of the NLE market, realtime continues to be a hottopic. So hot that I have to wonder why.

Is it because no new topic has arisen for the marketing and presssymbiosis to talk about? If so, perhaps 24p will take over for realtimein 2003. Or is there a deeper reason for the focus on production speed?When Alan Greenspan talks about increased productivity, he's oftenreferring to advances in technology that increase productivity.Realtime, especially when it costs almost nothing to obtain, is anideal productivity investment. It's an investment that pays off bothduring a recession and in the boom times that typically follow.

The question facing those making this investment is: What type ofrealtime do you want?

I used to think of realtime as simply a positive adjective —like “digital” — that was applied by zealousmarketers. But at least digital has a fairly clear meaning. That's notthe case with realtime. Both the $60,000 Media 100 844/Xe and Adobe's$600 Premiere 6.5 are touted as realtime, and both are, but in verydifferent ways and using very different technologies. So in whatway are products realtime? And how do you match realtimetechnology with your needs?

One key differentiator is whether the product can output acompressed (DV, DVCPRO25, and soon DVCPRO50) copy of the timelinewithout delay. Adobe, Apple, and Avid (XDV) offer software-onlyrealtime products that require the timeline to be processed before itcan be output via FireWire. For some editors, these products don'tqualify as realtime because of this limitation.

Personally, I remain unconvinced that realtime compressed output isa critical requirement for most production. When I hear folks blast a$1,000 product for not delivering instant results, I have to wonder ifthey have ever worked with film. When a film editor decides that atwo-second dissolve should occur between two shots, a laborious processbegins. First, 48 additional frames (after the effect starts) are addedto the A-roll before a splice is made to black leader. Second,preceding the point where the effect ends, 48 frames of black leader inthe B-roll must be cut back to allow 48 frames of the incoming shot tobe spliced into it.

The A- and B-rolls are then sent to a lab where they are copied to aprint. (This process traditionally required that first the A-roll beprinted, then the print-roll be rewound, and the B-roll printed.) Ifany changes need to be made, the entire physical process must berepeated. This process is not only costly, but it also involvesturn-around time that slows the creative process. So for me, what makesrealtime so valuable is its ability to support the creativeprocess.

Effects creation with tape-based video production evolved fromfilm's A-/B-roll model. A-roll and B-roll VTRs, an FX generator box,and a recorder VTR are required. An FX generator must operate inrealtime. With linear production, any effect on a 486-line, 720-line,or 1080-line frame must be generated in a frame-time.

The early online NLE systems used the same FX boxes. However, tobring costs down, effects box electronics were squeezed onto a computerboard. Special-effect PCI boards have been used by Avid and Media 100to lower the cost of realtime NLE systems.

The next step was Commodity LSI chips, like the MX-25 and MX-50 fromLSI Logic (originally C-Cube). These chips provide a small set ofsimple, permanently defined, realtime transitions. Pinnacle used theMX-25 chip by itself in the DV500 — to both generate effects anddecompress two streams of DV. Matrox used the MX-25 in theRT2000/RT2500 to decompress two DV streams, but used its Flex3Dgraphics processor to generate effects. The graphics processor approachoffered several advantages. Because it's used by gaming software tocreate realistic 3D graphics, a graphics processor offers verysophisticated and high-quality 2D and 3D video effects. And, unlike thelogic within the MX-25/MX-50 chips, the library of effects can beexpanded over time.

On the affordable end of the realtime spectrum is Adobe Premiere6.5. The application claims realtime capabilities, as do more expensivesolutions like Media 100’s 844/Xe. With so many realtime claims,buyers must examine what specific realtime features they need.

While Matrox and Pinnacle were developing hardware-based solutions,others such as Canopus and Sonic Foundry saw that computermicroprocessors were rapidly increasing in power. This computationalcapability could be used for two tasks. First, it could be used todecompress DV video. Second, it could generate special effects. Whilethis had the potential to lower the cost of realtime products by notrequiring special hardware, its true potential lay in the fact that ascomputer power increased, software-based products could play multiplevideo streams in realtime. (With today's dual Pentium 4 systems, theupper limit is five streams.)

Those who favor a software approach to realtime FX point out thathardware-based products are typically limited to only two videostreams. The hardware camp, while acknowledging this limitation,counters with the argument that software-based products can handle morethan two streams only when the effects applied are simple 2D effects orlow-quality 3D effects. The hardware camp also argues that an editor isnever certain when a timeline will be realtime. As this marketing waris certain to continue well into the future, let's move past itnow.

There's no reason for NLE designers to restrict themselves to eitherhardware or software. Hybrid solutions offer great benefits, asdemonstrated by the Matrox RT.X100. A CPU is ideal for decompressing DVbecause it allows the NLE to be more responsive. A CPU is also idealfor generating certain effects.

For example, most video FX chips work in YUV color space and don'tsupport RGB color alteration. That's why the Matrox DigiSuite's colorcorrection is limited to U and V phase alteration (“hue”)plus saturation adjustment. Other video chips — and most graphicschips — work only in RGB color space, preventing YUV phaseadjustments. (Now you know why MX-25 based products don't offer“hue” control.) A CPU can work in both color spaces.

Don't let the hype fool you. Making decisions about realtime isn'tconfined to software-versus-hardware issues. Decisions must be madeabout other requirements. For example, do you need an NTSC clientmonitor? Now that many NLE products offer sophisticated colorcorrection, an editor needs to be able to view a full frame on an NTSCmonitor. There are several ways an NLE can accomplish this.

For a desktop system, the capture (analog, FireWire, SDI) boardlikely offers analog and/or SDI output. However, if your NLE uses a PCIcard with only FireWire ports, or if the motherboard has a FireWireport, you've got another option. You can drive an NTSC monitor, viaS-Video, from some PCI or PCMCIA graphics cards. But here's where youneed to ask several questions. Can the graphics card outputhigh-quality video from the RGB overlay? And does your NLE applicationupdate the RGB overlay when it is playing the timeline in realtime?

The Matrox RT.X100 is a good example of a software/hardware hybridNLE. It uses a CPU to decompress DV and generate certain effects.

If your board has the latter capability, does your NLE softwaresupport it? Laptop users have the same questions to answer aboutbuilt-in S-Video output.

Laptop owners have discovered that a DV camcorder can be used as a“FireWire-to-analog video” converter. One of the newinexpensive converter boxes can be a cheaper alternative. (Of course,there is no reason desktop owners can't use the same strategy.) But youneed to check whether your NLE software will support realtime effectssuch as color correction while outputting compressed DV viaFireWire.

No matter how realtime is accomplished, there remains a major issue.No product can guarantee that every timeline can be played in realtime.Even the Media 100 844/Xe has limitations. If no product can guaranteeeverything in realtime, what is an acceptable compromise? Do you wantthe playback of many tracks of any complexity — even if thatmeans the playback will fall below 24/30fps — or even degenerateinto severe stuttering? Or would you rather always have perfectplayback, even though that means you must render complex timelinesegments before you are able to view them?

Avid's Xpress DV 3.5 gives you, the editor, the freedom to make thedecision about how you want a timeline to play back. If you can acceptthe possibility of slower than normal playback, just play the timeline.If the result is too slow or too unsteady to be of value, then renderthe area. However, by implementing a very fast codec and effectsgenerator, much like the Canopus software, Avid's realtime playback israrely poor enough to require rendering. Also, Avid has not restrictedrealtime performance to a subset of effects. All effects have thepotential to play back without rendering. Likewise, every effectattribute is available for realtime playback.

Apple has taken a different approach to realtime with Final Cut Pro3. First, only a small set of effects has the capability to play inrealtime. (And not all effect attributes are supported in realtime.)Second, FCP prevents you from playing a timeline that won't playperfectly. When FCP starts, a script is executed to determine CPU type(a G3 rules out realtime), CPU speed, number of CPUs, and the OS(according to the current script, OS X offers less performance thanOS9). The script sets a variable (cpuclass) that specifies yourMac's performance limit.

When effects are added to a timeline, each effect is looked up in atable to determine its performance cost. The total cost of all effectsis compared to cpuclass. If the cost is less thancpuclass, the frame can be executed in realtime. If the costexceeds cpuclass, the frame is marked as non-realtime and can'tbe played without rendering.

If Apple were to provide a way to disable this check, editors couldchoose to have FCP behave more like XDV. Hopefully Apple will providethis option in Final Cut Pro 4. Apple also needs to work on codecperformance so that all attributes of all effects can play without arender.

Commodity LSI chips, like this MX-25 from LSI Logic (left), are akey component in many realtime products, such as Pinnacle’s DV500production tools and Matrox’s RT2500 cards.

In the future, I expect some NLE systems to employ multiple,inexpensive graphics processors as effects generators. These hybridproducts — in conjunction with high-bandwidth IEEE 1394b harddisks — will support “many-stream” SD realtimeediting. These systems will also support “dual-stream” HDrealtime editing. Thus, just as Media 100 has introduced the lower-cost844/Xi at $25,000, other vendors will introduce higher-priced versionsof today's hybrid realtime products.

If I'm correct, unfortunately for those of us who have come toappreciate laptop editing, the gap between laptop and desktop realtimecapabilities will widen. Unless, of course, laptops of the future alsohave access to image engines. These engines could be part of the CPUchip as AltiVec is a part of G4 processors. Alternately, single ormultiple engines could be packaged into a chip. This could beaccomplished by manufacturers of mobile graphics chips. Or it could bedone by a computer company that manufactures laptops and also has ahuge stake in the success of laptop editing.

Contributing editor Steve Mullen is owner of Digital VideoConsulting, which provides consulting and seminars on digital videotechnology. Mullen can be reached at d-v-c@mindspring.com. His website is www.mindspring.com/~d-v-c.

To comment on this article, email the Video Systems editorialstaff at vsfeedback@primediabusiness.com.