On the Line

 Web Expanded

Photo Gallery

Denver CBS affiliate KCNC put its new XDCAM HD
infrastructure to the test at the Democratic National Convention. Cameraman Bob Burke captures the moment when Hillary Clinton nominated Barack Obama.

When Barack Obama turned up on stage in Denver's Pepsi Center at the Democratic National Convention (DNC) to congratulate his running mate, Joe Biden, he explained why the convention would move to Invesco Field the next day. “Because it's about ground up, not top down,” he said of his decision to accept the nomination in front of a stadium-sized crowd. It was, he said, about ordinary people doing extraordinary things.

As Obama looked out across an audience peppered with wireless camera operators, toward a wall of skyboxes filled with exhausted broadcast engineers and producers, he may not have realized that his words would start coming true the minute he stepped off the stage. In the coming 24 hours, ordinary people would need to do some extraordinary things.

NY1 cameraman Kevin Dugan takes aim at the vast crowd in Invesco Field shortly before Obama''s acceptance speech.

The last time broadcasters had to navigate a two-venue political convention was in 1960, when John F. Kennedy accepted the Democratic nomination at the Los Angeles Memorial Coliseum. In the 48 years since, the parties periodically threatened second venues that never materialized. So when the Invesco announcement came, just seven weeks before the convention was to open, it was hard to believe it was really going to happen. Broadcasters who had already spent a year preparing for unprecedented back-to-back Democratic and Republican conventions would now have to find a way (and a budget) to squeeze a third venue in between.

And they would have to do it — effectively and literally — in the dark. Information about the move from the Pepsi Center to Invesco Field was understandably scarce and hard to pin down. The security requirements of a sports stadium took precedence over broadcast infrastructure — and daylight. Broadcasters would be accommodated with an overnight move that would be inconvenient and painful. It could be no other way.

Neither station—or any of the world''s broadcasters—knew until the last minute that they would be making an overnight move from the Pepsi Center to a crowded camera riser in Invesco Field.

***

When dawn came to Invesco Field, local Denver CBS affiliate CBS4 (KCNC) was live from the Broncos' 50-yard line. Against a background of 76,000 empty seats and a sprawling Greco-Roman stage platform where the candidate would accept his nomination, anchors Brooke Wagner and Tom Mustin, engineer Robert Garibay, and field producer Libby Smith kicked off the day with one of the earliest live newscasts to broadcast from inside the arena. It was just one Sony XDCAM HD and a mic; the lights had to be quickly changed from 3200K HMIs to 5600K Kino Flos as sunlight started to pour in; the teleprompter caught up about 10 minutes late. But they were live and they were on the air, exactly as assistant news director Kristine Strain had dared to plan.

That accomplishment wouldn't be worth mentioning except for the sequence of events that started 6 hours earlier. At about 11 p.m., a handful of engineers who had already worked all day began to strike the Pepsi Center skybox gear. The deadline was 12:30 a.m., when the secret service would escort the last truck into the Invesco parking lot and circle the wagons. Everybody was pressed for time. They were also pressed for elbow room. With Melissa Etheridge and Jesse Jackson amassing crowds just outside the skybox doors, it was initially impossible to get the gear out of the rooms and down the stairs to the convoy.

Cory Baker, who would set up KCNC's opening shot, had left the Pepsi Center earlier to get some sleep. “As I was leaving, I pictured them packing the ‘Cory box'' with everything I'd need for the morning news,” he says. No time for such niceties.

When the secret service reopened the Invesco parking lot at 3 a.m., Baker found the truck, slid open the doors, and was staring down a wall of gear. “I needed a camera, a mic, some XLR jacks, and a teleprompter,” he says, “and I just had to start digging.”

Local 24-hour cable news network NY1 stood gamely alongside its flashier Time Warner colleagues at CNN, in the company of national networks and other heavyweights on Skybox Row in the Pepsi Center.

By 8 a.m., the main camera riser was so crowded that some producers had given up on picking their way through the cable and tripods and settled into the position they would occupy for the next 15 hours. The engineers had to keep moving. Microwave signals were dropping out — jammed by secret-service traffic and just plain overcrowding. Or was there just a bad connection somewhere in the cable spaghetti? Maybe the new engineer back at the station was doing something wrong? Why was there no audio sync?

The world's broadcasters were cheek-to-jowl, quietly problem-solving and generously jockeying with competitors in a too-small space. On-air reporters in suits and melting makeup climbed over the 10ft. temporary railing to use the bathroom or get water. The Rev. Al Sharpton looked like he might split his pants as he clambered over with the help of a chivalrous IT guy, navigating the riser obstacles to make his interview with NY1.

KCNC pushed the boundaries of local-level coverage, supporting anchors Karen Leigh and Jim Benemann with a five-source HD switched feed.

By now, NY1 producer Jeremy Bitz had gone past the point of stress and returned to a kind of Zen resignation. The previous day had actually been the hardest. An intense day 3 in the Pepsi Center, the overnight move, and the constantly changing security and access situation had all resolved into this moment — live at 3 o'clock in the afternoon. “It will be what it will be,” he said, smiling over his shoulder as he shoved his defeated cell phone in his pocket and reached for a landline plumbed into the camera riser.

NY1 producer Jeremy Bitz and reporter Josh Robin keep things in perspective midday at Invesco Field.

For NY1, the convention was already a gamble they had to take. In January, when they had to make infrastructure commitments, it was possible that the convention season would be all about New York: maybe even the senator from New York state vs. the former mayor of New York. For a 24-hour New York cable station, either one would justify the investment. So Time Warner's Steve Paulus bet big on the coverage: The regional VP of local news committed to a skybox next to Time Warner sibling CNN — no carpet, a fraction of the space, but they were there, on-air when Hillary Clinton nominated Obama. A three-camera anchor desk overlooked the convention floor, the wireless cameraman had an RF channel and a position near the New York delegates, reporters were roving outside and filing stories throughout the day; the entire NY1 political bureau was on the job in Denver.

In addition to the skybox, NY1's workspace trailer in the Pepsi Center parking lot was outfitted with laptops running Apple Final Cut Pro in support of NY1's standard tapeless DVCPRO 25 4:2:0 P2 workflow (Panasonic AJ-SPX800s and AG-HVX200s). An Omneon shared-storage infrastructure that mimicked the one NY1 reporters are migrating to back home filled a 20RU flypack that hovered near the door, outlasting a wheezing refrigerator that gave up on day 2.

Based on Omneon's Spectrum media server, the NY1 network allowed realtime ingest and playout supporting six Final Cut Pro edit systems linked by a simple Ethernet topology. With the Spectrum, reporters could edit during ingest; material was immediately available (as ProRes 422) to all Final Cut Pro seats simultaneously without transferring or copying files — the Spectrum simply mounts as a network drive. (Reporters could also ingest locally and access shared elements as needed.) Completed stories were formatted as .mov files and then sent as baseband video via fiber to NY1's Manhattan facility for transmission and webcasting. Because sequence editing was performed as usual — users opened and saved content on the Omneon file system through standard FCP controls — the system had the necessary simplicity to support the NY1 reporters, sister stations, and client stations.

Director of IT Bob Ruth delayed his flight to St. Paul to be on hand for the inevitable troubleshooting that extended past NY1 to colleagues who were quick to help each other in the shared chaos.

This was important because the business plan was funded in part by a service model: The switched feed (via Sony Anycast Station) and a wireless feed were available to client broadcasters — including India's Star News, New England Cable News, News 8 Austin in Texas, News 14 Carolina in North Carolina, The Charlie Rose Show, and In the Know — as well as NY1's sister stations Capital News 9 in Albany, News 10 Now in Syracuse, and R News in Rochester. There was a communal standup squeezed into the right corner of the skybox next to the three-camera anchor desk. The editing workspace and its IP-based distribution infrastructure were shared by all in relative simplicity despite the variety of production formats (including tape), distribution paths, and encoding/file requirements.

Most of the production infrastructure was relatively conservative and battle-tested; veteran integrator Arctek HD of Minneapolis outfitted the skybox. The service-bureau model is not uncommon for big events; it was NY1's second time doing it for a national convention. The biggest gamble was the transmission infrastructure, says NY1 Manager of Remote Operations Gerry Gallagher. He's a microwave man who still remembers the shock of seeing his first scope emulator, but he's ready to deploy a Streambox (or even a Slingbox) and book a transcontinental IP pipe when he can't confirm satellite time.

NY1 Director of Technical Operations Mike Chan prepares to direct the switched feed.

So it was in Denver. When the distribution plan was made, satellite time was sketchy and Qwest was ready with a guaranteed cross-country DS3 pipe. This suited Director of IT Bob Ruth in part because of the Time Warner Cable multicast network he had helped build and was familiar with. Time Warner's 3-year-old Multi-Service Aggregation Network (MSAN) is a 10GB full duplex video-transport/multicast and data network that, among other things, links NY1 with Capital News 9. This meant that last mile to NY1 was already reliably in place. The Denver and St. Paul media went to the Qwest POP in Albany and then into Capital News 9 via Time Warner Telecom. Once in the building, it went across the internal network to NY1; NY1 in turn sent it via Vvyx to client network News 14 Carolina. Ruth could use the CNN network that comes into Capital 9 to send to News 8 Austin. Confident in NY1's Tandberg encoders and decoders, with the hard part already tried and tested and with last mile costs eliminated, it was not so scary (and relatively affordable) to add the transcontinental runs from Denver and St. Paul to Albany.

NY1 Manager of Remote Operations Gerry Gallagher at Invesco Field.

NY1 had done Gov. Eliot Spitzer's inauguration in January 2007 as all-IP transmission, but with satellite time booked as backup. This time, no belt and suspenders, but the team was pretty confident. It was, Gallagher says, “swimming in a pool we were already dangling our feet in.”

“There was nothing preventing it from working,” Ruth says. “But until you run it up, you can't be totally sure. When I got all the paths up, I was happy. I had committed my reputation on this project.”

When it came time to move to Invesco, the transmission plan had to change. Now it was Qwest that couldn't guarantee the line, but the satellite provider could commit. The edited material could still travel from the trailer via IP, but the live switched feed would come from an Arctek satellite truck in the Invesco parking lot.

Arctek integrators Ron Rausch and Jim Berg in the skybox at the Pepsi Center.

By midday, Bitz was splitting his time between the overcrowded camera stand and the parking lot, where his director's chair sat under a capacious ripstop canopy not far from the ice chest — almost as if the satellite truck were parked in an RV campground instead of outside a major news event. Getting that far came after what Gallagher describes as “an almost surreal night,” full of mundane and time-consuming quests for power drops and phone lines, the ordinary tasks of broadcast operations overseen by “people in black suits with very large guns.” No one had been to bed, and the Arctek guys were grumpy with the 1/4in. audio inputs on the Anycast, as you would be. Back in the five boroughs, no one on the other end of a TV could know any of this; they were simply watching live coverage of something that was happening more than halfway across the country — on their local cable station.

KCNC Engineering Manager Eric Buckland multitasks on the riser at Invesco Field, swapping advice with Mark Doan, chief engineer at the Southern Colorado CBS affliate KKTV.

***

Sometimes a broadcast engineer just needs a welding torch. When the Pepsi Center inspectors saw the (expensive) lighting truss framing the KCNC skybox balcony, supporting a whopping 6lb. payload of lights, they just said no. Engineering manager Eric Buckland stopped by Home Depot on the way home, bought some steel bars, and fired up the MIG welder in his Denver garage. The new lighting frame fit nicely around the anchor desk — inside the set itself and outside the jurisdiction of the floor inspectors.

Engineer Chris Jones was key to building out the Invesco setup from one camera to five in time for primetime.

It was not Buckland's first Boy Scout move. When Denver was announced as the host city, he decided to join the volunteers on the RF coordination committee and help hammer out a plan for the torrent of signals that would ricochet through the convention facilities. “I thought it might help me get what I needed,” he says candidly. As he talks, it's clear he liked the whole business of solving the puzzle. Although Buckland was not part of the enforcement team, a spectrum analyzer in the KCNC skybox attested to how seriously the FCC and the DNC would take bandwidth bandits. For his part, Buckland minimized KCNC's RF exposure to the wireless cameras; at the Pepsi Center, they eschewed microwave in favor of fiber. “It seems like we made the right choice,” Buckland says, noting that on day 1, engineers at the KCNC studio 3 miles away were reporting wireless interference. Yet even if KCNC had wanted to use its fixed microwave links, they were unavailable; the Pepsi Center access points were inside the secret-service perimeter. Off limits.

“There were hardships and there were breaks, just like always,” says KCNC Chief Engineer David Layne.

Perhaps the biggest break came when it was time to pull the Pepsi Center cable. Based on KCNC's experience at the New York and Boston conventions, Layne had estimated something in the $40,000 range for the 22 pulls they were planning. When the estimate came in at $387,000, he says, “We thought they were missing a decimal point.” Layne offered to supply the cable; the price dropped by $5,000. “It was more than our whole budget,” he says. Even in the “heartfelt” negotiations that followed, even when they revised the plan to multicore cables, the price stayed out of reach. That is, until Layne understood that any union labor could do the job. Incredibly, KCNC is a union shop (a legacy from its GE ownership); the station's own two guys did the pull in 12 hours.

Production trucks and Sony Anycast Stations alternated in the same role for both KCNC and NY1.

Altogether, the convention coverage was the boldest effort KCNC had ever put on. Newly converted to all XDCAM HD (the station went live HD in April), the station faced its first major HD remote outing and the first major use of HD wireless, with two positions on the convention floor. Assistant news director Strain and her boss, Tim Wieland, handed Layne and Buckland an ambitious editorial plan. “Denver's a competitive market,” Strain says of the plan, including the goal to be HD and the 5 a.m. broadcast from Invesco. “And we had the additional responsibility to cover the town. In some ways, viewers care more about how the convention is affecting Denver than what is going on inside.”

With that in mind, Strain wanted the trucks on the street, so the Sony Anycast proved to be an invaluable tool in the skybox. “We're doing everything you would do out of a production truck out of a suitcase,” she says. KCNC also put together a set and a wireless position in Denver's Larimer Square on the roof of the Tamayo restaurant, where the team captured crowd scenes and then riffed from the ground on at least one night, preempted on their own set by one of the many convention parties.

At Invesco, it was an advantage being local and a coincidence that one of the freelance engineers, Kimo Quaintance, had been on the team that designed the fiber infrastructure at Invesco. Here, KCNC could — and did — use its two existing fixed microwave links. They weren't the greatest positions considering the air traffic, but after some tense hours of troubleshooting, they did the job.

Nearly forgotten in the harried preparation was that there would be a payoff of sorts. After all the engineering was done, the world's cameras turned to capture one of the most visually spectacular political events in history. On that camera riser, all were pretty much equal — the Tokyo station with its lowly Betacam SX, the mighty BBC, the networks, the Brazilians, CNN, and all the rest. Ted Koppel stood next to NY1's Dominic Carter, next to Jim Lehrer, next to KCNC's Karen Leigh and Jim Benemann. And they were all framed against the same stadium full of people and lights and fireworks on a history-making night.

After 24 hours of wrangling pixels, signals, feeds, and light, nearly 15,000 broadcasters and journalists reached the end of one very long day at work.

***

As the tempest of confetti settled into the Invesco crevices, NY1 went back into hyperdrive. While Arctek packed the satellite truck, all the gear — now split between the Pepsi Center and Invesco Field — had to be packed into the Penske truck and on the road by 3 a.m. to make Saturday's secret-service cutoff at the Xcel Energy Center in St. Paul. While outlets such as NBC network could supply and staff two virtually separate crews, NY1 Denver was now NY1 St. Paul. Such is the lot of the cable guys — even Time Warner Cable guys. For a station built on the loyalty of the five boroughs, Bitz says, the political coverage — and the national scale — is a differentiator. In many ways, it's more than that, a post-Sept. 11 mission in a city thrust into the crucible of modern geopolitics. For NY1 and its regional partners, it's also part of competing in the IT-driven, niche-sensitive world of broadcast as boundaries among local, national, and international coverage blur into an IT cloud.

It was an expensive couple of weeks, and Paulus didn't say it, but the Invesco move had to have hurt some. He does volunteer that it “hasn't been a slow year,” with the Pope Benedict XVI visit, the Giants' Super Bowl victory, and the Governor's … his voice trails off. The Invesco satellite truck, the extra Qwest phone lines, the cable pulls, and the overtime all had to layer onto a business plan based on two venues, not three, in a news cycle that is far from over. History has its price.

Of course, NY1 wasn't the only organization that had to stretch, and it is the news community's job to follow the news — across the street if necessary. But watching everyone suck it up and pull their weight, regardless of personal politics, it seemed clear that working people were already doing extraordinary things just by doing their jobs.

***

Eric Buckland was leaning on a wall outside Invesco having his last cigarette of the evening, but his mind was already in Phoenix, where KCNC was getting ready to broadcast a Broncos-Cardinals preseason game the next day. “It's our show, because it's preseason,” he says, “and we'll have a couple tricky things to pull off at the station tomorrow.”

All images by Joshua Touber