For more than three decades the PBS series Frontline has taken viewers to unusual, exotic and often dangerous places. Recently the series experimented with the documentary form itself for “On the Brink of Famine,” a virtual reality (VR) program that was released as part of a series of 360° Facebook videos. Filmed on the ground in war-torn South Sudan, the series transports viewers inside a hunger crisis devastating the region.
“Famine” is supported by Frontline and by a grant from the David and Helen Gurley Brown Institute for Media Innovation, which is a collaboration of Columbia and Stanford Universities.
Frontline executives have been interested in the possibilities of VR in journalism and documentary filmmaking for some time. “I’ve seen with my own children that they have different expectations from their media than I did at that age,” observes Raney Aronson-Rath, the Boston-based executive producer of Frontline. “They expect absolute immersion in an environment, as opposed to sitting back and watching it. They were frustrated watching TV that they couldn’t touch the screen. This made me pay serious attention to investigating what the next generation of documentary might be like.”
The VR aspect of this project was a significant departure for all the filmmakers, whose work has been in forms of traditional journalism. “This was the first time using VR for a documentary for all of us,” says Marcelle Hopkins, Famine’s New York-based writer/director. “The learning curve was very steep. We talked to anyone who knew anything about VR before we started,” she recalls.
Working for Al Jazeera’s United Nations bureau, Hopkins had been covering South Sudan for years—since before it had become a separate country in 2011. She brought on Benedict Moran, then also a reporter for Al Jazeera, as producer. Moran had been a humanitarian worker and had lived in the region prior to embarking on a career as a journalist in print, radio and documentaries. To round out the crew and address the myriad technical issues, she also included Evan Wexler, who’d worked previously at Frontline as a digital designer.
The team approached the project methodically. “We storyboarded what we thought we’d need for our narrative before arriving in South Sudan,” Moran recounts. Despite the preparation, however, they found that their preconceptions about documentary filmmaking simply didn’t hold up in VR.
The filmmaking team’s Freedom360 camera rig held six GoPro HERO3 cameras in a tight cubic geometry to film material for fully spherical 360 x 180 video.
“My inclination was to find experts to define the historical context, to speak directly to a spokesperson—the president or the leader of the rebellion—because those voices would add authority to the story being told,” Moran explains. But he would find that he needed to shift his focus away from words and toward immersive experiences to get the most out of the medium.
Hopkins’ understanding of documentary storytelling similarly evolved with this project. “I was thinking ‘OK, we need B-roll. I’ll shoot tractors and farmers and that kind of thing to cut in at appropriate times. But I realized that was the wrong way to think about a VR project.”
The world of VR is still in its infancy today—and in the summer of 2015, you could say it was embryonic. Working with multicamera rigs, setting appropriate exposure and frame rates, stitching the results together to look at as dailies and subsequently for final delivery—there was no established workflow or professional-grade toolset for any of it.
Managing the Production Process
The crew carried two Freedom360 Explorer rigs to South Sudan in the blistering heat of late summer to begin production. Freedom360 is a mount that holds six GoPro HERO cameras in a tight cubic geometry to film material for fully spherical 360 x 180 video.
Wexler, who oversaw technical issues, had spent several months researching 360° shooting and applied as much knowledge as he could to the project. He housed GoPro HERO3s in the rigs, even though the HERO4 was available at the time. “I was concerned about the thermal cutoff on the 4,” he explains. “I tested it and got advice, and people said [the HERO4s] could shut off after 15 minutes. And it’s hot and wet in South Sudan. The 3s were less likely to just shut down, so we felt more comfortable with them.”
Each camera was set at 1440 resolution with a frame rate of 48 fps. Projects for immersive presentation tend to benefit from being shot and played back at higher frame rates than the traditional 24 or 30 fps we’re used to.
GoPros are basically fully automatic cameras that determine aperture and shutter speed based on presets in the onboard chip. The cameras have a built-in “manual override” of sorts, however, when they are operated in Protune mode, which allows the operator to program some settings.
“Don’t go all automatic,” Wexler warns. “If one of the cameras is pointed at the sun, for example, it’s going to compensate with the iris or shutter or both, and then you won’t be able to reconcile that camera’s images with what comes from the other cameras in the rig. Protune let me set aperture and shutter speed the same for all the cameras.”
After a shot was completed, Wexler would import the data from the six individual microSD cards to his MacBook Air and back it up on an external drive. He used Autopano Video software from Kolor (subsequently acquired by GoPro) to stitch the images together quickly so the filmmakers could see what they had. Footage was stitched into an equirectangular or “dome” master file that represented the output of all the cameras in the rig as a flat Mercator projection on his computer monitor. This version was a long way from the fine-tuned stitching work necessary for final delivery, but it gave the team a fair idea of the status of the work in progress.
In a segment of the film, viewers are taken on a critical mission to deliver food by plane to those trapped in the swamplands of South Sudan.
The filmmakers’ notions about the project’s form developed as shooting progressed. “The idea is that the audience is a ‘floating brain’” Moran explains. “We want to minimize any evidence that the documentary-maker is there.”
One setup that made it into the final version of the film required the filmmakers to place their camera rig in the middle of the drop zone where 50 kilogram sacks of sorghum would land after being heaved out of a low-flying airplane
“If we shot from more than 15 feet away, everything looked incredibly distant,” Wexler says. “To get a really good shot, we needed to put the rig smack-dab in the middle of the drop zone.”
There was a roughly five-minute interval between payloads being dropped onto this giant X in the middle of a field, and the best place for the shot was in more than a foot of water. When the bags would land, the water would just explode. “We placed the rig, pressed record, clap-synced and ran away,” Moran recalls. “We were lucky. One bag landed literally a meter away and almost destroyed our cameras. But we got the shot.”
That was the type of process used on all the shots for the sequence: something from outside the cargo plane and a shot from inside the sacks being dropped. “Sometimes it took all day for one shot,” Moran says.
The Red Cross airdrops rations in a remote area of South Sudan where many people have moved to escape the continuing violence.
Postproduction for 360° Content
After shooting wrapped in late August, the team returned to New York, decompressed for a couple of weeks and then started editing. They delivered a locked picture in mid-December.
Knowing that it would take a significant amount of fine-tuning to get the six flat images manipulated and stitched together sufficiently well to work as a 360° experience, they started assembling their documentary with the rough-stitched versions that Hopkins, Moran and Wexler generated using Autopano in the field. They would then import these images into Adobe Premiere to edit shots together.
“We really tried to get a tight edit,” says Hopkins. “It was a very involved process stitching the pieces together for the final cut.”
Autopano only started the process. Wexler made use of Adobe Premiere, Adobe After Effects and The Foundry’s Nuke to contort and stretch each of the six images to work together seamlessly as a 360° presentation. Objects moving from one camera’s POV to the adjoining one might appear to be moving at different speeds or have different geometric attributes because of the positioning of the different cameras, distortion of the lenses and a host of other attributes that had to be smoothed out.
“That’s the really big challenge,” Moran notes. “That’s what’s holding the industry up at the moment, I think. The software is great, but it still has a lot of bugs and requires a technical expertise to fix shots. Sometimes shots came out great and needed almost no modifications, but other times it took a whole week to get one 20-second shot to work.”
In the editing process, Hopkins and Moran discovered that their initial impulse to steer clear of the standard tools of the documentary trade—talking heads, cutaways and voiceover—was appropriate but not quite strong enough. “We had to keep simplifying it,” she says. “We took out a lot of the information we wanted to include to provide context. We initially put in way too much.”
In testing the piece as a 360° interactive experience, the team learned how very differently people consume this type of presentation compared to a traditional documentary. “People are almost overwhelmed by their agency in this medium and are often not able to absorb information that comes through audio,” she says. “It’s almost something that’s biological in the brain. If you’re immersed in an environment wearing a headset and being bombarded with visual cues, your brain is just not going to process everything you’re hearing.”
Red Cross cargo plane
After screening their 15-minute cut, Moran recalls, “we’d ask people what they remembered. And they didn’t remember any of the details about what anyone said. They remembered impressions. The information felt like overload to them. It was a long process getting it into the simpler, more understandable [presentation] it became.
“I think that as computing power and these tools get more advanced, we’ll be able to do much more in the medium,” Moran sums up. “Most of what we’re doing is a hack. We’re taking tools made for something else and adapting them for VR. This is the beginning of a form. It will be fascinating to watch as linear documentary storytellers do more work in VR and adapt their expectations of how things work intellectually and emotionally, because it is actually very expansive, but it’s also extremely restrictive at the same time.”
Another section of the film takes place in the country’s largest displaced persons camp, where more than 120,000 have fled to escape violence.
“On the Brink of Famine” was released on Frontline’s Facebook page in three stages. In the first story, viewers are taken on a critical mission to deliver food by plane to those trapped in the swamplands of South Sudan. In the second 360° experience, viewers visit the country’s largest displaced persons camp, where more than 120,000 have fled to escape the violence. The full documentary, which also includes a look inside a clinic where Doctors Without Borders cares for children suffering from acute malnutrition, is available now.