The Future of Web Video, Part 2

Welcome back. In our last edition, I detailed what HTML5 is and why it's important and set the stage for my analysis of Ogg Theora, which was supposed to be the focus of this second part. But a funny thing happened on the way to the second part of this story. Specifically, I read the penultimate paragraph of my previous piece, where I wondered aloud how efficient Flash 10.1 will be and how playback in YouTube will compare between the new HTML5 versions and the Flash version.

Related Links

The Future of Web Video, Part 1 I admit that the nexus is a bit tenuous, but if you're an Apple Final Cut Pro producer, you're undoubtedly producing for the Web. Over the past few months, you've been hearing that Flash is going away, H.264 is going away, and soon you'll have to produce all your video in an open-source video codec... A Coming War Over Web Video? You could hear the jeers the instant Apple introduced its iPad last week. It was a common refrain: "No Flash support? What's the point?"... Distribute Expertise: Flash vs. Silverlight Since its introduction at NAB Show 2007, Microsoft Silverlight has achieved a number of high-profile design wins including mlb.com, nba.com, and NBC's...

So, I spent some time playing and comparing, with some interesting results. I'll summarize here, and you can read my analysis, titled "Flash Player: CPU Hog or Hot Tamale? It Depends," for the complete picture.

• First, on Windows, Flash Player 10.1 was a home run, dropping the CPU required to play back the Flash-based video by up to 73 percent. That's because Flash Player 10.1 can access the graphics processing unit for hardware acceleration during playback in a window (acceleration was only available for full-screen playback in previous versions of Flash).
• On the Mac, in Safari, the HTML5 YouTube player was about 66 percent more efficient than Flash-based version. However, as I mention in the article, Adobe can't access hardware playback on the Mac, while Apple can in Safari, which leads to the performance discrepancy.
• In addition, Safari, playing the Flash version, was significantly more efficient than Google's Chrome playing the HTML5 version.

As I comment in the article, "It seems reasonable to assume that if the Flash Player could access GPU-based hardware acceleration on the Mac (or iPod/iPhone/iPad), the difference between the CPU required for HTML5 playback and Flash playback would be very much narrowed, if not eliminated."

The article concludes, "Overall, it's inaccurate to conclude that Flash is inherently inefficient. Rather, Flash is efficient on platforms where it can access hardware acceleration and less efficient where it can't. ... At least from a CPU-utilization perspective, Flash isn't bad and HTML5 isn't good. It all depends upon the platform and implementation."

One major objection to Flash has been the CPU load, and historically, it's been very high. With 10.1, however, Adobe finally found the magic bullet—hardware acceleration—that reduces the load considerably. How this will impact the clamor for HTML5 is anyone's guess, but it's certainly a strong point in Flash's favor.

Now back to our regularly scheduled program.

Figure 1. H.264 is clearly more efficient than Ogg in these two examples. Ogg is on the left in both images.

Now, assuming that you decide to implement HTML5, the question becomes which codec to use: H.264 or Ogg? Clearly, the politically correct answer is both, which gives you the broadest browser reach, though it looks like H.264 will be the ultimate winner for multiple reasons.

First, understand that of the big four browser vendors, Mozilla Firefox is likely the only vendor adversely affected by H.264's licensing cost. Why? Because MPEG-LA's H.264 royalty structure has a $5 million maximum per vendor per year. Since Microsoft includes H.264 playback in its Silverlight plug-in and Zune devices, and H.264 encoding in its Expression Encoder, it's likely that Microsoft is already at the maximum. If the company decides to support the video tag, H.264 costs them nothing. Apple is almost certainly at the maximum with iPods, QuickTime, and the like, and as mentioned, Safari already supports H.264 via the video tag.

Google may not be at the maximum without royalties relating to Chrome, but it certainly can afford the $5 million much easier than Mozilla. Besides, YouTube has adapted H.264 video for most higher-resolution streams, and Google could hardly ship a browser that didn't support the format used by its own high-profile subsidiary.

Could YouTube switch to Ogg? Not if you believe Chris DiBona, an engineering manager at Google, who said: "If [YouTube] were to switch to Theora and maintain even a semblance of the current YouTube quality, it would take up most available bandwidth across the Internet. The most recent public number was just over 1 billion video streams a day, and I've seen what we've had to do to make that happen, and it is a staggering amount of bandwidth."

Though I couldn't confirm DiBona's math, my own quality comparisons did confirm that H.264 is superior to Ogg. In tests reported at my site, I found H.264 clearly more efficient in many scenes. Let me emphasize, however, that for talking head or low motion videos produced at fairly high encoding rates, Ogg is quite competitive, and only a step behind VP6 or VC1. It's also improving rapidly.

On the other hand, Ogg also lacks support among the most commonly used editing and encoding programs, and is too hard for most non-technical users. I ended up trying three different encoding programs—two command line, one QuickTime plug-in—and it was challenging. Today's video producers don't want challenging; they want the codec in their favorite batch encoding routine, or at least in a usable GUI on their favorite editing platform—Mac and Windows— and Ogg isn't close.

Ogg also lacks hardware support for live encoding and hardware acceleration for playback. Ogg fans (short for fanatics) will tell you all this is coming, and it may be. But it certainly doesn't appear to be here yet.

In addition, there have been rumblings out of MPEG-LA that the "submarine" patents are about to surface. Apple, in particular, has been concerned about these patents for years, as evidenced by this exchange between Ian Hickson, editor of the HTML5 specification, and Manuel Amador, an active poster on the spec's website.

Amador: "The Xiph developers were extremely zealous and almost fiduciary diligent in researching all possible patent threats to Vorbis technology, and for more than a year they found none— they even did the research *before* beginning to code, explicitly to avoid submarine patents."

Hickson: "While this is very true, and admirable, and impressive, it is sadly not a guarantee. Certain companies (Nokia and Apple among them) have reported that they still fear that undisclosed patents may exist that cover the relevant codecs, as they might exist for other formats like MPEG4/H.264. The difference is that while Apple (for example) have already assumed the risk of submarine patents with H.264, they currently have taken no risks with respect to the aforementioned codecs, and they do not wish to take on that risk. Given the extremely large sums of money that are awarded for patent violations (cf. Microsoft's recent settlements), it is understandable that companies with the high profile of Apple and Nokia would not wish to take on such risks."

I last spoke to the MPEG-LA folks when they announced that they were extending the free pricing for free internet video encoded with H.264, and there were indications that some members of the H.264 patent group felt that Ogg infringed upon their patents. It's a story I'm developing and will write about later, but here's a short quote from an MPEG-LA spokesperson: "Virtually all codecs are based on patented technology, and many of [Ogg's] essential patents may be the same as those that are essential to AVC/H.264."

I shared those thoughts with Monty Mongomery, the founder of Xiph, who commented, "MPEG-LA has had more than 10 years to say something substantial on that front. They have not."

I have no idea what's going to happen, and I have no desire to debate the fine points of patent law with anyone. Still, I don't mean to go conspiracy theory on you, but Apple is in the H.264 patent group, and (I presume) would know if the group considered some of Ogg's patents as infringing. Apple has consistently cited the potential for submarine patents as one of its main objections to Ogg—perhaps because the company is or knows of a submarine patent owner. We'll have to wait and see if the dots ever connect.

Figure 2. Firefox can't play YouTube video with the HTML5 page.

Most importantly, of the big four browser vendors, Mozilla is the only one with a real financial interest in promoting Ogg, while Microsoft and Apple are already committed to H.264 encode/decode, and Google to H.264's stream efficiency. Google has already adapted Ogg, but Apple seems to have decided to avoid Ogg at least until all submarine patents are resolved. Microsoft has been completely opaque on the issue, but also sits on the MPEG-LA H.264 committee, and seems unlikely to support Ogg in Internet Explorer until all patent issues are resolved. That means today, buggy Google Chrome aside, Ogg plays in less than 35 percent of all browsers.

Firefox is my browser of choice, and in general, I support Mozilla's efforts for standardization and a "free" web. In this instance, however, Mozilla's refusal to license H.264 may force all websites wishing to support HTML5 to encode two streams of video, easily doubling the related storage and administration requirements. To achieve the same quality as H.264, Ogg will require significantly more bandwidth than H.264, another significant cost to producers who adapt it. Though this is just a guess, it feels pretty certain that these storage and bandwidth costs would far outweigh the $5 million in royalties that Mozilla would have to pay to support H.264. In this regard, Mozilla's stance is almost certainly doing more to retard the future of HTML5 than to promote it.

Going into Carnac mode, what does the future hold? Well, as much as Mozilla likes to play the nonprofit card and its goal of making the "Internet better for everyone," the organization actually grossed more than $78 million in revenue in 2008, with 2009 results as yet unreported, according to internetnews.com. Most of the revenue comes from search fees from Google and other sites. There's nothing wrong with making money, of course, but this revenue highlights the fact that Mozilla is a business just like any other, and the company has to keep its browser competitive to keep the dollars rolling in and the doors open.

I would guess that over the next 12 months, submarine patents on Ogg will begin to emerge, and Mozilla will be forced to either dispute the patents or pay for either Ogg or H.264. Even if this doesn't happen, Firefox's market share may start to decline as web publishers balk at encoding both H.264 and Ogg files, particularly if Internet Explorer starts supporting the video tag with H.264 video. At that point, Mozilla may be forced to choose between the purity of its philosophy and economic viability.

Ironically, if Mozilla doesn't adapt H.264, the major proponent of the HTML5 effort may end up depending upon Flash for its ability to play H.264 video and its continued relevance. Sound far-fetched? Today, the only way to play video from YouTube— by far the world's most popular video-related website— using Firefox is via the Flash plug-in.

Finally, just to confuse the issue even further, Google just completed its acquisition of On2 Technologies, most notably the VP7 and VP8 codecs. Already there have been cries for Google to open source the codec. I haven't seen VP8, but VP6 is superior to Ogg as it stands today. If VP8 includes the quality improvements you would expect, and Google does donate it to the open source community and replaces H.264 with VP8 in YouTube, it could stop Ogg development in its tracks.