HTML5 audio / video and mp3 / H.264 is the future of new media but does not replace Flash

HTML5 is the future of new media (also known as downloadable media and podcasting). Anyone who has used an iPad or Google Chrome and watched a video knowing it was through an HTML5 video element knows what I’m talking about. Playback is instant, smoother and is much more responsive than via Flash. It is also very easy to develop in your web pages, removing a level of complexity that was previously much more involved and relied upon Adobe Flash.

The remainder of this post explains everything in detail, why use one format over another, what Flash is still good for, and where the media can end up.

Note: An update to this original post may be found at the bottom of this page.

What is HTML5 and what’s the deal with audio/video?

HTML5 is a new version of HTML (markup that creates web pages that you see in your web browser) that adds a number of new elements (special tags that do things in your HTML, e.g. <p> tag indicates a paragraph of text). HTML5 adds two new elements called audio and video. These new elements allow for web developers like myself to easily add an audio/video player in a web page. Without the HTML5 audio/video element, a web developer needs to implement more complicated HTML utilizing the “embed” tags to include a Flash developed player in the web page. In this case, the Flash player is required to be installed on the end user’s computer in order for the audio/video to playback.

What audio and video formats should I use?

Audio: mpeg3/MP3 (.mp3 file extension) is, for the most part, the most ideal audio format. As of current, all but Firefox have/will have support for mpeg3 audio. AAC audio (.m4a) may also work, but if you are looking for a format that will work in absolutely every situation, mp3 is the best bet.

Video: H.264 (.mp4 or .m4v file extensions) is the most ideal video format. As of current, all but Firefox have/will have support for H.264 video.

Why is MP3 and H.264 recommended for audio/video?

MP3 is the audio format of choice because it is the most widely playable audio format. Nearly every media player application, portable media player, and automobile/car stereo can play mp3 files.

The MP3 format became widespread in 1997 when the Windows application WinAmp was released. Created by Justin Frankel, WinAmp made it easy for music enthusiasts to exchange and listen to audio on computers. The small size of MP3 files enabled widespread distribution initially via file sharing applications such as Napster and on wide-area networks such as university dormitory networks. It was such a popular format that Apple included support for MP3 in it’s first release of the iPod in 2001.

H.264 is the video format of choice because it is the most widely playable video format. Though not as widespread as the mp3 format, H.264’s common denominator is that it can be played on the most popular portable media players, MAC OSX, Microsoft Windows, Internet connected TV’s and smart phones. You can thank Apple for making H.264 the standard in its portable hardware (iPod/iPhone), which has dominance in the portable media player market.

What can Flash do that HTML5 audio/video cannot?

Live streaming! Many sites such as Qik and UStream.tv provide live online content, their use of Flash will not change due to HTML5 audio/video.

The difference between “downloadable media” and “live streaming” is the “live” part. Downloadable media is not live, which has an advantage to providing the consumer the ability to save the media and play it back at their convenience. Live streaming on the other hand, is real time audio/video playback that cannot be paused/played without the help of a device to record the content. For this reason, Flash may not be as important but still has the purpose of providing a means to stream live content.

As far as downloadable media is concerned, Flash is perceived as dead, but Flash will play a key role on portable devices such as Android phones in providing live streaming content over the coming years.

The Flash Video Secret

Though most know that Flash is used to play audio mp3 files, many web developers are not aware that Flash can play H.264 video. Since Flash version 9 released in the Winter of 2007, Flash has the ability to play .mp4/.m4v video. Before Flash 9, Flash could only play Flash video (.flv file extension) files.

As far as video is concerned, this solidifies the H.264 format as the most widely playable video format. It allows a web developer to alternatively allow its web visitors the ability to play H.264 video in the event the browser itself cannot play the HTML5 video format.

Google Chrome / Apple Safari

Both Google Chrome and Apple Safari web browsers support mpeg3 (.mp3) and H.264 (.mp4/.m4v). Safari has one glitch though, it will auto download the media files linked in the audio/video tags, which does bring up a challenge for web developers to deal with.

Internet Explorer

Currently, Internet Explorer (IE) does not support HTML5, but the signs back in Fall of 2009 were obvious that they were planning on supporting it in the future. With the recent blog post announcement for support for H.264 video in IE9 and past blog post announcing MP3/AAC audio support, it looks like the next version of IE is on its way of being HTML5 audio/video friendly. There is only one problem though, IE9 will only be available for Windows 7 and Vista, Windows XP will not have IE9 as an upgrade option.

Firefox and the OGG format

Firefox supports a niche audio and video format called OGG. The reason for this is simple, it doesn’t cost Firefox anything to support OGG formats. Since Firefox is essentially a free foundation and not a real company selling products/services, it does not have the money or resources to purchase licenses to include support for the H.264 video format. From the last post I read about the subject, Firefox would have to pay a 5 million dollar license fee in order to use the H.264 video format and it would still be limited to which versions of Firefox could include H.264 (source based compiled versions distributed through different versions of Linux would not be included in the license for example). It is a bit more complicated than this, but you get the idea why Firefox doesn’t support H.264. Read why Firefox does not support H.264.

The OGG format is a combination of a number of formats, two of which are supported by Firefox. The OGG audio format, also referred to as Vorbis (.ogg or .oga file extensions) is a truly open source audio format. The OGG video format, also referred to as Theora (.ogg or .ogv file extensions), is a free video format based upon a patent by On2 Technologies. As of current the patent behind Vorbis is not enforced, allowing the format to be used with out paying any royalty or fees. Example of a potential Theora problem.

I should point out that both Google Chrome and Apple Safari support the two OGG audio and video formats mentioned above. Internet Explorer, Apple iPod/iPhone/iPad/TV and most other portable media / TV hardware most likely do not support OGG, limiting this format’s reach in the market.

Firefox, H.264 and MP3

Firefox will most likely not support H.264 without help from Apple/Microsoft/Google. I predict by years end one of those companies will sponsor Firefox’s H.264 five million dollar license to include H.264 support in Firefox. There could also be a plugin for Firefox that provides H.264 functionality. More interestingly, Apple/Microsoft and Google hold patents related to H.264 so it is possible they could come together and influence MPEG LA (folks who enforce the H.264 licenses) to give the Mozilla Foundation (Firefox) a special license for using H.264. Who knows what will really happen, but it is definitely to Google’s best interest with it’s YouTube property that all web browsers can play back its video content.

I have no idea why Firefox does not support mp3. Mp3 and Ogg video are identical as far as having patents that are not enforced (no one is asking for royalties for using these formats). As far as audio is concerned, I think it is hypocritical of Firefox not to support mpeg3 but support OGG Theora.

Apple iPod/iPhone/iPad/TV

All of Apple’s products/hardware support both MP3 audio and H.264 video formats.

Android/Blackberry/Palm WebOS

The other remaining popular smart phone platforms support both mp3 audio and H.264 video formats.

Other Internet Connected TV Hardware

Other Internet Connected TV hardware (also referred to as OTT TV/Over The Top TV, Set-Top boxes, and IPTV) such as the Roku add the icing to the cake as far as picking audio/video standards are concerned. All of the Internet Connected hardware devices that are planned or that are already available support MP3 audio (.mp3) and H.264 video (.mp4/.m4v).

Conclusion

I am sure this post will upset some folks (Flash developers, Linux/open source enthusiasts, etc…) and I apologize. I love Linux and open source, but I’m sorry to say OGG is not going to become the standard for media. As for Flash, there is still a lot of cool stuff you can do with Flash including live streams, but Flash as far as downloadable media (new media/podcasting) is concerned, Flash is dead.

Update on March 25, 2011:

It appears my prediction may have been wrong about H.264 being sponsored by another vendor for inclusion into Firefox. Over the past year, Google has acquired On2 Technologies (OGG Video) and has launched a new project called WebM which is completely royalty free. This is a game changer both for the WebM video format, but also for OGG Vorbis audio. It also means that Flash is not dead in the short term for downloadable media and can be used to fill in the gap for when a specific audio/video format is not supported in a given browser.

WebM the Game Changer

WebM is significant for a number of reasons. First, it’s important to note one of On2’s past clients, Adobe. One of On2’s older video codex is used for Flash video (.flv). With the launch of WebM video format (.webm), Adobe has promised to include WebM support in future versions of Flash, and seeing it’s past relationship with On2, I don’t see how there would be a problem. In addition, Opera, Firefox and Google Chrome web browsers also support WebM playback. Ogg Theora is essentially replaced by WebM, though the OGG Vorbis audio format that is packaged with OGG Theora and WebM may be the other winner in this HTML5 media tug-of-war.

Also important to note that anything Google related will include WebM support, this means future versions of Android, YouTube and the new Google TV video platform.

Google removes H.264 from Chrome, adds WebM and Launches Google TV

Since the Google acquisition of On2, Google has decided to no longer include H.264 (.mp4) support with the Google Chrome browser, opting instead to include WebM as the supported HTML5 video format. Four significant changes have occurred, which warrant noting:

• Chrome browser can no longer play H.264 video
• Chrome browser can no longer play AAC (.m4a) audio
• Chrome browser can now play WebM video
• Chrome browser can now play OGG Virbis Audio

Along with Firefox and Opera, this now means that 3 of the 5 major web browsers require WebM for video and OGG Vorbis/Mpeg3 for audio. Also important to note Firefox 4 still does not support Mpeg3 (.mp3) audio, which I think is a major letdown.

With this new WebM format, we can assume that the older OGG Theora video format is no longer a player in the HTML5 video wars. OGG audio on the other hand, is another story.

What will be significant is if future versions of Google TV (also packaged in Sony high end TV’s and Blu-ray players) will be WebM exclusive. If this happens, along with adoption on Android based phones may have enough impact that WebM could quickly become an important video format.

M4a Audio growth stalled, OGG Vorbis Audio growth continues

With the HTML5 Video Wars between WebM and H.264, it means that the AAC (.m4a) Audio format growth is now stalled. Looking at AAC last year, I would have thought by now almost every device and hardware out there would support the format. Important to note video hardware vendors include AAC support mainly because it is required for  H.264. As devices come to market that do not have H.264, it is only natural for those devices will also not support AAC (.m4a). AAC almost had the capability to play almost everywhere, but now it seems the Mpeg3 (.mp3) format will continue to still have wider distribution.

Flash will continue to Bridge the Gap

Many of the TV devices like Boxee rely upon Flash for audio and video playback. These devices may be the winner as things play out since Flash can play mpeg3 (.mp3), H.264 (.mp4), AAC (.m4a via the video player) and WebM (.webm). I also suspect that once WebM support is added, it would only be natural for Flash to then also be capable of playing OGG Vorbis (.ogg/.oga) via the WebM player.

What I recommend as of March, 2011

For Video, I recommend creating H.264 (.mp4) and WebM versions of your video. This way you are able to harness HTML5 video on all five web browsers as well as support nearly every video playing device whether it includes one format or the other.

For Audio, I recommend Mpeg3 (.mp3), it still plays on devices and in applications. Though AAC (.m4a) is a close second, if your not using any of the Audiobook features found in m4a (which by the way only work on Apple hardware and software), there’s no real significant advantage to using m4a over mp3.