In case you haven’t heard yet, Microsoft released a new preview release of Internet Explorer 9 with all kinds of great goodies we have been waiting for, including HTML5 video support. I did notice that this new preview didn’t score any bonus points on the HTML5 test for its video and audio support. This was pretty strange, because it should have scored bonus points for the H.264 codec. The article below is the result of a little investigation about why IE9 doesn’t pass the H.264 codec test and the problems I found in other browsers.
Codecs, containers and mime types
First a little background on audio and video. Both the audio and video are encoded by a codec. A codec turns a large number of images into a compressed video stream or a large number of audio samples into a compressed audio stream and vice versa. There are many codecs to choose from with different qualities, but some are more popular than others. That is just half of the story. When you want to merge the compressed audio and video streams in a single file, you also need a container format. Again, there are many container formats to choose from, but not all container formats support all codecs. Currently there are 2 popular combinations for HTML5 video:
- H.264 video and AAC audio in a MP4 container
- VP8 video and Vorbis audio in a WebM container
The following two combinations are also used, but are less popular:
- Theora video and Vorbis audio in an Ogg container
- MPEG-4 video and AAC audio in a MP4 container
Each combination of codecs and containers can be described by an identifier. That identifier contains the MIME type of the container and the codecs parameter, a combination of unique strings for each of the codecs that have been used.
For example, the MIME type for the Ogg container is video/ogg. The unique string for the Theora codec is theora and the string for the Vorbis codec is vorbis. The identifier would look like this: video/ogg; codecs="vorbis, theora"
H.264, MPEG-4 and AAC are a bit more complicated, because these codecs use profiles which determine which features are used by a codec. This only makes the unique string for each codec a bit more lengthy, but the principle is the same. For example Baseline Profile H.264 video has a unique string of avc1.42E01E and High Profile H.264 video has a unique string of avc1.64001E.
The MIME type for the MP4 container is video/mp4. The unique string for Low-Complexity AAC audio is mp4a.40.2. Combined with the unique string for Baseline Profile H.264 video you would get the following identifier: video/mp4; codecs="avc1.42E01E, mp4a.40.2".
Detecting which codecs are supported
The HTML5 specification does not specify which combination of codecs and containers should be supported in a browser, but it does provide a way to detect which actually are. This has become quite important because not every browser supports the same codecs. The reasons for this are complex and beyond the scope of this article. What is important, is that by calling the canPlayType() function with either the MIME type of the container or the complete identifier for a specific combination you can ask the browser if it supports that MIME type or combination. The browser then tells you one of three things:
- an empty string: it does not support it,
- the string
maybe: it does not know if it is supported, but it is not sure it isn’t supported either, - the string
probably: it supports the combination of container and codec.
Especially if you only specify the MIME type of the container the browser may not be entirely sure it can play it. The reason for this is that some containers support many codecs and the browser may only support one of them. So if you specify only the MIME type you should expect only an maybe answer at best.
The string probably is the best answer you can get. The reason why it isn’t a simple ‘yes’ is that there are other factors that determine whether or not a file can be played by a browser. This answer is as definitive an answer as you will get. The only way to be even more sure is to actually try to load the file and see if it fails or succeeds.
The rest of this article is a detailed explanation of how the different browsers respond to various canPlayType() requests. You’ll notice there are still quite some problems and Internet Explorer isn’t the only one affected. It’s just the first to be broken in such a way that actually prevents proper detection of codecs. I’ve only looked at the results for video codecs, so each codec parameter given to the canPlayType() function only contained the unique identifier of the video codec. I used a special testcase for determining the results: the HTML5test.com video testcase, which you try for yourself.
The results
Internet Explorer 9
Let’s start with the browser that started this little investigation: Internet Explorer 9 preview release 3. It is the first version of IE that supports the video element: it currently has only support for H.264 in an MP4 container, but in the future they may also support WebM on systems where this codec has been manually installed.
Without a codec parameter, IE will report back maybe for MIME types of containers it supports. If it doesn’t recognize the MIME type, it simply returns an empty string. This is absolutely correct.
However IE does not seem to do anything with the codec parameter. If I specify the codecs parameter for the H.264 codec it still returns maybe while it should have returned probably. Also if I specify the codecs parameter for a codec that it does not support, such as MPEG-4, then it still returns maybe, while it should return an empty string.
This makes it impossible to reliably detect which codecs it supports, without actually loading a video file and waiting for it fail or not. This is a real problem and should be fixed as soon as possible.
Safari
Safari uses the QuickTime media framework for playing back video instead of shipping any build-in decoders. This means it will support all formats supported by QuickTime. On Macs this isn’t a problem, but when Safari 5 was released, I did receive a couple of reports from users who installed Safari on a Windows system without QuickTime. The result was that Safari failed the video and audio tests of html5test.com.
QuickTime supports the H.264 and MPEG-4 codecs by default. It also supports the MP4 file format.
If a codecs parameter is absent, Safari will report back maybe for all MIME types that it recognizes. That also includes all MIME types for containers that are not normally used for HTML5 video. The good news is that QuickTime lets you manually install additional codecs such as Ogg Theora which are also properly detected.
If a codecs parameter is present, Safari will report back probably for all MIME types that it recognizes, regardless of the contents of the codec parameter. Even if the codec is not supported or simply gibberish, Safari will report back probably. This is a huge bug and should be fixed.
Chrome
Chrome performs slightly better than Safari, even though both are based on the same rendering engine. Chrome actually has decoders build in for H.264 and Theora and natively supports both the Ogg and MP4 container. Future versions will also support WebM.
Without a codecs parameter, Chrome will report maybe for all MIME types of containers that it recognizes. This is correct behavior.
However if a codecs parameter is present, and Chrome recognizes both the MIME type and the codec parameter, it will return probably. If recognizes the MIME type, but not the codecs parameter, it will report back maybe which seems reasonable, although it would probably be better to simply return an empty string in this case. Because Chrome ships with build in decoders, it knows exactly which codecs it supports and which won’t work.
There is one more thing that can be improved though. Chrome also reports probably for combinations of containers and codecs that it does not support, for example Theora in a MP4 container does not make sense, but Chrome does report probably. This is just a minor problem that does not lead to any real world problems, but it should be fixed nevertheless.
Firefox
For Firefox I used the latest nightly which includes support for both Ogg Theora and WebM. The build I tested has the best support for canPlayType() so far and I could only find a simple, small problem.
Without a codecs parameter, Firefox will report back maybe for the Ogg MIME type and probably for the WebM MIME type. That last one is something that the specification specifically discourages. If the codecs parameter is not specified, it should report back maybe at best.
However when I did specify the codecs parameter, Firefox passed all tests. It reports back probably if it recognizes the MIME type and the codecs parameter, but only when the combination actually makes sense. Also if it doesn’t recognize the codecs parameter, it will simply return an empty string. Great!
Opera
Finally a browser where I couldn’t find anything wrong. I tested the beta version of Opera 10.6 on Mac which supports both Ogg Theora and WebM. I haven’t been able to test Opera on Linux which uses the system GStreamer framework which might also support other codecs.
Without a codec parameter, Opera will report back maybe for all MIME types it recognizes. This is correct.
When a codec parameter is specified, Opera will report back probably for all MIME types and codecs it recognizes, but only when the combination makes sense. So a VP8 stream in an Ogg container simply returns an empty string. Perfect!
Safari for the iPhone
The iPhone should support both H.264 and MPEG-4 in a MP4 container. Safari on the iPhone seems to suffer from similar problems as Safari on the desktop, but in a worse way.
Unlike its desktop equivalent it will report back probably for containers that it supports when only a MIME type is specified. This is something that is explicitly discouraged by the specification. When no codecs parameter is present, it should report maybe at best.
When a codecs parameter is present, Safari will strangely report back maybe for all MIME types it recognized, regardless of the contents of the codecs parameter. The proper response would be to report back probably for supported codecs and an empty string for codecs it doesn’t recognize. This needs to be fixed as soon as possible.
Palm WebOS 1.4
WebOS should support both MPEG-4 and H.264 in a MP4 container, but I haven’t found a way to actually play the video files. canPlayType() does seem to work though and what it reports seems actually quite good.
If only a MIME type is specified, it will return maybe if it supports the container. If it doesn’t recognize the container, it will return an empty string.
If the codecs parameter is present and it supports the codec, it returns probably which is good. However if the parameter is not recognized, it still returns maybe which should probably be and empty string in this case. PalmOS knows exactly which codecs it supports, so it should also report back with certainty when it encounters a codec which it doesn’t support.
Android 2.2
The Android browser seems to be the worst of all. While it does support the video element, it didn’t want to play any of my test files and canPlayType() always returned an empty string regardless of mime type or codecs parameter. Perhaps it is because I used the Android emulator and not a real phone.