Author Archives: Multimedia Mike

HTMLOL5 Video

Last week brought us a lot of news in the web browser space: Mozilla released Firefox 3.6 (nice fullscreen video, BTW, especially on Linux); YouTube and Vimeo grabbed headlines by announcing HTML5 video support for their video sites.

I resolved a few months ago to not bother reading so many tech news sites since they consist of 99% misinformed drivel, and I’m a happier person for that decision. But when there’s big news that can be seen as tangentially related to what I do at my day job, it gets hard to resist.

From everything I read, there was surprisingly little Flash hatred in the wake of these announcements. Really, the situation just erupted into an all-out war between the devotees of Firefox (and to a lesser extent, Opera) and supporters of Google (and to a lesser extent Apple and their Safari browser). It gets boring and repetitive in a hurry when you start reading these discussions since they all go something like this:


HTML5 Video Tag Arguments

As you can see from the infographic, at least both sides can agree on something. I would also like to state my emphatic support for Mozilla’s principled, hardline stance against the MPEG stack for HTML5 video. Please don’t budge on your position. Stand firm on the moral high ground.

That graphic is just the beginning; there are so many problems with HTML5 video that it’s hard to know where to even begin. That’s why I need to remember to just laugh gently at its mention and move along. I only get a headache trying to understand how HTML5 video could ever have the slightest chance of mattering in the grand scheme of things.

However, a pleasant side effect of this attention is that more and more people are actually being exposed to the video tag. One nagging detail people invariably notice is that the video tag performs exceptionally poorly, likely because browsers have to deal with the exact same limitations that the Flash Player does, namely, converting decoded YUV data to RGB so that it can be plopped on a browser page. And if you try to claim that you can just download the media and use a standalone player, you continue to miss the entire point of web video.

Another aspect I have to appreciate about the debate surrounding HTML5 video is the way that it brings out the positive spirit in people. Online discussions are normally overwhelmingly negative. But advocates of the HTML5/Xiph approach truly believe this could all work out: If Apple decides to adopt the Xiph stack, and if some benevolent hardware company would churn out custom ASICs for decoding Xiph codecs, and if those ASICs were adopted in next quarter’s array of mobile computing devices and netbooks, and if Google transcodes their zillobytes of YouTube videos to the Xiph stack, and if Google throws the switch and forces the 60% of IE-using stragglers to either change browsers or go without YouTube, and if Google thereby forgoes many opportunities to monetize their videos, then absolutely! HTML5 video could totally unseat Flash video.

Okay, that’s it for me. I’m going to go back to ignoring the insular, elitist tech world at large except for the few domains in which I have some influence.

See Also:

Systematic Benchmarking Adjunct to FATE

Pursuant to my rant on the futility of comparing, performance-wise, the output of various compilers, I wholly acknowledge the utility of systematically benchmarking FFmpeg. FATE is not an appropriate mechanism for doing so, at least not in its normal mode of operation. The “normal mode” would have each of every configuration (60 or so) running certain extended test specs during every cycle. Quite a waste.

Hypothesis: By tracking the performance of a single x86_64 configuration, we should be able to catch performance regressions in FFmpeg.

Proposed methodology: Create a new script that watches for SVN commits. For each and every commit (no skipping), check out the code, build it, and run a series of longer tests. Log the results and move on to the next revision.

What compiler to use? I’m thinking about using gcc 4.2.4 for this. In my (now abandoned) controlled benchmarks, it was the worst performer by a notable margin. I’m thinking that the low performance might help to accentuate performance regressions. Is this a plausible theory? 2 years of testing via FATE haven’t revealed any other major problems with this version.

What kind of samples to test? Thankfully, Big Buck Bunny is available in 4 common formats:

  • MP4/MPEG-4 part 2 video/AC3 audio
  • MP4/H.264 video/AAC audio
  • Ogg/Theora video/Vorbis audio
  • AVI/MS MPEG-4 video/MP3 audio

I have the 1080p versions of all those files, though I’m not sure if it’s necessary to decode all 10 minutes of each. It depends on what kind of hardware I select to run this on.

Further, I may wish to rip an entire audio CD as a single track, encode it with MP3, Vorbis, AAC, WMA, FLAC, and ALAC, and decode each of those.

What other common formats would be useful to track? Note that I only wish to benchmark decoding. My reasoning for this is that decoding should, on the whole, only ever get faster, never slower. Encoding might justifiably get slower as algorithmic trade-offs are made.

I’m torn on the matter of whether to validate the decoding output during the benchmarking test. The case against validation says that computing framecrc’s is going to impact the overall benchmarking process; further, validation is redundant since that’s FATE’s main job. The case for validation says that since this will always be run on the same configuration, there is no need to worry about off-by-1 rounding issues; further, if a validation fails, that data point can be scrapped (which will also happen if a build fails) and will not count towards the overall trend. An errant build could throw off the performance data. Back on the ‘against’ side, that’s exactly what statistical methods like weighted moving averages are supposed to help smooth out.

I’m hoping that graphing this idea for all to see will be made trivial thanks do Google’s Visualization API.

The script would run continuously, waiting for new SVN commits. When it’s not busy with new code, it would work backwards through FFmpeg’s history to backfill performance data.

So, does this whole idea hold water?

If I really want to run this on every single commit, I’m going to have to do a little analysis to determine a reasonable average number of FFmpeg SVN commits per day over the past year and perhaps what the rate of change is (I’m almost certain the rate of commits has been increasing). If anyone would like to take on that task, that would be a useful exercise (‘svn log’, some text manipulation tools, and a spreadsheet should do the trick; you could even put it in a Google Spreadsheet and post a comment with a link to the published document).

Benchmark Bitch No More

You know what? I’m tired of being the benchmark bitch.

I have come to realize that benchmarks are exclusively the domain of people who have a vested interest in the results. I’m not financially or emotionally invested in the results of these compiler output performance comparisons; it was purely a matter of academic curiosity. As such, it’s hard to keep the proper motivation. No matter how carefully I set up a test and regardless of how many variables I control, the feedback is always the same: “Try tweaking this parameter! I’m sure that will make my favorite compiler clean up over all the other options!” “Your statistical methods are all wrong!” “Your graphs are fraudulently misleading.” Okay, I wholeheartedly agree with that last one, but blame OpenOffice for creating completely inept graphs by default.

Look, people, compilers can’t work magic. Deal with it. “–tweak-optim-parm=62” isn’t going to net the 10% performance increase needed to make the ethically pure, free software compiler leapfrop over the evil, proprietary compiler (and don’t talk to me about profile-guided optimization; I’m pretty sure that’s an elaborate hoax and even if it’s not, the proprietary compilers also advertise the same feature). Don’t put your faith in the compiler to make your code suck less (don’t I know). Investigate some actual computer science instead. It’s especially foolhardy to count on compiler optimizations in open source software. Not necessarily because of gcc’s quality (as you know, I think gcc does remarkably well considering its charter), but because there are so many versions of compilers that are expected to compile Linux and open source software in general. The more pressing problem (addressed by FATE) is making sure that a key piece of free software continues to compile and function correctly on a spectacular diversity of build and computing environments.

If anyone else has a particular interest in controlled FFmpeg benchmarks, you may wish to start with my automated Python script in this blog post. It’s the only thing that kept me sane when running these benchmarks up to this point.

I should clarify that I am still interested in reorganizing FATE so that it will help us to systematically identify performance regressions in critical parts of the code. The performance comparison I care most about is whether today’s FFmpeg SVN copy is slower than yesterday’s SVN copy.

See Also:

Update: I have to state that I’m not especially hurt by any criticism of my methods (though the post may have made it seem that way). Mostly, I wanted to force myself to quit wasting time on these progressively more elaborate and time-consuming benchmarks when they’re really not terribly useful in the grand scheme of things. I found myself brainstorming some rather involved profiling projects and I had to smack myself. I have far more practical things I really should be using my free time for.

Compiler Smackdown 2010-1, 64-bit

It’s time to do a new compiler smackdown for a few reasons:

  1. It has been quite awhile since the last one.
  2. I received a request to know how icc 11.1 measured up.
  3. I wanted an excuse to post a picture of the GCC cheerleaders.


GCC Cheerleaders (from "Community" TV show)

For this round, I tested x86_64 on my Core 2 Duo 2.0 GHz. I compiled FFmpeg with 6 versions of gcc (including gcc 4.5, svn 156187), 3 versions of icc, and the latest (svn 94292) of LLVM. Then I used the resulting FFmpeg binaries to decode both a Theora/Vorbis video and an H.264/AAC video.

Ogg/Theora/Vorbis, 1920×1080 video, 48000 Hz stereo audio, nearly 10 minutes:


compsmack-2010-1-64bit-theora

MP4/H.264/AAC: 1280×720 video, 48000 Hz stereo audio, 4.5 minutes:


compsmack-2010-1-64bit-h264

Wow! Look at LLVM go. I take back all, or at least some, of the smack I’ve typed about it in previous posts. Out of the free compiler solutions, LLVM makes my Theora code suck the least.

Other relevant data about this round:

  • FFmpeg SVN 21390 used for this test
  • Flags: ‘–disable-debug –disable-amd3dnow –disable-amd3dnowext –disable-mmx –disable-mmx2 –disable-sse –disable-ssse3 –disable-yasm’ used for all configurations; also used ‘–disable-asm’ which might make a lot of those obsolete now.
  • gcc 4.3-4.5 used “-march=core2 -mtune=core2”; icc versions used “–cpu=core2 –parallel”

See Also: