The IETF has recently put out a request for an audio codec. This may strike some of you as remarkable that anyone would need another audio codec since, at the time of this writing, we have cataloged 137 audio codecs via the MultimediaWiki. You have to give the request some attention, though– it acknowledges that there are already lots and lots of audio codecs in existence and explains why each category is unsuitable to the goals of the request. I’m not going to be the one to audit every one of those 137 codecs and identify why each is unsuitable for the outlined goals.

I am a bit concerned about some of their stated goals, such as the very first one: “Designing for use in interactive applications (examples include, but are not limited to, point-to-point voice calls, multi-party voice conferencing, telepresence, teleoperation, in-game voice chat, and live music performance).” Generally, one of those examples is not like the others (unless, perhaps, “live music performance” refers to a cappella singing. Then again, the request later states that optimizing for very low bitrates (2.4 kbps and lower) is out of scope.

I accidentally used the main ‘ffmpeg’ binary as an input to itself. Its best guess is that it’s an MP3 container with MPEG-1, layer 1 audio data:

[NULL @ 0x1002600]Format detected only with low score of 25, misdetection possible!
[mp1 @ 0x1003600]Header missing
    Last message repeated 35 times
[mp3 @ 0x1002600]max_analyze_duration reached
[mp3 @ 0x1002600]Estimating duration from bitrate, this may be inaccurate
Input #0, mp3, from 'ffmpeg_g':
  Duration: 00:03:20.29, start: 0.000000, bitrate: 256 kb/s
    Stream #0.0: Audio: mp1, 32000 Hz, 2 channels, s16, 256 kb/s
At least one output file must be specified

What an Easter egg it would be if the compiled binary could actually decode to something — anything — valid.

You know what? I’m tired of being the benchmark bitch.

I have come to realize that benchmarks are exclusively the domain of people who have a vested interest in the results. I’m not financially or emotionally invested in the results of these compiler output performance comparisons; it was purely a matter of academic curiosity. As such, it’s hard to keep the proper motivation. No matter how carefully I set up a test and regardless of how many variables I control, the feedback is always the same: “Try tweaking this parameter! I’m sure that will make my favorite compiler clean up over all the other options!” “Your statistical methods are all wrong!” “Your graphs are fraudulently misleading.” Okay, I wholeheartedly agree with that last one, but blame OpenOffice for creating completely inept graphs by default.

Look, people, compilers can’t work magic. Deal with it. “–tweak-optim-parm=62” isn’t going to net the 10% performance increase needed to make the ethically pure, free software compiler leapfrop over the evil, proprietary compiler (and don’t talk to me about profile-guided optimization; I’m pretty sure that’s an elaborate hoax and even if it’s not, the proprietary compilers also advertise the same feature). Don’t put your faith in the compiler to make your code suck less (don’t I know). Investigate some actual computer science instead. It’s especially foolhardy to count on compiler optimizations in open source software. Not necessarily because of gcc’s quality (as you know, I think gcc does remarkably well considering its charter), but because there are so many versions of compilers that are expected to compile Linux and open source software in general. The more pressing problem (addressed by FATE) is making sure that a key piece of free software continues to compile and function correctly on a spectacular diversity of build and computing environments.

If anyone else has a particular interest in controlled FFmpeg benchmarks, you may wish to start with my automated Python script in this blog post. It’s the only thing that kept me sane when running these benchmarks up to this point.

I should clarify that I am still interested in reorganizing FATE so that it will help us to systematically identify performance regressions in critical parts of the code. The performance comparison I care most about is whether today’s FFmpeg SVN copy is slower than yesterday’s SVN copy.

See Also:

• The final entry in the compiler smackdown series

Update: I have to state that I’m not especially hurt by any criticism of my methods (though the post may have made it seem that way). Mostly, I wanted to force myself to quit wasting time on these progressively more elaborate and time-consuming benchmarks when they’re really not terribly useful in the grand scheme of things. I found myself brainstorming some rather involved profiling projects and I had to smack myself. I have far more practical things I really should be using my free time for.

It’s time to do a new compiler smackdown for a few reasons:

1. It has been quite awhile since the last one.
2. I received a request to know how icc 11.1 measured up.
3. I wanted an excuse to post a picture of the GCC cheerleaders.

For this round, I tested x86_64 on my Core 2 Duo 2.0 GHz. I compiled FFmpeg with 6 versions of gcc (including gcc 4.5, svn 156187), 3 versions of icc, and the latest (svn 94292) of LLVM. Then I used the resulting FFmpeg binaries to decode both a Theora/Vorbis video and an H.264/AAC video.

Ogg/Theora/Vorbis, 1920×1080 video, 48000 Hz stereo audio, nearly 10 minutes:

MP4/H.264/AAC: 1280×720 video, 48000 Hz stereo audio, 4.5 minutes:

Wow! Look at LLVM go. I take back all, or at least some, of the smack I’ve typed about it in previous posts. Out of the free compiler solutions, LLVM makes my Theora code suck the least.

Other relevant data about this round:

• FFmpeg SVN 21390 used for this test
• Flags: ‘–disable-debug –disable-amd3dnow –disable-amd3dnowext –disable-mmx –disable-mmx2 –disable-sse –disable-ssse3 –disable-yasm’ used for all configurations; also used ‘–disable-asm’ which might make a lot of those obsolete now.
• gcc 4.3-4.5 used “-march=core2 -mtune=core2”; icc versions used “–cpu=core2 –parallel”

See Also:

• Final wrap-up for this series
• Previous smackdown in the series