{"id":2127,"date":"2010-01-25T20:44:33","date_gmt":"2010-01-26T04:44:33","guid":{"rendered":"http:\/\/multimedia.cx\/eggs\/?p=2127"},"modified":"2010-06-21T10:35:05","modified_gmt":"2010-06-21T17:35:05","slug":"benchmark-bitch-no-more","status":"publish","type":"post","link":"https:\/\/multimedia.cx\/eggs\/benchmark-bitch-no-more\/","title":{"rendered":"Benchmark Bitch No More"},"content":{"rendered":"

You know what? I’m tired of being the benchmark bitch<\/a>.<\/p>\n

I have come to realize that benchmarks are exclusively the domain of people who have a vested interest in the results. I’m not financially or emotionally invested in the results of these compiler output performance comparisons; it was purely a matter of academic curiosity. As such, it’s hard to keep the proper motivation. No matter how carefully I set up a test and regardless of how many variables I control, the feedback is always the same: “Try tweaking this parameter! I’m sure that will make my favorite compiler clean up over all the other options!” “Your statistical methods are all wrong!” “Your graphs are fraudulently misleading.” Okay, I wholeheartedly agree with that last one, but blame OpenOffice for creating completely inept graphs by default.<\/p>\n

Look, people, compilers can’t work magic. Deal with it. “–tweak-optim-parm=62” isn’t going to net the 10% performance increase needed to make the ethically pure, free software compiler leapfrop over the evil, proprietary compiler (and don’t talk to me about profile-guided optimization; I’m pretty sure that’s an elaborate hoax and even if it’s not, the proprietary compilers also advertise the same feature). Don’t put your faith in the compiler to make your code suck less (don’t I know<\/a>). Investigate some actual computer science instead. It’s especially foolhardy to count on compiler optimizations in open source software. Not necessarily because of gcc’s quality (as you know, I think gcc does remarkably well considering its charter<\/a>), but because there are so many versions of compilers that are expected to compile Linux and open source software in general. The more pressing problem (addressed by FATE<\/a>) is making sure that a key piece of free software continues to compile and function correctly on a spectacular diversity of build and computing environments.<\/p>\n

If anyone else has a particular interest in controlled FFmpeg benchmarks, you may wish to start with my automated Python script in this blog post<\/a>. It’s the only thing that kept me sane when running these benchmarks up to this point.<\/p>\n

I should clarify that I am still interested in reorganizing FATE so that it will help us to systematically identify performance regressions in critical parts of the code. The performance comparison I care most about is whether today’s FFmpeg SVN copy is slower than yesterday’s SVN copy.<\/p>\n

See Also:<\/strong><\/p>\n