Because I have a short memory, I wanted to write down some of the knowledge and wisdom I’ve collected in the past few months as I have been working on optimizing FFmpeg’s VP3/Theora decoder.

Profiling Methods
These are some of the general tools:
Read the rest of this entry »

I have a habit of prepending ‘time’ to all of my ‘make’ commands in order to keep a rough estimate of how long build jobs take.

Adhering to this custom, I performed a ‘make’ command on a project that didn’t actually require any rebuilding. So how does the following happen?

$ time make -j5

[...]

real    1770m35.893s
user    0m12.408s
sys     0m11.692s

Answer: The machine (virtual machine, actually) had just been started, had a grossly out-of-sync clock, and must have synced to the time server during that narrow window that the build was occurring:

make[2]: Warning: File `...' has modification time 1.8e+04 s in the future
make[2]: warning:  Clock skew detected.  Your build may be incomplete.

I dug up this old security alert. It’s very dear to me in that I’m directly responsible for the security problem outlined. Whenever I feel like my work doesn’t matter, I just have to remind myself that I have written code that has become widespread enough that it warrants security notices. Many programmers likely go their whole career without making that kind of impact. (That kind of positive spin might be similar to not knowing or caring about the difference between positive and negative attention.)

For the curious, I wrote an AIFF demuxer (among many others) for the xine project. For some reason, I allocated a static buffer of 100 bytes on the stack and proceeded to read a number of bytes from user input, a number that was also determined by the same user input. Big no-no, and I really don’t know what I was thinking; hardcoded, arbitrary constants (char buffer[100]) aren’t usually my style. After that was found, I audited the rest of my demuxers for similar mistakes and found none. It may seem like this would only be a problem if a user directly loaded a malicious file into xine. However, since AIFF has a MIME type, and because there was a Mozilla plugin version of xine, it would have been possible to send a malicious AIFF through a web page.

The reason I was reflecting on this was due to a major security problem I found in FATE recently as I was investigating another problem. It has to do with the data logging script that receives FFmpeg build and test information from FATE clients. I’ll let my commit message to my private git repository tell the tale:

    Get rid of mind-boggling security hazard that actually prints out the
    user's actual hash key when passed an invalid hash. This was obviously
    added for debugging purposes and was only triggered if a user had access
    to insert data for a particular configuration.

If an attacker knew a valid username, the system would cheerfully reveal the corresponding hash key if the HMAC failed. Using this vector, an attacker could have polluted the FATE database with loads of bad data. Not a huge deal in the grand scheme of things. But given that this is the only attack that the system is trying to guard against, a total failure in context.

Honestly, sometimes I can’t believe people let me anywhere near a programming environment.

One last — and fascinating — note about that AIFF exploit: It was the result of an infamous university course (perhaps this one?) given by D. J. Bernstein in which students were required to find 10 security holes in open source software programs during the term. Reportedly, all of the students failed the class since none actually found 10 holes. I don’t know if the class was ever held again.

No matter how much I think I know about about character encoding or trying to work around issues arising from the same, I’ll always get bitten.

For a long time, one particular FATE configuration has shown 87/127 tests succeeding, even though the total number of test specs has crept up over to 300. I investigated this errant configuration on the client side and concluded that it was, in fact, executing all of the tests and sending all the results over the server in one neat package. Apparently, the problem was on the server side. Since it was an older Intel C compiler configuration, I didn’t care about investigating much further.

At one point, some bad bit of code was checked into FFmpeg and all of the results started showing xy/127 tests succeeded. This made the issue a bit more pressing. Mans discovered that the problem had to do with the svq3 test spec failing. The bad code affecting the SVQ3 test was quickly fixed so I didn’t worry about it again until yesterday when, once again, FATE’s various configs were only reporting that 127 tests had been run.

Here’s what was happening: FATE stores the stderr output of a test only if the test spec fails. This is a key data point since everything is fine when the test is successful and FATE tosses the output. The sample used for the SVQ3 test outputs the following metadata (among other data) in the stderr (seen, for example, in this test result):

    copyright       : ? Vertical Online 2001
    copyright-eng   : ? Vertical Online 2001

Those mystery characters map to the byte 0xA9 which is the c-in-a-circle copyright symbol according to UTF tables I can find (at least, U+00A9 is). That byte is making the system choke somewhere along the line, which annoys me greatly. When the client-side Python script executes the test and stuffs the stdout and stderr into the SQLite database, the relevant field is supposed to be a blob– a binary large object. The receiving PHP script on the server is also supposed to honor that blob schema.

Mans’ solution is to specifically encode the stdout/stderr blobs as UTF8 strings in the client-side Python script. That fixes this problem. But I’m confused as to why this is necessary in the first place. Was the PHP script doing its best to interpret the data inside the blob and falling over? Or was the SQLite engine on the server confused by the 0xA9 character in the blob?

Also, I suddenly find myself wondering how the A9 search company got its name.

Update: Thanks for MichaelK. for pointing out the problem. While I was properly converting (since that’s necessary) stdout/stderr from build records to binary type, I never did the same for test result stdout/stderr. I had to do it for the build record output since that was compressed before going into the database. Since the test result data is “just strings”, or so I thought, no reason to do so.

In the middle of 2004 I purchased a Creative Nomad Zen Xtra portable MP3 player. “MP3 player” was not quite a commonplace concept yet but the word “iPod” was beginning to catch on. When describing this new toy to people, I usually described it as “about the same as an iPod but about 1/2 the price” which was absolutely true when I purchased it.

Here is my Nomad compared to a 1st generation Apple iPod Touch, my current MP3 player (and more):

The Nomad Zen Xtra served me well for 3 solid years until I finally got a proper iPod in summer of 2007. I have kept the unit around since then for no particular reason. I decided to disassemble and photograph it before I send the battery and electronics off to their respective recycling destinations.

The Nomad Zen Xtra was highly user-servicable and upgradeable. At the time I put it out of service, the battery could barely run for 5 hours (whereas, 10-12 hour was no problem when it was new). A replacement battery would be easy enough to order from assorted battery shops on the internet.

Have more than 40 GB of songs? Take off the back of the unit, remove the 2.5″ 40 GB IDE HD and replace with a larger one. That never proved to be necessary for me; in fact, I soon realized after I bought it that the lower-end 30 GB model would have been well more than enough.

The 40 GB HD from the unit is still perfectly good. I decided to hook it up to a Linux computer and see if there is anything I could work out about the filesystem. Before I got too far into it, a little Googling led me to a Python utility called zenrecover.py. Works famously:

$ python zenrecover.py /dev/sdc songs /home/melanson/mnt/zen
0% 3.6MB/s "Bizet_Intermezzo_from_Carmen.mp3" (6.8MB)

Just for fun, I dumped all the songs from the unit. I discovered a few things I had long forgotten and had never made the transition into my iPod. Curiously, the very first items that the utility dumped (likely because they occupied the first parts of the filesystem) were a selection of classical tunes as played by the “Beijing Central Phil Orchestra”. These songs came with the unit. It’s notable that the software transferred them off because the packaged software did not allow the user to do so (I’m pretty sure it allowed all music that was downloaded to be transferred off).

Ugh, that packaged software had to be the worst part about the Nomad Zen Xtra. I know lots of users like to chastise iTunes over a range of pet peeves. I think such people have simply never been exposed to anything worse, like this software.

Someone recently updated a link in the MultimediaWiki page for mirrored documents. Naturally, that doesn’t automatically update the mirrored copy @ multimedia.cx (having me poll the page history and manually update the mirrored copy hardly counts as an automated process). I suddenly thought that it would be desirable to have a content management system that allows authorized users to upload and organize documents, particularly PDF documents which comprise many of these mirrored documents. Sort of a… document management system.

“Document Management System.” Sounds enterprise-y. Here’s what I want:

• Free, open source solution in which I do not have to modify a single line of code
• Allows me to create a list of users who have permissions to upload or delete PDF documents
• Allows authorized users to upload or delete PDF documents
• Manages at least a minimum of metadata

The key thing here is to allow authenticated users to upload and manage these mirrored documents. I know many will say, “Drupal/WordPress/MyFaveCMS can be coerced to do just that!” And I don’t dispute any such claims. It’s also true that nearly any program you need to write can be written in straight C, eschewing any higher level languages. I was just hoping for a more turnkey solution that doesn’t require me to learn a lot or do my own coding or customization.

I guess the problem here is that no one sets out to just write such a simple CMS. A CMS might start out simple but eventually grows into the next Drupal. I probably need to come to terms with that fact that there is no prepackaged solution that exactly fits this simple need without at least some tinkering.

The IETF has recently put out a request for an audio codec. This may strike some of you as remarkable that anyone would need another audio codec since, at the time of this writing, we have cataloged 137 audio codecs via the MultimediaWiki. You have to give the request some attention, though– it acknowledges that there are already lots and lots of audio codecs in existence and explains why each category is unsuitable to the goals of the request. I’m not going to be the one to audit every one of those 137 codecs and identify why each is unsuitable for the outlined goals.

I am a bit concerned about some of their stated goals, such as the very first one: “Designing for use in interactive applications (examples include, but are not limited to, point-to-point voice calls, multi-party voice conferencing, telepresence, teleoperation, in-game voice chat, and live music performance).” Generally, one of those examples is not like the others (unless, perhaps, “live music performance” refers to a cappella singing. Then again, the request later states that optimizing for very low bitrates (2.4 kbps and lower) is out of scope.

I accidentally used the main ‘ffmpeg’ binary as an input to itself. Its best guess is that it’s an MP3 container with MPEG-1, layer 1 audio data:

[NULL @ 0x1002600]Format detected only with low score of 25, misdetection possible!
[mp1 @ 0x1003600]Header missing
    Last message repeated 35 times
[mp3 @ 0x1002600]max_analyze_duration reached
[mp3 @ 0x1002600]Estimating duration from bitrate, this may be inaccurate
Input #0, mp3, from 'ffmpeg_g':
  Duration: 00:03:20.29, start: 0.000000, bitrate: 256 kb/s
    Stream #0.0: Audio: mp1, 32000 Hz, 2 channels, s16, 256 kb/s
At least one output file must be specified

What an Easter egg it would be if the compiled binary could actually decode to something — anything — valid.

Last week brought us a lot of news in the web browser space: Mozilla released Firefox 3.6 (nice fullscreen video, BTW, especially on Linux); YouTube and Vimeo grabbed headlines by announcing HTML5 video support for their video sites.

I resolved a few months ago to not bother reading so many tech news sites since they consist of 99% misinformed drivel, and I’m a happier person for that decision. But when there’s big news that can be seen as tangentially related to what I do at my day job, it gets hard to resist.

From everything I read, there was surprisingly little Flash hatred in the wake of these announcements. Really, the situation just erupted into an all-out war between the devotees of Firefox (and to a lesser extent, Opera) and supporters of Google (and to a lesser extent Apple and their Safari browser). It gets boring and repetitive in a hurry when you start reading these discussions since they all go something like this:

As you can see from the infographic, at least both sides can agree on something. I would also like to state my emphatic support for Mozilla’s principled, hardline stance against the MPEG stack for HTML5 video. Please don’t budge on your position. Stand firm on the moral high ground.

That graphic is just the beginning; there are so many problems with HTML5 video that it’s hard to know where to even begin. That’s why I need to remember to just laugh gently at its mention and move along. I only get a headache trying to understand how HTML5 video could ever have the slightest chance of mattering in the grand scheme of things.

However, a pleasant side effect of this attention is that more and more people are actually being exposed to the video tag. One nagging detail people invariably notice is that the video tag performs exceptionally poorly, likely because browsers have to deal with the exact same limitations that the Flash Player does, namely, converting decoded YUV data to RGB so that it can be plopped on a browser page. And if you try to claim that you can just download the media and use a standalone player, you continue to miss the entire point of web video.

Another aspect I have to appreciate about the debate surrounding HTML5 video is the way that it brings out the positive spirit in people. Online discussions are normally overwhelmingly negative. But advocates of the HTML5/Xiph approach truly believe this could all work out: If Apple decides to adopt the Xiph stack, and if some benevolent hardware company would churn out custom ASICs for decoding Xiph codecs, and if those ASICs were adopted in next quarter’s array of mobile computing devices and netbooks, and if Google transcodes their zillobytes of YouTube videos to the Xiph stack, and if Google throws the switch and forces the 60% of IE-using stragglers to either change browsers or go without YouTube, and if Google thereby forgoes many opportunities to monetize their videos, then absolutely! HTML5 video could totally unseat Flash video.

Okay, that’s it for me. I’m going to go back to ignoring the insular, elitist tech world at large except for the few domains in which I have some influence.

See Also:

• The last time I put any effort into evaluating the HTML5 video tag

Pursuant to my rant on the futility of comparing, performance-wise, the output of various compilers, I wholly acknowledge the utility of systematically benchmarking FFmpeg. FATE is not an appropriate mechanism for doing so, at least not in its normal mode of operation. The “normal mode” would have each of every configuration (60 or so) running certain extended test specs during every cycle. Quite a waste.

Hypothesis: By tracking the performance of a single x86_64 configuration, we should be able to catch performance regressions in FFmpeg.

Proposed methodology: Create a new script that watches for SVN commits. For each and every commit (no skipping), check out the code, build it, and run a series of longer tests. Log the results and move on to the next revision.

What compiler to use? I’m thinking about using gcc 4.2.4 for this. In my (now abandoned) controlled benchmarks, it was the worst performer by a notable margin. I’m thinking that the low performance might help to accentuate performance regressions. Is this a plausible theory? 2 years of testing via FATE haven’t revealed any other major problems with this version.

What kind of samples to test? Thankfully, Big Buck Bunny is available in 4 common formats:

• MP4/MPEG-4 part 2 video/AC3 audio
• MP4/H.264 video/AAC audio
• Ogg/Theora video/Vorbis audio
• AVI/MS MPEG-4 video/MP3 audio

I have the 1080p versions of all those files, though I’m not sure if it’s necessary to decode all 10 minutes of each. It depends on what kind of hardware I select to run this on.

Further, I may wish to rip an entire audio CD as a single track, encode it with MP3, Vorbis, AAC, WMA, FLAC, and ALAC, and decode each of those.

What other common formats would be useful to track? Note that I only wish to benchmark decoding. My reasoning for this is that decoding should, on the whole, only ever get faster, never slower. Encoding might justifiably get slower as algorithmic trade-offs are made.

I’m torn on the matter of whether to validate the decoding output during the benchmarking test. The case against validation says that computing framecrc’s is going to impact the overall benchmarking process; further, validation is redundant since that’s FATE’s main job. The case for validation says that since this will always be run on the same configuration, there is no need to worry about off-by-1 rounding issues; further, if a validation fails, that data point can be scrapped (which will also happen if a build fails) and will not count towards the overall trend. An errant build could throw off the performance data. Back on the ‘against’ side, that’s exactly what statistical methods like weighted moving averages are supposed to help smooth out.

I’m hoping that graphing this idea for all to see will be made trivial thanks do Google’s Visualization API.

The script would run continuously, waiting for new SVN commits. When it’s not busy with new code, it would work backwards through FFmpeg’s history to backfill performance data.

So, does this whole idea hold water?

If I really want to run this on every single commit, I’m going to have to do a little analysis to determine a reasonable average number of FFmpeg SVN commits per day over the past year and perhaps what the rate of change is (I’m almost certain the rate of commits has been increasing). If anyone would like to take on that task, that would be a useful exercise (’svn log’, some text manipulation tools, and a spreadsheet should do the trick; you could even put it in a Google Spreadsheet and post a comment with a link to the published document).