Not all builds should be treated equally. Some are more important than others. I propose that the FATE should distinguish between important and less important configurations. My motivation for this is that I want to implement a meter that indicates the health of the overall code base. While it would be ideal for all FATE configurations to be 100% green at all times, I don’t think it’s fair to penalize the entire FFmpeg codebase just because some less prevalent platforms aren’t performing up to spec.

What platforms should be considered first class? I’m thinking latest gcc 4.3 and 4.2 series for Linux on x86_32, x86_64, and PowerPC, at a minimum.

In other FATE news, I have started computing percentages of test coverage. According to my numbers, FATE currently tests 58% of FFmpeg’s total mux/demux/encode/decode features. It’s a start, I suppose.

There have been some problems with FATE audio testing. First off, the qt-ima4-stereo test spec was testing against the wrong file for the past year. Stereo IMA ADPCM decoding could have broken in QuickTime and we might have never been alerted. Sloppy.

More seriously, I found out that many of my existing, bitexact audio tests have not been constructed properly. This is due to the fact that these 2 commands:

ffmpeg -i file.ext file.wav
ffmpeg -i file.ext -f wav - > file.wav

do not yield equivalent sets of bytes inside file.wav. Part of the reason is that, after writing out all the audio samples, the muxer needs to rewind to the header so that it can write the data payload length. When writing data to stdout, the program does not have the option to rewind the output stream. However, I don’t understand the entire discrepancy. Using the file qt-ima4-mono with the above command lines:

1156652 surge.wav
1146924 surge-stdout.wav

The file that is routed through stdout is notably smaller (9728 bytes smaller). I was going to write this off as the stdout file failing to be flushed. However, the behavior is consistent across all machines and platforms.

My proposed solution is to update all of the audio tests to use this raw format target:

ffmpeg -i file.ext -f s16le -

Since the output is equivalent to:

ffmpeg -i file.ext -f s16le file.s16le

1156608 surge.s16le
1156608 surge-stdout.s16le

Moving right along, there is the much bigger task of testing perceptual audio decoders. Working down the FATE Test Coverage list, these perceptual audio codecs will get the naive, one-off wave reference treatment in lieu of a proper conformance suite: ATRAC3, RealAudio Cooker, DCA (DTS), IMC, Nellymoser, Qcelp, QDesign, RealAudio 28.8, Truespeech, Vorbis, and WMA v1.

Then there is the matter of MPEG audio codecs for which we have access to extensive conformance suites. Thanks to Kostya and Benjamin for furnishing pointers to precise information discussing how to verify if your MP1/2/3 or AAC audio decoder is up to snuff. This page at Underbit describes exactly how the spec describes conformance for MPEG 1, layers 1, 2, and 3, and also evaluates the conformance of various implementations. The comparison ostensibly predates FFmpeg. This Mp4-tech mailing list post shows the way regarding AAC conformance.

So I need to automate the MP1/2/3 and AAC test entries. I estimate the automated process will work something like this:

• Decode encoded file
• Run comparison of decoded wave against original wave
  • For MP1/2/3, this seems to entail converting both the FFmpeg output and the original wave output floating point numbers to a normalized range of -1.0..1.0, computing the root mean square of the difference signal, and verifying that the RMS is less than 1 / (32768 * sqrt(12))
  • For AAC, well, I’m still researching the precise criteria
• If the decoded wave is within tolerance, add a new test

The part where I get a bit fuzzy is: what should the test spec be? Should I generate a reference wave and test future decoded waves against it using my one-off wave reference method? Or, should I just go ahead and compute the RMS of the difference signal? I assume that if I use the nifty numpy library for the task, it couldn’t possibly make any measurable difference in the performance of FATE testing vs. using the one-off wave reference method (computing absolute value of the difference signal and checking that no discrete points exceed 1).

One trade-off is that I would need to store the full 24-bit reference waves in order to properly compute RMS, which is 50% more data than I would need with the one-off method. And I’m still not sure how to process the 24-bit data in any event.

Michael suggested on the FFmpeg-devel list that Doxygen documentation ought to be continuously generated so that any errors and warnings during documentation generation can be caught, logged, analyzed, and minimized. However, the consensus was that it’s not especially useful to add this to the master FATE suite of test specs.

Another item that came up in the discussions of a possible release is that one of our tests should be the processing of an entire DVD-length movie to catch any problems (like memory leaks) that only manifest over a long runtime. Obviously, that’s not especially appropriate for a normal FATE test spec.

And another type of test that I envisioned when I was originally brainstorming the system (for a year and a half) is a way to continuously fuzz-test FFmpeg. But, like the previous 2 items, it does not need to be performed on every code commit.

I realized that all of these tasks (and probably more– be creative) can be run on a less frequent basis — say, once per day — and on one machine (like the fastest machine on my farm). It can be set up as an adjunct project to FATE.

Now I need a good FFmpeg command line for converting a ripped DVD image to another format that will maximally stress the program, in a multithreaded manner, no less.

Check out snakebite. Some folks associated with the Python programming language put together a farm of computers that all Python committers have access to so that they can code and test. They put together an impressive network, though the PowerPC architecture is suspiciously unrepresented in any incarnation that I can find. They received some notable corporate sponsorship too, according to the announcement email on python-committers.

Wouldn’t it be neat to set up something like this for FFmpeg folks? Actually, I tend to think our first and foremost concern would be to get a community-accessible PowerPC machine for debugging various woes on that platform. My PowerPC-based Mac Mini is overcommitted as it is. I happen to be personally familiar with at least one large corporation that has an unbelievable pile of PowerPC-based Macs in its basement (along with loads of other computers), waiting to recycled one day. Regrettably, they have no policy for repurposing the computers for non-corporate functions.

And I don’t want to hear anything about how hard it would be to debug problems in a multimedia program on a remote computer halfway around the world while only interacting via terminal. I did a significant amount of debugging and performance profiling of FFmpeg’s VP3 decoder once upon a time under those very circumstances. My trick, when I had to view the results of a decode operation, were to write the video frames to individual JPEG files. Then, I would run webfs to serve those JPEGs via HTTP to the localhost IP address and tunnel it back to my own machine via SSH to view in a local web browser.

Don’t look now, but the idea of making a formal FFmpeg release in the very near future is gaining traction over on the FFmpeg-devel list. There is a determined and growing team of individuals committed to this goal in the near future. To that end, there is even a Wiki page listing the steps that need to occur leading up to a release.

There is some debate about what kind of release schedule we should commit to. Some take the stance that we should follow the model that Wine apparently follows by releasing every 2 weeks. Really, every new weeks? I had never heard of that before. It strikes me as a bit severe. I personally would like to see a little more time between releases, but see them often enough that we get really good at it, and it becomes part of our shared community mindset.

After all, there are a few projects out there that depend on FFmpeg.

I admit, whenever I get to thinking about an FFmpeg release, I get a little nervous about the ensuing discussion. Not the discussion about whether or not a release should happen; that’s a foregone conclusion because it will happen. No, the item that makes me lose sleep is the biggest bikeshed hue discussion of them all — how should we version the upcoming release of FFmpeg? I don’t think I even want to get involved in that one.

Do you remember the DEC Alpha CPU? Måns Rullgård does. He still has his unit in service and running Linux. So why not run FATE? Just like FFmpeg supports impossibly obscure multimedia formats, why shouldn’t it also support CPU types of approximately the same prevalence? In both cases, as long as the support overhead is not too arduous, it’s not a problem.

What I’m saying here, in case I was unclear, is that Måns has an Alpha CPU contributing build/test results to the central FATE database.

In other FATE news, today was a very busy day. I carefully evaluated and entered 18 new test specs:

With this, I have eliminated much of the low-hanging fruit on the FATE Test Coverage wiki page. My excuses to avoid testing the more complicated (read: bit inexact) formats are dwindling.

Meanwhile, the front page of FATE is reaching critical mass. I really need to come up with a new presentation because the front page is getting harder and harder to digest.

Thanks to Reimar, Linux on 64-bit PowerPC is now a supported configuration in FATE.

Let’s keep the architectures coming.

I feel a little more confident about the lossless audio decoders in FFmpeg — some of them, at least — so I activated the FATE tests that I had staged for them:

• lossless-meridianaudio
• lossless-monkeysaudio
• lossless-shortenaudio
• lossless-wavpack

Further, I have added some tests related to Sierra game formats:

• sierra-audio
• sierra-vmd

See Also:

• Lossless Audio Anomalies, where some lossless audio decoders do strange things with incomplete chunks

I was catching up on about 3 months worth of QuickTime movie trailers when I viewed the trailer for a movie called Miss March. The part that stood out for me is that the main characters meet a rapper whom they address as “Dot MPEG”. The IMDb page for the movie actually lists the rapper’s name as — ahem — Horsedick.MPEG.

No big meaning to this post; I just thought it would be interesting to the multimedia nerd readership of this blog.

Some years ago, I took a 1-minute sample of a song from a CD and compressed it using every lossless audio coder I knew of at the time — a dozen of them in all. I put the samples here. This effort predated FATE by a number of years. Nowadays, FFmpeg contains native decoding support for 7 of those lossless audio algorithms. And since lossless decoders, by definition, are supposed to have bitexact results, this should be an easy task to add automated tests to FATE.

My pragmatic rule for FATE samples is to try to keep the samples under 2 MB where feasible. Since most of these lossless samples I created weigh in between 6-7 MB, I set about slicing off the first megabyte of each supported sample. And that’s when I noticed that the output was not bitexact across configurations, at least not for all the algorithms. This struck me as odd.

The following lossless algorithms produced identical output across platforms, even with an incomplete final chunk:

• FLAC
• Monkey’s Audio/APE
• Meridian Lossless Packing (MLP)
• Shorten
• WavPack

Apple Lossless generates 4 results — all Linux/x86_32 agreed, all Linux/x86_64 agreed, all Linux/PPC agreed, and Mac OS X x86_64 and PPC agreed.

True Audio Lossless generates 19 different results — all configurations disagreed except for Mac OS X x86_64 and PPC, which agreed with each other.

Digging deeper using ‘-f framecrc’ demonstrates that all frames agree across configurations until the last, incomplete frame (and, of course, the complete files are bitexact). No big emergency; I just thought it was interesting. I will be able to contrive a new, smaller ALAC sample using iTunes (or FFmpeg using Jai’s encoder), and it looks like True Audio’s encoder is still around as well, and available for Linux to boot.