I’m ecstatic that the FATE Server has demonstrated its first real success by highlighting a failure. Specifically, the system already caught a discrepancy in a new test designated alg-mm. This tests the American Laser Games MM file format. FATE did precisely what I wanted in that it was able to report that the demux operation worked, the PCM audio data was all there, but that the video frames were decoding differently on different CPU architectures.

What, did you think that I was really carrying out this FATE project for the general benefit of FFmpeg and its users? Ha! Now we come to the real, fiendish rationale behind FATE: to make sure all of the lesser known modules (read: fringe, game-related formats) in the project remain operational over time. How many times have I returned to an old fringe format in FFmpeg after a long absence and found that it broke during one of many API upgrades? In fact, that’s precisely why I have been hesitant to repair any breakages I have found in the past 1.5 years (when I first started brainstorming what would become FATE) because I wanted this test infrastructure in place to notify me of further breakages.

I investigated the alg-mm issue further and it appears that each frame also has its palette prepended. The palette is stored in native endian format, which is obviously causing trouble for automated testing. Palette handling is a longstanding issue in FFmpeg that has yet to be solved and also falls outside the scope of the immediate task of getting as many working tests into the database at the outset as possible. As such, I have disabled the test for now. Fortunately, I should feel more motivated to develop proper palette handling later on once I am confident that future breakages will be detected early.

Anyway, I have started adding more tests, beginning with the bit exact tests in the good old QuickTime Surge audio suite (a particular audio sample encoded in just about every audio format QuickTime supports). Now I am ready to move on to the next big challenge that I knew this project would present– how to test data that is not defined to decode in a bit exact manner. A prime example is MP3 audio data.

I have been told to look at tiny_psnr.c for this exercise.