I’ve been feeling a bit scattered for the last week since I was fired from my volunteer position as the FFmpeg QA manager, wondering if there is anything I should attempt to do with the project. It can’t be denied that the new system is working well. But one area I’ve wondered about is test coverage.

Under my old regime I tracked test coverage as a wiki page which was a highly flawed method– tedious and error-prone. There are those 2 adjectives again– tedious and error-prone; whenever I see those, I search for automation methods. I think that might be more plausible thanks to the new FATE’s tighter integration with the FFmpeg build system.

I don’t think anyone is working on this problem so I wanted to toss out a brainstorm:

1. First, run ‘ffmpeg -formats’, ‘ffmpeg -codecs’, etc. and parse the output to collect a list of all the features (full list: -formats, -codecs, -bsfs, -protocols, -filters, -pix_fmts). Transform these lists into a standardized list of features, e.g., "DEVSD ffvhuff Huffyuv FFmpeg variant" represents features ‘decode-video-ffvhuff’, ‘encode-video-ffvhuff’, ‘ffvhuff-horizband’, and ‘ffvhuff-dr1’.
2. Next, tag each individual test spec with the features that it exercises. E.g., test ‘fate-vqa-cc’ exercises features ‘demux-wsvqa’, ‘decode-video-vqavideo’, and ‘decode-audio-adpcm_ima_ws’.
3. Finally, compare the data from parts 1 and 2. Print a list of all the features that are not exercised in FATE.

I think a lot of this could be implemented at the GNU make level. Then again, I’m no expert on GNU make syntax so I may be overestimating its capabilities. Or there might be simpler ways to automatically track test coverage stats based on the improved testing infrastructure.

The FATE main page exposes a lot of data. The manner in which it is presented has always been bounded by my extremely limited web development abilities. I wrestled with whether I should learn better web development skills first and allow that to inform any improved design, or focus on the more useful design and invest my web development learning time towards realizing that design.

Fortunately, Mans solved this conundrum with an elegantly simple solution:

The top of the page displays a status bar that illustrates — at a glance — how functional the codebase is. The web page source code identifies this as the failometer. It took me a few seconds to recognize what information that status bar was attempting to convey; maybe it could use a succinct explanation.

Mini-Book Review
Continue reading →

At any given time, I have between 20-30 blog posts in some phase of development. Half of them seem to be contemplations regarding the design and future of my original FATE system and are thus ready for the recycle bin at this point. Mans is a man of considerably fewer words, so I thought I would use a few words to describe the new FATE system that he put together.

Overview
Here are the distinguishing features that Mans mentioned in his announcement message:

• Test specs are part of the ffmpeg repo. They are thus properly versioned, and any developer can update them as needed.
• Support for inexact tests.
• Parallel testing on multi-core systems.
• Anyone registered with FATE can add systems.
• Client side entirely in POSIX shell script and GNU make.
• Open source backend and web interface.
• Client and backend entirely decoupled.
• Anyone can contribute patches.

Client
The FATE build/test client source code is contained in tests/fate.sh in the FFmpeg source tree. The script — as the extension implies — is a shell script. It takes a text file full of shell variables, updates source code, configures, builds, and tests. It’s a considerably minor amount of code, especially compared to my original Python code. Part of this is because most of the testing logic has shifted into FFmpeg itself. The build system knows about all the FATE tests and all of the specs are now maintained in the codebase (thanks to all who spearheaded that effort– I think it was Vitor and Mans).

The client creates a report file which contains a series of lines to be transported to the server. The first line has some information about the configuration and compiler, plus the overall status of the build/test iteration. The second line contains ‘./configure’ information. Each of the remaining lines contain information about an individual FATE test, mostly in Base64 format.

Server
The server source code lives at http://git.mansr.com/?p=fateweb. It is written in Perl and plugs into a CGI-capable HTTP server. Authentication between the client and the server operates via SSH/SSL. In stark contrast to the original FATE server, there is no database component on the backend. The new system maintains information in a series of flat files.

Mans sent a message to the FFmpeg-devel list today:

A new FATE

Mike’s FATE system has done a great job over the last few years. It
is however beginning to prove inadequate in various ways:

[various shortcomings already dissected at length on this very blog]

To address the above-mentioned issues, I have been working on a
replacement system which is now ready to be announced.

Check it out: http://fate.ffmpeg.org/.

Considering that he just obsoleted something I’ve poured a lot of time and energy into over the last 2.5 years, is my first reaction to this news supposed to be unbridled joy? Hey, I’m already on record as stating that I wouldn’t mind throwing away all of FATE if there was a better alternative.

I’m not certain but I’m pretty sure that at this point, the original FATE server is practically obsolete. Mans is already testing all of his configurations as well as the configs I test. As soon as the other FATE installations switch over to the new server, I should be able to redirect fate.multimedia.cx -> fate.ffmpeg.org, sell most of my computers, and spend more time with my family.

Thanks, Mans!