Ever since I figured out that FATE falls into the category of continuous integration (CI) software, I have been evaluating alternate, established, mature software packages that might be able to supplant FATE for managing FFmpeg automated building and testing. I don’t think it’s fair to say that I have been disappointed with the offerings I have found out there; it’s just that many other software packages have apparently approached this problem from a slightly different angle, with a slightly different set of assumptions, and slightly different resources on which to deploy a solution.

For starters, when I read the overviews for a number of these packages, I am never quite sure if they actually do what FATE already does. Many packages brag about their amazing abilities to perform continuous builds on many platforms. I might be missing something crucial here, but that fact alone doesn’t impress me. From my perspective, and in my experience, automating the build among multiple independent machines is the easy part. The difficult part is managing tests and aggregating the results back to a central location.

Aside: Sometimes I back up and wonder: Why aren’t our tests integrated into the build process? Oh yeah, they are: ‘make test’. But why can’t that general test be made to cover all the functionality? Because the ‘make test’ regression suite depends on the existence of both an encoder and a decoder for a particular format. FFmpeg covers a large amount of formats that it can only decode, not encode (and no one is going to step up to write even naive corresponding encoders for these formats in the near future, nor would I argue that they should). Therefore, it becomes necessary to test asymmetric decoder (and demuxer) functionality using individual tests as I designed FATE to do.

I designed FATE such that a client program would run on diverse platforms, constantly check for and rebuild new FFmpeg code, run a battery of tests, and log the results of the build and tests (including status, stdout, stderr, and performance stats) back to a central server. I wanted the client to be written in Python and use only the (expansive) standard Python library, if possible. More stringent were the server requirements: I needed something that operated via HTTP backed by PHP or Perl — because that’s what my web host provides, at least if I want to talk to MySQL (Ruby is also an option; so is Python as long as the CGI script doesn’t need to talk to MySQL).

“But you could host the central server at home on your own broadband and use whatever configuration you want.” Actually, administering things like web and email services is something I would rather pay a few dollars per month for someone else to do on my behalf. I am quite adamant about this. I feel the same way about laundry.

“If FATE ain’t broke, why fix it?” Well, it is sort of broken, or at least incomplete. Many of these packages already boast features like email notifications and even IRC notifications, in addition to better web interfaces– features that I would like to implement when time permits. And at the very least, surveying other CI software might give me ideas about how to improve FATE.

Can existing CI packages solve my problems?

First, there’s BuildBot. I like the cut of its jib. It’s written in Python so I have a fighting chance of understanding its inner workings. But more importantly, its documentation isn’t terribly buzzwordish and gets straight to the point: “The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes.” It has an architecture similar to FATE and could probably accomplish what FATE does. The dealbreaker is that the server requires a little too much in the way of resources. I need to be able to install an entire Python package on the server, something I lack the privileges to do and something my provider is reticent to do.

BuildBot uses the paradigm of the source control repository pushing changes to the BuildBot central server via one channel or another. Then, build slaves query the server for specific build assignments. Actually, I think this is a fairly common paradigm for CI based on my survey (even more common seems to be the model of the central server connecting to build slaves and ordering builds, something that definitely isn’t going to happen in FATE’s topology). Following this model would be useful for ensuring that each FATE client builds every single change. I don’t really know if that would be useful or not, though; mostly, I think it would just create a huge backlog for the slower machines on the farm. However, revising FATE such that clients fulfill build orders from the central server might create a method for developers to check source into a testing branch and put in a special request for the code to be built on all FATE platforms for validation.

Now that I think about it, maybe that’s just the kind of thing that various CI packages are bragging about when it comes to automated building.

Check out the first item on Cabie’s feature list, prominently displayed on their home page: “Written in EXTREMELY EASY TO READ PERL…” Now why would the author feel it necessary to stipulate — forcefully — that this Perl code is readable? Maybe Perl has a reputation. I won’t hold the language against the overall package, though; if it’s written well enough, I shouldn’t have to hack it anyway. The dealbreaker comes early in this setup tutorial: “The remote build servers must have network mounted (NFS or SAMBA for MS/Win) file access which allows the web server access to the build logs and converted postbuild HTML pages.” Okay, that’s not happening. Next.

Hudson comes to me by way of Xuggle, which has something to do with FFmpeg and Java. Unsurprisingly, Hudson is built in Java, which is pretty much a dealbreaker by itself. Java doesn’t run on near enough platforms as Python. Plus, I’m pretty sure I can’t run a Java servlet on my web host.

I can’t discount the idea of a Java-based build client entirely, though. There is always the opportunity to run the client on a platform that does support Java, only to interact (probably via SSH) with the actual build/test machine to get work done. I am in the process of revising FATE to support this very model so that it can be used to test, e.g., the ARM/Linux platform. A lot of these CI systems seem to be single-language affairs, however, so a Java-based client implies a Java-based server. I might be the only one crazy enough (or necessity-driven enough) to have developed a CI system based on 2 languages.

Xinc is a CI system written in PHP. I honestly don’t know at which level the system operates, or what it tests. I used to think that PHP was restricted to server-side web page processing and was stunned the day I realized I could run a snippet of PHP code from the Unix command line. Watching the Xinc slideshow grants me a little more clue — PHP functions on both the client and the server — and leads me to believe that maybe, just maybe, Xinc is similar to what I want for FATE.

It turns out that Xinc actually stands for Xinc Is Not CruiseControl. What is CruiseControl? It’s a popular name associated with CI. I think the Java-based CruiseControl claims to be the original CI system. There are also .NET-based and Ruby-based versions of CruiseControl. Java and .NET generally disqualify a CI system from consideration, but what about the Ruby thing? Let’s look for a dealbreaker there… ah, found it in the Getting Started guide: “Dashboard and all builders need to run on the same computer.” Again, not making any statements about the quality of the software, just acknowledging that the designers obviously have a different set of requirements than I do.

If I understand the ancestry correctly, Tinderbox is the spiritual predecessor of BuildBot. There are several iterations of the software but the latest is Tinderbox 3. The feature page lists that the backend database server is required to be PostreSQL, and I’m pretty sure that’s a dealbreaker. Further, I get the impression that the thing is only set up for builds, no tests (though I suppose tests can be an implicit part of builds). Here’s one interesting idea I took away from the Tinderbox page: Tinderbox attempts to “keep multiple build machines even–i.e. if there are 6 machines uploading builds to this machine, it will try to keep the builds at the same frequency for them–one build per hour, or one build per three hours, etc. This way each machine has the same coverage.” Their auto-updating feature for client software is also tantalizing.

I checked out as many other open source CI packages as I could find but they all seemed to be .NET-based or Java-based. Do I really impose such unique requirements for a CI system? Maybe so, and maybe other people have that same set of requirements because I am starting to get requests from other projects about using FATE as a general CI system.