Investigating Alternate Continuous Integration Software

Ever since I figured out that FATE falls into the category of continuous integration (CI) software, I have been evaluating alternate, established, mature software packages that might be able to supplant FATE for managing FFmpeg automated building and testing. I don’t think it’s fair to say that I have been disappointed with the offerings I have found out there; it’s just that many other software packages have apparently approached this problem from a slightly different angle, with a slightly different set of assumptions, and slightly different resources on which to deploy a solution.

For starters, when I read the overviews for a number of these packages, I am never quite sure if they actually do what FATE already does. Many packages brag about their amazing abilities to perform continuous builds on many platforms. I might be missing something crucial here, but that fact alone doesn’t impress me. From my perspective, and in my experience, automating the build among multiple independent machines is the easy part. The difficult part is managing tests and aggregating the results back to a central location.

Aside: Sometimes I back up and wonder: Why aren’t our tests integrated into the build process? Oh yeah, they are: ‘make test’. But why can’t that general test be made to cover all the functionality? Because the ‘make test’ regression suite depends on the existence of both an encoder and a decoder for a particular format. FFmpeg covers a large amount of formats that it can only decode, not encode (and no one is going to step up to write even naive corresponding encoders for these formats in the near future, nor would I argue that they should). Therefore, it becomes necessary to test asymmetric decoder (and demuxer) functionality using individual tests as I designed FATE to do.

I designed FATE such that a client program would run on diverse platforms, constantly check for and rebuild new FFmpeg code, run a battery of tests, and log the results of the build and tests (including status, stdout, stderr, and performance stats) back to a central server. I wanted the client to be written in Python and use only the (expansive) standard Python library, if possible. More stringent were the server requirements: I needed something that operated via HTTP backed by PHP or Perl — because that’s what my web host provides, at least if I want to talk to MySQL (Ruby is also an option; so is Python as long as the CGI script doesn’t need to talk to MySQL).

“But you could host the central server at home on your own broadband and use whatever configuration you want.” Actually, administering things like web and email services is something I would rather pay a few dollars per month for someone else to do on my behalf. I am quite adamant about this. I feel the same way about laundry.

“If FATE ain’t broke, why fix it?” Well, it is sort of broken, or at least incomplete. Many of these packages already boast features like email notifications and even IRC notifications, in addition to better web interfaces– features that I would like to implement when time permits. And at the very least, surveying other CI software might give me ideas about how to improve FATE.

Can existing CI packages solve my problems?

BuildBot logo

First, there’s BuildBot. I like the cut of its jib. It’s written in Python so I have a fighting chance of understanding its inner workings. But more importantly, its documentation isn’t terribly buzzwordish and gets straight to the point: “The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes.” It has an architecture similar to FATE and could probably accomplish what FATE does. The dealbreaker is that the server requires a little too much in the way of resources. I need to be able to install an entire Python package on the server, something I lack the privileges to do and something my provider is reticent to do.

BuildBot uses the paradigm of the source control repository pushing changes to the BuildBot central server via one channel or another. Then, build slaves query the server for specific build assignments. Actually, I think this is a fairly common paradigm for CI based on my survey (even more common seems to be the model of the central server connecting to build slaves and ordering builds, something that definitely isn’t going to happen in FATE’s topology). Following this model would be useful for ensuring that each FATE client builds every single change. I don’t really know if that would be useful or not, though; mostly, I think it would just create a huge backlog for the slower machines on the farm. However, revising FATE such that clients fulfill build orders from the central server might create a method for developers to check source into a testing branch and put in a special request for the code to be built on all FATE platforms for validation.

Now that I think about it, maybe that’s just the kind of thing that various CI packages are bragging about when it comes to automated building.

Cabie logo

Check out the first item on Cabie’s feature list, prominently displayed on their home page: “Written in EXTREMELY EASY TO READ PERL…” Now why would the author feel it necessary to stipulate — forcefully — that this Perl code is readable? Maybe Perl has a reputation. I won’t hold the language against the overall package, though; if it’s written well enough, I shouldn’t have to hack it anyway. The dealbreaker comes early in this setup tutorial: “The remote build servers must have network mounted (NFS or SAMBA for MS/Win) file access which allows the web server access to the build logs and converted postbuild HTML pages.” Okay, that’s not happening. Next.

Hudson comes to me by way of Xuggle, which has something to do with FFmpeg and Java. Unsurprisingly, Hudson is built in Java, which is pretty much a dealbreaker by itself. Java doesn’t run on near enough platforms as Python. Plus, I’m pretty sure I can’t run a Java servlet on my web host.

I can’t discount the idea of a Java-based build client entirely, though. There is always the opportunity to run the client on a platform that does support Java, only to interact (probably via SSH) with the actual build/test machine to get work done. I am in the process of revising FATE to support this very model so that it can be used to test, e.g., the ARM/Linux platform. A lot of these CI systems seem to be single-language affairs, however, so a Java-based client implies a Java-based server. I might be the only one crazy enough (or necessity-driven enough) to have developed a CI system based on 2 languages.

Xinc is a CI system written in PHP. I honestly don’t know at which level the system operates, or what it tests. I used to think that PHP was restricted to server-side web page processing and was stunned the day I realized I could run a snippet of PHP code from the Unix command line. Watching the Xinc slideshow grants me a little more clue — PHP functions on both the client and the server — and leads me to believe that maybe, just maybe, Xinc is similar to what I want for FATE.

CruiseControl logo

It turns out that Xinc actually stands for Xinc Is Not CruiseControl. What is CruiseControl? It’s a popular name associated with CI. I think the Java-based CruiseControl claims to be the original CI system. There are also .NET-based and Ruby-based versions of CruiseControl. Java and .NET generally disqualify a CI system from consideration, but what about the Ruby thing? Let’s look for a dealbreaker there… ah, found it in the Getting Started guide: “Dashboard and all builders need to run on the same computer.” Again, not making any statements about the quality of the software, just acknowledging that the designers obviously have a different set of requirements than I do.

If I understand the ancestry correctly, Tinderbox is the spiritual predecessor of BuildBot. There are several iterations of the software but the latest is Tinderbox 3. The feature page lists that the backend database server is required to be PostreSQL, and I’m pretty sure that’s a dealbreaker. Further, I get the impression that the thing is only set up for builds, no tests (though I suppose tests can be an implicit part of builds). Here’s one interesting idea I took away from the Tinderbox page: Tinderbox attempts to “keep multiple build machines even–i.e. if there are 6 machines uploading builds to this machine, it will try to keep the builds at the same frequency for them–one build per hour, or one build per three hours, etc. This way each machine has the same coverage.” Their auto-updating feature for client software is also tantalizing.

I checked out as many other open source CI packages as I could find but they all seemed to be .NET-based or Java-based. Do I really impose such unique requirements for a CI system? Maybe so, and maybe other people have that same set of requirements because I am starting to get requests from other projects about using FATE as a general CI system.

6 thoughts on “Investigating Alternate Continuous Integration Software

  1. nine

    I’m sure you could find someone willing to run such a python application on a machine for you. You could even find someone to simply volunteer to pay the $40/mo or so to get a decent VPS and run it on that.
    Whether or not that is more work is left as an exercise for the reader :)

  2. John Koleszar

    I’ve done a lot of research into various CI software because I want to do something very similar to what you’re doing with FATE at work. What I’ve found is that all the existing software sucks at doing cross (embedded) development.

    Many of them don’t support making the build and test separable at all, so you have problems like having to control the test machine remotely, and even more fun is to get sequential builds but concurrent tests (ie start the next build while the test slaves are busy testing). Remote control isn’t too hard if all your test slaves speak ssh, but if ssh isn’t installed, or isn’t available (say a Windows CE device), then what? They all generally speak a private protocol between the client and the server, sometimes using language specific features like pickling, which makes it hard to write your own client in a different language. A language like perl/java/python is just too heavy of a dependency on my targets.. I’d love to have a light weight client in C+libcurl just talking HTTP to a webapp.

    I’m leaning towards hudson to handle the builds, as it’s got some slick features for publishing/distributing build artifacts. I’ll probably end up writing a webapp for aggregating test results and a daemon for the test nodes that polls for test specs, downloads the build artifacts and test material, runs the test, and publishes the results to the test server. Some of my targets don’t have real shells (or even run kernels, in some cases), so the daemon will probably just ship logs back to the server to be analyzed there, rather that trying to do all that in C.

    So no, it’s not just you. But we’re in the minority :)

  3. Art Clarke

    I’m normally not a proponent of “build don’t buy”, but FATE tests (and FFMPEG cares about) a variety of platforms that other CI systems are unlikely to support (can’t see a lot of active development in BuildBot OS/2 support).

    My $0.02 is given the nature of the FFMPEG project, and the culture of the community of folks working on it (i.e. very very busy developers who have little time for technology that doesn’t exactly fit their needs), I still think you’ve taken the right approach with FATE: meet the needs of FFMPEG, but use other CI servers as inspiration for features to add.

    At Xuggle we use Hudson (although, despite appearances to the contrary, over 75% of our software is written in C++, not Java) because we only support three OS-varieties today (Windows, Linux, Mac-OS) all of which have robust JVM implementations. It’s not perfect, but it works fairly well for us. Fate doesn’t have the “luxury” of supporting only those platforms for building though.

    – Art

  4. Multimedia Mike Post author

    @JohnK: As soon as FATE needs to test FFmpeg on a device with those types of parameters, I’m confident I’ll adapt my CI system to accommodate. :-) Actually, I thought I would have to radically alter the thing in order to continuously test an ARM device. But thanks to a device like the Beagle Board which runs a Linux environment, a lot of re-org becomes unnecessary.

    You brought up an interesting idea that I have also entertained– parallelizing building and testing, since the operations are (or could be) occurring on different platforms. Perhaps I should task a very fast computer with a very fast disk with performing continuous builds which are then queued for testing in parallel on separate machines via SSH remote commands.

    It’s an interesting idea but won’t happen in FATE anytime soon. PowerPC testing could probably benefit from this, though.

  5. Multimedia Mike Post author

    @Art: Thanks for the affirmation that I’m not completely crazy for persisting in working on my own CI system. I know you have extensive experience in the matter.

  6. follower

    > I need to be able to install an entire Python package
    > on the server, something I lack the privileges to do
    > and something my provider is reticent to do.
    I’ve been working on setting up BuildBot recently ( and wanted to clarify something you mentioned:

    Did you mean you lack the privileges to install a system-wide Python packge or that you can’t even install something in your home directory on the server? AFAICT BuildBot supports installation via the latter.


Comments are closed.