Ever since I figured out that FATE falls into the category of continuous integration (CI) software, I have been evaluating alternate, established, mature software packages that might be able to supplant FATE for managing FFmpeg automated building and testing. I don’t think it’s fair to say that I have been disappointed with the offerings I have found out there; it’s just that many other software packages have apparently approached this problem from a slightly different angle, with a slightly different set of assumptions, and slightly different resources on which to deploy a solution.
For starters, when I read the overviews for a number of these packages, I am never quite sure if they actually do what FATE already does. Many packages brag about their amazing abilities to perform continuous builds on many platforms. I might be missing something crucial here, but that fact alone doesn’t impress me. From my perspective, and in my experience, automating the build among multiple independent machines is the easy part. The difficult part is managing tests and aggregating the results back to a central location.
Aside: Sometimes I back up and wonder: Why aren’t our tests integrated into the build process? Oh yeah, they are: ‘make test’. But why can’t that general test be made to cover all the functionality? Because the ‘make test’ regression suite depends on the existence of both an encoder and a decoder for a particular format. FFmpeg covers a large amount of formats that it can only decode, not encode (and no one is going to step up to write even naive corresponding encoders for these formats in the near future, nor would I argue that they should). Therefore, it becomes necessary to test asymmetric decoder (and demuxer) functionality using individual tests as I designed FATE to do.
I designed FATE such that a client program would run on diverse platforms, constantly check for and rebuild new FFmpeg code, run a battery of tests, and log the results of the build and tests (including status, stdout, stderr, and performance stats) back to a central server. I wanted the client to be written in Python and use only the (expansive) standard Python library, if possible. More stringent were the server requirements: I needed something that operated via HTTP backed by PHP or Perl — because that’s what my web host provides, at least if I want to talk to MySQL (Ruby is also an option; so is Python as long as the CGI script doesn’t need to talk to MySQL).
“But you could host the central server at home on your own broadband and use whatever configuration you want.” Actually, administering things like web and email services is something I would rather pay a few dollars per month for someone else to do on my behalf. I am quite adamant about this. I feel the same way about laundry.
“If FATE ain’t broke, why fix it?” Well, it is sort of broken, or at least incomplete. Many of these packages already boast features like email notifications and even IRC notifications, in addition to better web interfaces– features that I would like to implement when time permits. And at the very least, surveying other CI software might give me ideas about how to improve FATE.
Can existing CI packages solve my problems? Continue reading