Category Archives: FATE Server

Finally Building

At long last, I have a reasonable — and much improved — autobuild/test infrastructure running for FFmpeg. Visit it at builds.multimedia.cx. It’s a bit sparse right now as it only covers builds for 2 platforms (x86_32 and PowerPC, both on Linux) and only a handful of tests, which are actually pseudo-tests, only logging the filesizes of the ‘ffmpeg’, ‘ffplay’, and ‘ffserver’ binaries. Also, you will notice that the testbed makes an effort to keep up with recent SVN builds of gcc.

I’m hopeful for this new infrastructure; the sky’s the proverbial limit. In the short term, I will be adding x86_64/Linux builds. Solaris/Sparc and Mac OS X builds might be on the way as well. My other high priority right now is to create an administrative web form that will enable me (and hopefully some co-admins) to add and edit test specifications easily. It’s not ergonomic to do this through the MySQL console. Here are the top test ideas so far:

  • All of those ‘make test’ regression tests– break those up into individual tests
  • Type ‘ffmpeg -formats’– set up tests for each and every one of those individual modules; use the libavformat/framecrcenc module here
  • Fetch collections of conformance vectors for various MPEG standards and run through them

The comments section is open for suggestions. Be ambitious. However, here’s an algorithmic challenge if you’re up to it. Look at the stderr output from this build of gcc 2.95.3/x86_32 that failed. It would be useful to highlight the errors. How do I search the text and find the error so I can highlight them, and also insert HTML anchor links for easy skipping? The best I can think to do is searching for ‘***’ to indicate errors and search back somehow.

Revenge Of The Autobuilds

Takis has been a busy FFmpeg hacker: He recently established an experimental server to automatically build the current source-controlled copy of FFmpeg and perform some rudimentary tests with the output. This is some great initiative on his part.

(Oh, and look what else Takis has been up to while no one is looking: a graph of FFmpeg code change over time.)

I have wanted to build an automated building and testing infrastructure for FFmpeg for a long time now. I got my first concept up and running late last November. I just realized that I never blogged about it although I did announce it on the ffmpeg-devel mailing list. The concept lives at http://builds.multimedia.cx/, though be advised that the script that updates it went offline in late December.

Predictably, people seemed to think the autobuild system was a good idea but that my implementation needed a lot of work. And they were right. The reason that I never blogged about it is likely that I figured I was about to deploy a better concept very soon.

It is now July and I have had months to brainstorm ideas for an improved autobuild and test infrastructure. Unfortunately, as can often happen with revision 2 of an unproven idea, I fear my concept has devolved into an exercise in architecture astronomy.


Architecture Astronomy

Read Joel Spolsky’s excellent essay, “Don’t Let Architecture Astronauts Scare You”. It’s about people who heavily theorize in the abstract but rarely accomplish anything useful. Personally, I consider it a clear indicator of architecture astronomy when a program’s fundamental paradigm revolves around the idea that, “Everything is an object (or module)!” It is my opinion that declaring everything in your architecture to be an object is the abstraction endgame (to be more specific, everything is a swappable, user-configurable module, even the central engine of the program that is supposed to coordinate everything between other modules).

I’ll explain the evolution of my autobuild idea: It started simply enough with a script that iterated through a bunch of compiler versions and ran the configure/make commands to build each. It logged stdout and stderr separately and logged general information about success/failure, SVN version, etc. into a rudimentary database table that could be simply queried with a PHP script.

I soon realized that this is wholly inadequate to the overall goals I wished to accomplish in this endeavor (building and testing on many platforms). Security is a major issue, which I blogged about before, and which I solved in the first iteration using the most paranoid policies of chroot’ing the configure/make steps and prohibiting network access during the process. Another problem is the eventuality of infinite loop bugs. Any build or test step could conceivably encounter such a condition.

This realization led me to redesign the autobuild/test system as a series of individual executable steps, all stored in a database, of which the primary script has no hardcoded knowledge. And this is where the “Everything is a module” philosophy comes into play. Unfortunately, the further I plot this out on paper, the harder it becomes because the execution module concept is too generic; it’s hard to do certain specific things. I realize I need to back off a bit on the abstraction.

FFmpeg Autobuilds

I have been sitting on this for at least a month. As brainstormed in this post, I have developed a system on a spare, headless, always-on x86 Linux box that automatically updates its FFmpeg SVN copy and compiles it with 6 different gcc versions, essentially the latest in each of the 2.95, 3.2, 3.3, 3.4, 4.0, and 4.1 series. That’s all working quite well. The part that I’m trying to resolve right now is what to do with the results. I would like to aggregate the results into a concise format for easy web reading. Plus, it would be good to have a history of build successes/failures. I think an RSS aggregation would be useful as well. And for bonus points, some halfway intelligent system that figures out which warnings occur in all or most builds. This would reveal good code janitor work for aspiring FFmpeg developers.

This is just the first phase. Of course, I want to add functional tests later, such as the standard regression. I am still trying to get the infrastructure up.

Related post:

Secure Automated Builds

I have a small, low-power x86 Linux box sitting on the internet. I want to use it to set up an automated build system for FFmpeg. To what end? At the outset, I would like to validate that the entire codebase builds, with all extra modules enabled, under different gcc versions and post results to a webpage.

The process outline is pretty straightforward. At periodic intervals:

  • check out a fresh copy of the FFmpeg SVN tree
  • ‘configure’ with all relevant options and the desired compiler version
  • ‘make’, log the stdout and stderr text in separate text files
  • log the status report to a public website

Pretty simple so far. Other ideas include compiling with multiple compiler versions, including cross-compilers for multiple platforms; automated regression testing; aggregated results sent to a new mailing list [insert your brainstorm here]. I’m starting small because this particular machine requires almost a half hour to complete just one FFmpeg build.

Here is the big item I’m concerned about: How to guard against malicious script injections? Not to say that I don’t trust my FFmpeg brethren but… I can’t think of a good way to end that sentence. But think about it: When you run the configuration script and Makefile, you’re executing free-form shell commands. One of the worser-case scenarios:

all:
    find / | xargs rm -rf

“But Mike,” I hear you exclaim, “as long as you run it as your own unprivileged user vs. root, it won’t destroy the entire system.” That’s terrific news; the easily replaced Linux base system would be safe. So what about running the build process as a heretofore non-existent user (something other than ‘nobody’ since a standard Linux box is going to have files owned by him)? Random shell commands could still, for example, read world-readable files and transmit them offsite. What kind of files? I’m not interested in the fine details, it’s still a security hole.

So there is the possibility of switching to the unknown user in its own chroot’d shell. Ideally, this environment would:

  • prohibit network access (setup process gets the fresh SVN tree before entering chroot shell)
  • have access to various build tools

I guess what I’m wondering is: Have these problems already been solved? Are there open source projects that already provide good solutions to these problems? Otherwise, I already know it’s a lot of work to set up such an chroot environment (and I have no idea how to prohibit network access for a particular user).