I really appreciate the flurry of SVN activity on FFmpeg today– it helped build confidence in my revised build/test script. FATE work items completed tonight:

• set up a configuration to build and test 64-bit FFmpeg binaries for Mac OS X; thanks to David Conrad for schooling me on how to do this in the comments to the last post
• re-instated the Linux/x86_32 and x86_64 configurations
• upgraded all gcc-svn configurations to the latest version
• disabled all gcc 4.3.1 configurations; replaced with latest and greatest 4.3.2 configurations

I still haven’t gotten Python 2.5.2 on Gentoo to import sqlite3, so the PowerPC is still cheerfully running the classical, less efficient build/test script.

I sat back and watched the various configurations rip through their build/test assignments. Then I noticed a result on the main FATE page that had passed 85/94 tests. That’s a bug (too few tests). So I did what came naturally, deleted the record, and am waiting to see if it happens again. I think I may know what the problem is, though. That’s the nice thing about being the sole developer of a system– it can actually be possible to juggle all the pieces in your head, understand all the design decisions behind them, and intuitively understand what might be wrong.

Okay, FATE is back online, somewhat. First and foremost, you will notice that Mac OS X autobuilds are at the top of the page. I know it doesn’t look like much is different but I changed a whole lot of stuff under the covers to get to that point. I have not re-instated the x86_32 or x86_64 build/test cycles for Linux yet because I would like to see this new script bake for a day or 2 before copying it to other systems. The PPC build/test cycles are running because I can’t figure out how to make the new script run on that machine (hi Gentoo!), so it gets to stick to the older, still functional script for now.

If you will indulge a little self back-patting, I’m rather pleased with how this new system is shaping up. I set out to solve one problem but I wound up realizing solutions and better approaches for a whole lot more things. The new system is already faster and more resilient to intermittent network problems (that will never go away completely no matter what availability and bandwidth guarantees we have). Going forward, I have new ideas about how to make the system easier to administer, and to allow co-administrators to help out as well. Look for more platforms on FATE in the near future as it should be much easier for others to run the client program and automatically submit data back to the server. And it may even be possible to adapt the system for other projects.

I look forward to writing up more notes about the infrastructure changes. Most of them boil down to my new love affair with SQLite.

The FATE server started getting frustrating and dispiriting to maintain, so I decided to scrap it a little while ago. But I have since started to heavily revise the infrastructure so it can come back online. I have been sitting on a pile of brainstorms about how to make the system work better. Once I finally got down to implementing the changes, it sort of snowballed and I thought of even more improved ways that the various pieces could work together.

But this growth is not without its associated pains. I largely blame PHP for this. Whenever I have a bad day at work, I just remind myself that things could be a lot worse. For example, it could be my job to write PHP code full time. I have lots of gripes with the language, but a few new ones due to this experience.

There must be 7 different ways to interface to one library I want to use, and I can’t get any of them to work. And since it’s all server-side, it’s incredibly difficult to diagnose why the server is having trouble.

PHP is hyper-paranoid about security. When you GET or POST data, PHP’s site-specific (that I can’t change) setting is to escape quotes and backspaces before it makes the data available to you, whether you like it or not. And I don’t like it. I really don’t want the data escaped, but I can’t turn it off. The manual states that the next version of PHP will remove this annoyance.

But there’s no point in complaining about PHP. As Jeff Atwood eloquently expressed, PHP Sucks, But It Doesn’t Matter. It’s still serves as the backbone of some of the most important sites on the internet. And I know I will eventually coerce PHP to be the backbone of FATE once more.

Python isn’t blameless in this either. I need a key feature that, for once, is not provided by the expansive Python standard library (even though the library handles everything else associated with this type of functionality). A few hackers around the net have attempted to fill in the missing piece but I haven’t successfully adapted their code yet.

On the plus side, I should mention that I have gotten FATE running on Mac OS X. It’s currently watching FFmpeg SVN and performing build/test cycles while saving the data locally. That was the easy part. Getting the data to the server is the troublesome part and the foregoing issues described are all components of that problem.

I have a confession– it has been a very long time since I created any meaningful backup of the FATE database. The thing has been operational for over 8 months now and has collected hordes of data. I already had one adventure with dramatically trimming the data size. But when I try to create a backup, it just goes on and on and on.

I finally thought to look up what MySQL facilities might be able to help me diagnose this. Here’s something: SHOW TABLE STATUS. It turns out that the table that stores the build records is nearly 600 MB large. The table that holds the test results exceeds 230 MB (at least it’s not 2.3 GB like the last time I visited the size issue). The same culprit is to blame– stdout and stderr:

mysql> SELECT
    ->   SUM(LENGTH(stdout)),
    ->   SUM(LENGTH(stderr))
    -> FROM build_record;

+---------------------+---------------------+
| SUM(LENGTH(stdout)) | SUM(LENGTH(stderr)) |
+---------------------+---------------------+
|           424930434 |           165123451 | 
+---------------------+---------------------+

The database has evolved before and it’s time for the next evolution, based on the premise that there is almost no reason whatsoever to store the stdout data. I know of one circumstance where it helps– it is useful to read the configure script output to verify configuration options (especially after I update the gcc SVN configurations). And there is definitely a good reason to keep stderr around– if the build fails, the stderr is the first stop for diagnosing the problem. Even if the build succeeds, it should be theoretically possible to compare stderr across builds to find common warnings that should be eliminated (possible future expansion).

So, my solution: Periodically retire old data. In this case, I was planning to retire stdout/stderr on build records from before September. But, well… mistakes were made and I managed to retire ALL the stdout/stderr data for all existing build records, thus further underscoring the need for responsible periodic backups. Eh, I don’t think anyone looked at that data anyway. And the table is nicely compact now:

+---------------------+---------------------+
| SUM(LENGTH(stdout)) | SUM(LENGTH(stderr)) |
+---------------------+---------------------+
|              665532 |              665532 | 
+---------------------+---------------------+

It’s actually possible to back it up now. A curious observation: MySQL’s SHOW TABLE STATUS still reports a Data_length for this table of 591471720 bytes, which concerned me. But then I spied the Data_free column which reports 588306412 bytes. I think the numbers are related.

I’m still deciding on more efficient policies for the test results table, especially since I have plans to expand to other platforms soon.