For 2 years, I’ve been wrangling with the overall architecture of the FATE system. Right now, there’s a primary fate-script.py program that manages building and testing on FATE installations that is sorely in need of many upgrades. Then there is fate-client.py which is a somewhat lighter-weight testing program. I have long wanted to merge the 2 scripts but could never settle on how to do it. At the base level, should the script assume that it’s operating in continuous build/test mode (a la fate-script.py) or single-shot test mode (a la fate-client.py) by default? Or neither? Should it force the user to specify a mess of command line options?

I had an epiphany recently while reading the documentation for Python’s standard (as of v2.3) optparse module, specifically the section entitled “What are options for?”

Options are used to provide extra information to tune or customize the execution of a program. In case it wasn’t clear, options are usually optional. A program should be able to run just fine with no options whatsoever… Lots of people want their programs to have ‘required options’. Think about it. If it’s required, then it’s not optional!

This led me to consider the possibility of a FATE script that could do something without any options and then build from there.

Default Operation
Create a script that can run a set of tests using an already-built binary. Per default, assume that the samples directory is ‘fate-suite/’ in the current directory, that the ffmpeg binary is in the user’s path, that the libraries (if needed) are in the LD_LIBRARY_PATH, and that the test specs are to be downloaded via HTTP. By default, the script should leverage all available CPUs. The script would be fairly verbose about its operation. Results will be logged to an SQLite file.

Tuning Parameters
(Be advised that this blogging software never shows double dashed options correctly. Scratch that— thanks, nine, for the tip about <code> tags.)

Samples might be in a different path, hence: -s, --samplespath

FFmpeg binary might be in a different path, hence: -f, --ffmpegbin

FFmpeg supporting libraries might be in a different path, hence: -l, --librarypath

The test cache might be local or in a different remote location, hence: -c, --cachefile;
if address starts with ‘http://’, attempt to download file; otherwise, try a local file

Only certain tests should be run, instead of all tests, hence: -t, --tests;
this needs to provide flexibility to run ranges of tests, individual tests, e.g.: “-t 4,9,13-16,19-78”

The script should allow control of how many working processes it will occupy, hence: -p, --processes

The script may need to emit no chatter, just error conditions, hence: -q, --quiet

Going Farther
It might be desired to build FFmpeg in advance of running the test suite, hence, add an option to specify a file that has building parameters (because I think it’s too far-fetched to specify these parameters via the command line): -b, --buildfile

The build file will be a Python file with a Python data structure that enumerates a list of building command lines, serving much the same purpose as the current fateconfig.py file. Each item will be a dictionary with keys for config_id (None by default) and build_string. The latter option will support a {FATE_CPU_COUNT} that will be substituted for the number of actual parallel processes that should be used. It may also be desirable to add a compiler_string to each dictionary (and add a column to the corresponding database table) so that continuously upgraded, experimental compilers can have more specific version information attached.

The build/test results will probably be sent to a central server, hence: -r, --results

This results option requires a credential file, another Python file with a single dictionary specifying keys: user_id and hash_key. Again, these items shouldn’t be specified directly on any command line.

The results are sent to a hardcoded location. But it should be override-able with another option (e.g., testing server). Hence, --server.

Per default in the build-then-test mode, and if a configuration ID has been assigned, the script should query the server for the latest build number corresponding to that ID and only conduct a new build/test cycle if there is a higher/later code revision available. This should be override-able, hence: --force-build

New feature: Forcing a build and submitting results for a code revision that has already been logged will effectively overwrite the results already on the server. In effect, FATE installation maintainers can initiate a do-over if, for example, the samples directory wasn’t mounted and all the tests failed.

New feature: In querying for the latest build number corresponding to a configuration, wrap the request in an HMAC authentication. Not for security, but as an early warning mechanism to indicate that the credentials are wrong (rather than learning at the end of the build/test cycle when trying to send the results).

Continuous Operation
By default, this revised model supports the operation that Mans prefers at his installations– running FATE periodically from cron. I prefer to run the script continuously from a ‘screen’ prompt. Running the script continuously should be a matter of a new option, hence: --continuous. Further, in support of Mans’ cron-style operation, there needs to be an option to indicate that the script should run as long as there are no more new code revisions to be built and tested, hence: I’m not sure about this one, maybe --run-while-new-code?

Configuration File
Now that all of those options are defined, it seems reasonable that many of them could be configured via an optional file and specified via --configfile. I imagine that the policy should be to set the default parameters, load the options for a config file if specified, and then override those parameters using any additional options.

Conclusion
This gives me hope. Thanks, Python standard library documentation! This also lays a framework for some other ideas I have, like versioning the test suite, and I’m certain others have good ideas that would be workable within this same framework.