This evening, I finally got my fate-client.py script minimally ready for general consumption. fate-client.py is the unimaginatively named program I threw together some time ago to allow me to validate test specs before I activate them so that FATE will automatically test them. It works like this:
- download the script (http://fate.multimedia.cx/fate-client.py)
- rsync the FATE suite of samples that live on mphq: ‘rsync -aL rsync://rsync.mplayerhq.hu:/samples/fate-suite/ samples’ (without the quotes, of course) — this presently amounts to ~150 MB
- build FFmpeg as normal
- ‘./fate-client.py -f </path/to/ffmpeg-binary> -s </path/to/fate-suite/samples>’
That’s it. The script will ask the FATE server for a set of test specifications and run through them. You may also need to specify -l/–libpath= if you built and installed FFmpeg with shared libraries. Naturally, ‘./fate-client.py -h’ will spell out all the options.
You would do well to make sure that all the options are valid or else suffer Python bailout exceptions. I just added the command line options tonight and have not made them very resilient. I have been promising this utility for a long time and I wanted to get something out there sooner than later.
Remember that I’m still a rank amateur at Python, so don’t be afraid to call me out if I’m doing anything in the worst Pythonic way imaginable.
Ideas for future improvement:
- Better logging– Instead of dumping to stdout, maybe dump all the results to a CSV file (for spreadsheet analysis) and/or an HTML file for easy viewing
- Proper versioning– I track the script via a local git repository, but how do I communicate the current version? Would this be version dd394ef8f3dad056c39ab4e1c76951190621cf8b?
- Robust error handling
- Range testing (run all tests up to ID n, or run all tests after ID n, or from IDs m to n)
- Skip a list of tests (for example, it would be useful to skip test #128 — the internal FFmpeg regression test — since it’s not that helpful in this particular scenario)
- [Your idea here]
It’s open source, GPL v2, so patches welcome. Moreover, I would love to hear if this script works at all for anyone else. Then, I would like to hear how it works on platforms outside of the 3 that FATE now rigorously tests– I speak of Mac OS X, *BSD, Win32 with either MSVC or MinGW, Open/Solaris on all its various platforms, even PlayStation 3 and whatever else. I actually did get that OpenSolaris VMware session to boot after I waited long enough but I had no idea how to do anything useful with it. That’s when I decided to get down to it and get this script out there so that hopefully someone else will test those platforms.
Extra credit: Figure out why, when bailing out of the test sequence early with Ctrl-C, terminal character echo is off. I.e., the terminal refuses to print keystrokes.
Time to learn Python then…
I think you should also add associative arrays of tests – like “all game video codecs” or “all codecs written by Mike Melanson” (or are these sets the same?) or “all lossless audio codecs”.
> Figure out why, when bailing out of the test sequence early with Ctrl-C, terminal character echo is off. I.e., the terminal refuses to print keystrokes.
Because ffmpeg set it that way (like echo off in DOS). Run “stty sane” after to restore it
The grouping idea is not too far-fetched. In fact, I hope to do something like that anyway in order to categorize test specs that only make sense for one platform or another. E.g., desktop tests vs. embedded device tests.
Thanks for the tip on turning echo on.
“stty sane” as far as I can tell also disables utf8, so it would be better to use something more specific to fix it.
I just use “reset”, while that has the disadvantage of clearing the display you in exchange get that nice “Erase is backspace.” message ;-)
Hey Mike, just curious, did you ever think about using TAP (http://testanything.org/) as a serialization format for test results?
Thanks for the link, Brian. However, reviewing the materials, I really don’t see how it solves any problems I’m currently facing (like how to transport the data to a database engine living on a remote server).