Granular Regression Tests
August 25th, 2008 by 
Multimedia Mike
I have been studying the FFmpeg regressions test infrastructure ever so carefully so I can decide the best way to convert all of its functionality and test coverage into the FATE system as individual test specs. I have some ideas but I never know when I will have both the time and the motivation to implement them. In the interim, I am retiring the master ‘make test’ full regression test spec. In studying the regression architecture, it’s immediately obvious that ‘make test’ invokes these 3 sub-regression suites, which each have their own test specs now:
That should help until I break down the tests even further. Oh, and I also added a test spec for ‘ffmpeg -h’. Hey, FFmpeg got into a state once where it segfaulted on help. We don’t want to let that go uncaught again.
Posted in FATE Server | 
5 Comments »
August 28th, 2008 at 4:19 am
As someone suggested in the ML, doing something like
make tests/data/lavf.regression > /dev/null 2> /dev/null; diff -u -w tests/libav.regression.ref tests/data/lavf.regression
And testing the stout (fail if there is any) might allow FATE to print the diff when the test fail.
August 28th, 2008 at 8:15 am
Caveat: You would also need to play games with the return value from the first part of the command (since the second part, the diff, trashes the return value from the earlier part).
August 29th, 2008 at 4:11 am
Hello Mike. Have you seen the new Wine Weekly News issue?
They introduced a “patch watcher”, build bot that tests non-committed patches floating on the wine mailing list. Maybe something of interest related to your work on FATE.
http://www.winehq.org/site?issue=351#The%20PatchWatcher
August 29th, 2008 at 4:17 pm
@Mike: Indeed. Maybe something like
make tests/data/lavf.regression > /dev/null 2> /dev/null && diff -u -w tests/libav.regression.ref tests/data/lavf.regression
would work?
September 6th, 2008 at 7:16 pm
Thanks for the tip, Martin. I blogged about The Patch Watcher separately.