I have been studying the FFmpeg regressions test infrastructure ever so carefully so I can decide the best way to convert all of its functionality and test coverage into the FATE system as individual test specs. I have some ideas but I never know when I will have both the time and the motivation to implement them. In the interim, I am retiring the master ‘make test’ full regression test spec. In studying the regression architecture, it’s immediately obvious that ‘make test’ invokes these 3 sub-regression suites, which each have their own test specs now:
That should help until I break down the tests even further. Oh, and I also added a test spec for ‘ffmpeg -h’. Hey, FFmpeg got into a state once where it segfaulted on help. We don’t want to let that go uncaught again.
As someone suggested in the ML, doing something like
make tests/data/lavf.regression > /dev/null 2> /dev/null; diff -u -w tests/libav.regression.ref tests/data/lavf.regression
And testing the stout (fail if there is any) might allow FATE to print the diff when the test fail.
Caveat: You would also need to play games with the return value from the first part of the command (since the second part, the diff, trashes the return value from the earlier part).
Hello Mike. Have you seen the new Wine Weekly News issue?
They introduced a “patch watcher”, build bot that tests non-committed patches floating on the wine mailing list. Maybe something of interest related to your work on FATE.
http://www.winehq.org/site?issue=351#The%20PatchWatcher
@Mike: Indeed. Maybe something like
make tests/data/lavf.regression > /dev/null 2> /dev/null && diff -u -w tests/libav.regression.ref tests/data/lavf.regression
would work?
Thanks for the tip, Martin. I blogged about The Patch Watcher separately.