Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Archives:

Granular Regression Tests

August 25th, 2008 by Multimedia Mike

I have been studying the FFmpeg regressions test infrastructure ever so carefully so I can decide the best way to convert all of its functionality and test coverage into the FATE system as individual test specs. I have some ideas but I never know when I will have both the time and the motivation to implement them. In the interim, I am retiring the master ‘make test’ full regression test spec. In studying the regression architecture, it’s immediately obvious that ‘make test’ invokes these 3 sub-regression suites, which each have their own test specs now:

That should help until I break down the tests even further. Oh, and I also added a test spec for ‘ffmpeg -h’. Hey, FFmpeg got into a state once where it segfaulted on help. We don’t want to let that go uncaught again.

Posted in FATE Server | 5 Comments »

5 Responses

  1. Vitor Says:

    As someone suggested in the ML, doing something like

    make tests/data/lavf.regression > /dev/null 2> /dev/null; diff -u -w tests/libav.regression.ref tests/data/lavf.regression

    And testing the stout (fail if there is any) might allow FATE to print the diff when the test fail.

  2. Multimedia Mike Says:

    Caveat: You would also need to play games with the return value from the first part of the command (since the second part, the diff, trashes the return value from the earlier part).

  3. Martin Lindhe Says:

    Hello Mike. Have you seen the new Wine Weekly News issue?
    They introduced a “patch watcher”, build bot that tests non-committed patches floating on the wine mailing list. Maybe something of interest related to your work on FATE.

    http://www.winehq.org/site?issue=351#The%20PatchWatcher

  4. Vitor Says:

    @Mike: Indeed. Maybe something like

    make tests/data/lavf.regression > /dev/null 2> /dev/null && diff -u -w tests/libav.regression.ref tests/data/lavf.regression

    would work?

  5. Multimedia Mike Says:

    Thanks for the tip, Martin. I blogged about The Patch Watcher separately.