I finally got the tool outlined in this post operational and doing the right thing. This allowed me to automatically test 136 H.264 draft conformance vectors and then add them to the FATE test suite if they are presently working. Even more useful, I can rerun the same tool at any time and it will skip over any files that already have corresponding tests in the database and re-test samples that weren’t known to work before.
Out of those 136 tests, 51 of them presently work in FFmpeg. “Work” is defined rather strictly as FFmpeg decoding data that matches precisely, bit for bit, to the reference output provided. So at the very least, the FATE Server will be tracking regressions on this set from this point on.
At the time of this writing, the configuration involving gcc 2.95.3 on x86_32 is acting up. I am not sure why, but when the test is done and the results are ready to be logged, the script always stalls while talking to the MySQL server and eventually makes the script bail out. Now that I think about it, I am hypothesizing that perhaps the fact that 2.95.3 compiles FFmpeg with an absolutely inordinate number of warnings (880 Kbytes for 2.95.3 vs. 32 Kbytes for 4.2.2) might be causing trouble. This is a very repeatable failure and does not occur with any other configurations. I don’t blame this on FFmpeg warnings; the FATE tools should be resilient enough to deal with this.
Another curious but minor artifact in the database– there are brief windows in time when I can refresh the main FATE page and see a particular build configuration with a “Tests passed” stat of, say, 16/17. Wow, weird. But when I refresh again immediately, I see the full 55/55 tests passed that I expect for the current state of the database. The problem is pretty straightforward– one of the build machines is in the middle of inserting its results when I happen to refresh the page. For the first time in any database project, I am entertaining the idea of using transactions. Per my reading of the documentation, however, I am not sure if my MySQL installation supports transactions (it requires a certain configuration in the underlying storage tables). Plus, I’m not sure if it’s all that big of a deal, especially since it will stall the database for all other uses in the meantime. Perhaps I’ll just add a static note advising users to reload the page since strange results are probably transient.
Hi,
Maybe you’ve already saw it, but this post about mingw cross-compiling may be useful to FATE:
http://article.gmane.org/gmane.comp.video.ffmpeg.user/14852
Thanks. I’m not on ffmpeg-user so I didn’t catch that. I’m almost at the point where I am comfortable expanding this operation to more machines and architectures: x86_64/Linux; Mac OS X; Sun/Sparc; even Win32.
Don’t forget to add the VC1 reference test also. Kostya wrote a parser for those files IIRC. And when looking at the output of a failed test can’t you include the commandline? It would be nice to have.
And can you show the warning count in the stderr files from the compilation process?
The warning count is a feature on my TODO list; it shouldn’t be too hard to deploy.
Display the command line… ahh, I was about to say, “it’s already there” but I guess it’s not. Yeah, it should be. I’ll add that. A bunch of tests are failing right now because of a small mistake I made in the test script but I should have it fixed soon.
Thanks for the tip on VC-1 conformance. I’m sure I have those samples laying around somewhere. We’re going to make these build/test machines earn their keep.