While I have thought a lot about the problem of testing perceptual audio decoders in FFmpeg via FATE, I have put little thought into testing perceptual video decoders. I surmise that my 1-off testing method for perceptual audio decoders is probably not tractable for video decoders. I suspect that, while rounding errors also cause off-by-1 errors in video decoders, compounded errors during multiple frames could stretch the threshold. There’s also the minor matter that it would require a much larger amount of space to store raw video data vs. the audio data required for the perceptual audio tests.

Testing video decoders isn’t such a big deal, though. The FFmpeg devs have had this problem solved via the ‘make test’ regression suite for a long time. There are a few flags to tack onto a command line which will invoke a bit exact inverse discrete cosine transform (IDCT). The trade-off is that optimized IDCT code paths are not exercised.

I’m thinking of a 2-part solution:

1. As a first pass, enter video decoding tests that leverage the bit exact IDCT.
2. As a second pass, craft special test cases that decode one frame of video and leverage the default, optimized IDCT. Test the result using the 1-off method.

See Also:

• Perceptual Audio Test Plan