Sam Hocevar — who I last saw with his groundbreaking counter-CAPTCHA research named PWNtcha — has been working on a fuzzing program called zzuf. Fuzzing is when you create random, implicitly fraudulent data and feed it as input into a program which is supposed to process the data and make sure that the program fails gracefully when chewing on the pathological data. I learned about this technique some time ago and have always wanted to launch it against our body of open source multimedia software. Sam has done just this. None of the multimedia players tested fared particularly well.
FFmpeg devs have already corrected a number of the issues uncovered by this cursory fuzz test. According to the program’s description, zzuf intercepts file operations and changes random bits in the program’s input. This is a great start to the process. However, it will likely only attack the demuxer layer of the various multimedia applications. A deeper method would be to fuzz, e.g., individual chunks of codec data on the way from the demuxer to its corresponding decoder. This could probably be rigged up in FFmpeg fairly easily (just be sure to make the behavior deterministic so any bugs can be reproduced).