Compatibility Testing With FATE

Once upon a time, I made a valiant effort to create a compatibility page for FFmpeg. The goal was simple– document how to use FFmpeg to generate media files that would play back in the most common media players of the day. At the time, this included Windows Media Player and Apple QuickTime Player. The exercise was naturally obsolete quite quickly due to the fact that I used some random SVN revision (actually, this is so old that I think we were still using CVS at the time).

I still think this is a worthwhile idea. I now realize that the idea can be improved by FATE. Create a series of test specs that map to a certain encoding combination known to work on a popular media player (or even a popular media device– this could be used for testing, e.g., iPod profiles) and document the command lines on the website. When one of the command lines breaks for all configurations, it will be worth investigating to learn why and whether the website example needs to be updated, or whether FFmpeg needs to be fixed.

Testing process:

  • Develop command line that should produce content playable for popular software player or device
  • Transcode sample media with command line
  • Test media on target player to make sure it plays
  • Enter FATE test spec
  • Document command line on compatibility page

Such a FATE test will be contingent upon a built-in file checksumming feature as I have outlined for a future revision.