{"id":2654,"date":"2010-08-09T22:22:23","date_gmt":"2010-08-10T05:22:23","guid":{"rendered":"http:\/\/multimedia.cx\/eggs\/?p=2654"},"modified":"2010-08-09T22:22:23","modified_gmt":"2010-08-10T05:22:23","slug":"new-fate-test-coverage-system","status":"publish","type":"post","link":"https:\/\/multimedia.cx\/eggs\/new-fate-test-coverage-system\/","title":{"rendered":"New FATE Test Coverage System"},"content":{"rendered":"

I’ve been feeling a bit scattered for the last week since I was fired from my volunteer position as the FFmpeg QA manager<\/a>, wondering if there is anything I should attempt to do with the project. It can’t be denied that the new system<\/a> is working well. But one area I’ve wondered about is test coverage.<\/p>\n

Under my old regime I tracked test coverage as a wiki page<\/a> which was a highly flawed method– tedious and error-prone. There are those 2 adjectives again– tedious and error-prone; whenever I see those, I search for automation methods. I think that might be more plausible thanks to the new FATE’s tighter integration with the FFmpeg build system.<\/p>\n

I don’t think anyone is working on this problem so I wanted to toss out a brainstorm:<\/p>\n

    \n
  1. First, run ‘ffmpeg -formats’, ‘ffmpeg -codecs’, etc. and parse the output to collect a list of all the features (full list: -formats, -codecs, -bsfs, -protocols, -filters, -pix_fmts). Transform these lists into a standardized list of features, e.g., \"DEVSD ffvhuff Huffyuv FFmpeg variant\"<\/code> represents features ‘decode-video-ffvhuff’, ‘encode-video-ffvhuff’, ‘ffvhuff-horizband’, and ‘ffvhuff-dr1’.<\/li>\n
  2. Next, tag each individual test spec with the features that it exercises. E.g., test ‘fate-vqa-cc’ exercises features ‘demux-wsvqa’, ‘decode-video-vqavideo’, and ‘decode-audio-adpcm_ima_ws’.<\/li>\n
  3. Finally, compare the data from parts 1 and 2. Print a list of all the features that are not exercised in FATE.<\/li>\n<\/ol>\n

    I think a lot of this could be implemented at the GNU make level. Then again, I’m no expert on GNU make syntax so I may be overestimating its capabilities. Or there might be simpler ways to automatically track test coverage stats based on the improved testing infrastructure.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Accounting for FFmpeg’s test coverage in an automated manner<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[101],"tags":[],"class_list":["post-2654","post","type-post","status-publish","format-standard","hentry","category-fate-server"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t