When I started investigating code coverage tools to analyze FFmpeg, I knew there had to be an easier way to do what I was trying to do (obtain code coverage statistics on a macro level for the entire project). I was hoping there was a way to ask the GNU gcov tool to do this directly. John K informed me in the comments of a tool called gcovr. Like my tool from the previous post, gcovr is a Python script that aggregates data collected by gcov. gcovr proves to be a little more competent than my tool.
Results
Here is the spreadsheet of results, reflecting FATE code coverage as of this writing. All FFmpeg source files are on the same sheet this time, including header files, sorted by percent covered (ascending), then total lines (descending).
Methodology
I wasn’t easily able to work with the default output from the gcovr tool. So I modified it into a tool called gcovr-csv which creates data that spreadsheets can digest more easily.
- Build FFmpeg using the
'-fprofile-arcs -ftest-coverage'in both the extra cflags and extra ldflags configuration options 'make''make fate'- From build directory:
'gcovr-csv > output.csv' - Massage the data a bit, deleting information about system header files (assuming you don’t care how much of /usr/include/stdlib.h is covered — 66%, BTW)
Leftovers
I became aware of some spreadsheet limitations thanks to this tool:
- OpenOffice can’t process percent values correctly– it imports the percent data from the CSV file but sorts it alphabetically rather than numerically.
- Google Spreadsheet expects CSV to really be comma-delimited– forget about any other delimiters. Also, line length is an issue which is why I needed my tool to omit the uncovered line number ranges, which it does in its default state.
Looks like high time I finally start adding subtitle tests.
And figure out a way to generate an encrypted MXF file so AES is tested.
How about adding a –enable-coverage to configure?
Btw. no need to do “make” if you do “make fate”, it will compile FFmpeg before trying to run the tests :-)
Just wondering if maybe sorting by the number of untested lines is more suitable. A file of 20 lines that is completely uncovered is better than a 1000-line file that is 5% covered.
@Reimar: All these years and I’m still a build system n00b. It would be very useful for someone to add a
--enable-coverageoption to the build system since we are getting serious about tracking this.@Vitor: My thinking was for the largest files with the most uncovered code to float to the top of the list.
Thanks for this great blog post! Where can I get gcovr-csv? It doesn’t seem to be included in the ffmpeg distribution. Can you post it somewhere?
@Curious Visitor: Frustratingly, I can’t find that script right now. That’s why I like to post all my work in public– so I can find it again!
I will try to re-derive it and get it posted.
Scratch that; I found it. I’ll get it posted soon (or email it to you).