Last year, I delved into code coverage tools and their usage with FFmpeg. I learned about using GNU gcov, which is powerful but pretty raw about the details it provides to you. I wrote a script to help interpret its output and later found another script called gcovr to do the same, only much better.
I later found another tool called lcov which is absolutely amazing for understanding code coverage of your software. I’ve been meaning to use it to further FATE test coverage for the multimedia projects.
Click for larger image
Basic Instructions
Install the lcov tool, of course. In Ubuntu, 'apt-get install lcov' will do the trick.
Build the project with code coverage support, i.e.,
./configure --enable-gpl --samples=/path/to/fate/samples \ --extra-cflags="-fprofile-arcs -ftest-coverage" \ --extra-ldflags="-fprofile-arcs -ftest-coverage" make
Clear the coverage data:
lcov --directory . --zerocounters
Run the software (in this case, the FATE test suite):
make fate
Let lcov work its magic:
lcov --directory . --capture --output-file coverage.info mkdir html-output genhtml -o html-output coverage.info
At this point, you can aim your web browser at html-output/index.html to learn everything you could possibly want to know about code coverage of the test suite. You can sort various columns in order to see which modules have the least code coverage. You can drill into individual source files and see highlighted markup demonstrating which lines have been executed.
As you can see from the screenshot above, FFmpeg / Libav are not anywhere close to full coverage. But lcov provides an exquisite roadmap.
Can we see the output?
Well, the reason I posted the specific, detailed instructions was so that anyone else could easily replicate them. Any results I could post would be quickly obsoleted.
However, it may be worth setting up a quick automated job to generate these pages and post them to a public place. I’ll try to get that going.
am i right to guess that the reason the branches column is low is because most of the ‘branches’ in ffmpeg are ‘if something_unexpected_found_with_fuzzing then return -1’ ?
is there a way to set it to ignore the ‘return -1’ code ? i wonder what the graph would look like then?
I think there is enough work to do on line coverage before we have to go into branch coverage…
Anyway Mike: The proper way is to add –enable-coverage to configure, and a covreport target that generates the coverage data (might be a bit tricky to get 100% right) and a reasonable way to clean the coverage way (can’t think of one right now, except and extra target).
@Reimar: I like the idea of –enable-coverage. I think I proposed that to ffmpeg-devel last year when I started with gcov. That could take the place of the extra CFLAGS and LDFLAGS.
Meanwhile, clearing the profiling is a matter of deleting all .gcno and .gcda files throughout the directory structure.
My problem is that I don’t know how I would describe to make a dependency of the form “before doing the next step (e.g. running fate) you have to delete all these files”.
Oh well, I decided to send a not-quite-perfect patch.