Coverage testing with CMake and GCov

This took me 3 days of on and off work to figure out. So I decide to write it down in case I need it again. And I want this to be more widely known.

CMake is the most popular build system (generator) for C++. While test coverage is an important tool to find where bugs can hide. Testing usually tries to exercise all features and make sure every edge case is covered. But how can you sure that's the case? Coverage records where code execution reaches and tells you which line has been executed when running. This is standard tool for Go and other languages. In C and C++. We have GCov, yet build system level support seems to be lacking. Turns out CMake supports coverage testing out of the box. It's just that it's not used much nor documented well.

The answer is hidden in a CMake forum post[1] from 2019. Turns out you can just add -fprofile-arcs -ftest-coverage and let CTest generate a report.

Given you already have tests setup and integrated with CTest. That'll likely be the case if you are using the big name test frameworks like GTest, Catch2 and doctest.

cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG="-g -fprofile-arcs -ftest-coverage" \
    -DCMAKE_C_FLAGS_DEBUG="-g -fprofile-arcs -ftest-coverage"
make -j
ctest -T Test -T Coverage

Now CTest should output the test coverage report. For me running on drogon[2] nets the following log:

❯ ctest -T Test -T Coverage
<..test status..>

100% tests passed, 0 tests failed out of 75

Total Test time (real) =   7.87 sec
Performing coverage
   Processing coverage (each . represents one file):
    .................................................. processed: 50 out of 149
    .................................................. processed: 100 out of 149
    .................................................
   Accumulating results (each . represents one file):
    .................................................. processed: 50 out of 283
    .................................................. processed: 100 out of 283
    .................................................. processed: 150 out of 283
    .................................................. processed: 200 out of 283
    .................................................. processed: 250 out of 283
    .................................
        Covered LOC:         5812
        Not covered LOC:     18403
        Total LOC:           24215
        Percentage Coverage: 24.00%

I guess the coverage is poor because drogon only runs the basic unit tests over CTest. The heavy, integration tests ran trhough a shell script or manually. Well, CTest has it's limitations. To run drogon's integration tests I \run the following commands.

cd build/lib/test/
./integration_test_server

# (in another terminal)
cd build/lib/test/
./integration_test_client

Then I can trick CTest into collecting the coverage data.

❯ ctest -T Coverage
   Site: nina
   Build name: Linux-c++
Performing coverage
   Processing coverage (each . represents one file):
    .................................................. processed: 50 out of 179
    .................................................. processed: 100 out of 179
    .................................................. processed: 150 out of 179
    .............................
   Accumulating results (each . represents one file):
    .................................................. processed: 50 out of 339
    .................................................. processed: 100 out of 339
    .................................................. processed: 150 out of 339
    .................................................. processed: 200 out of 339
    .................................................. processed: 250 out of 339
    .................................................. processed: 300 out of 339
    .......................................
        Covered LOC:         10508
        Not covered LOC:     16281
        Total LOC:           26789
        Percentage Coverage: 39.23%

Dang that coverage is poor. After looking into it. A large chunk that 60% seems to be just network/malformed request handling. Which we can't really test. And DB functions that is covered by the DB tests (not included in the main integration test). I bet adding the DB tests would bring coverage up to 60~70%. Which would be ok.

For more detailed view you can use your IDE's viewer. For VSCode I found the GCov Viewer plugin[3] be quite good. After running the integration test. Hit Ctrl-Shift-P and type "gcov load" to load the report (from gcov, not CTest). Then Ctrl-Shift-P and type "gcov show" to show the result.

GCov report visualized in VSCode

Nice!

Troubleshooting

If you ever end up with a cryptic error message Binary directory is not set. No coverage checking will be performed.. You need to add include(CTest) at the end of your CMakeLists.txt. That will tell CMake to collect information about your environment and setup coverage.

Author's profile
Martin Chang
Systems software, HPC, GPGPU and AI. I mostly write stupid C++ code. Sometimes does AI research. Chronic VRChat addict

I run TLGS, a major search engine on Gemini. Used by Buran by default.


  • marty1885 \at protonmail.com
  • GPG: 76D1 193D 93E9 6444
  • Jami: a72b62ac04a958ca57739247aa1ed4fe0d11d2df