Generating coverage reports

With Zephyr, you can generate code coverage reports to analyze which parts of the code are covered by a given test or application.

You can do this in two ways:

  • In a real embedded target or QEMU, using Zephyr’s gcov integration

  • Directly in your host computer, by compiling your application targeting the POSIX architecture

Test coverage reports in embedded devices or QEMU

Overview

GCC GCOV is a test coverage program used together with the GCC compiler to analyze and create test coverage reports for your programs, helping you create more efficient, faster running code and discovering untested code paths

In Zephyr, gcov collects coverage profiling data in RAM (and not to a file system) while your application is running. Support for gcov collection and reporting is limited by available RAM size and so is currently enabled only for QEMU emulation of embedded targets.

Details

There are 2 parts to enable this feature. The first is to enable the coverage for the device and the second to enable in the test application. As explained earlier the code coverage with gcov is a function of RAM available. Therefore ensure that the device has enough RAM when enabling the coverage for it. For example a small device like frdm_k64f can run a simple test application but the more complex test cases which consume more RAM will crash when coverage is enabled.

To enable the device for coverage, select CONFIG_HAS_COVERAGE_SUPPORT in the Kconfig.board file.

To report the coverage for the particular test application set CONFIG_COVERAGE.

Steps to generate code coverage reports

These steps will produce an HTML coverage report for a single application.

  1. Build the code with CONFIG_COVERAGE=y.

    west build -b mps2/an385 -- -DCONFIG_COVERAGE=y -DCONFIG_COVERAGE_DUMP=y

  2. Capture the emulator output into a log file. You may need to terminate the emulator with Ctrl-A X for this to complete after the coverage dump has been printed:

    $ ninja -Cbuild run | tee log.log

  3. Generate the gcov .gcda and .gcno files from the log file that was saved:

    $ python3 scripts/gen_gcov_files.py -i log.log

  4. Find the gcov binary placed in the SDK. You will need to pass the path to the gcov binary for the appropriate architecture when you later invoke gcovr:

    $ find $ZEPHYR_SDK_INSTALL_DIR -iregex ".*gcov"

  5. Create an output directory for the reports:

    $ mkdir -p coverage-report

  6. Run gcovr to get the reports:

    $ gcovr -r $ZEPHYR_BASE . --html -o coverage-report/coverage.html --html-details --gcov-executable <gcov_path_in_SDK>

Coverage reports using the POSIX architecture

When compiling for the POSIX architecture, you utilize your host native tooling to build a native executable which contains your application, the Zephyr OS, and some basic HW emulation.

That means you can use the same tools you would while developing any other desktop application.

To build your application with gcc’s gcov, simply set CONFIG_COVERAGE before compiling it. When you run your application, gcov coverage data will be dumped into the respective gcda and gcno files. You may postprocess these with your preferred tools. For example:

west build -b native_sim samples/hello_world -- -DCONFIG_COVERAGE=y

$ ./build/zephyr/zephyr.exe
# Press Ctrl+C to exit
$ lcov --capture --directory ./ --output-file lcov.info -q --rc lcov_branch_coverage=1
$ genhtml lcov.info --output-directory lcov_html -q --ignore-errors source --branch-coverage --highlight --legend

Note

You need a recent version of lcov (at least 1.14) with support for intermediate text format. Such packages exist in recent Linux distributions.

Alternatively, you can use gcovr (at least version 4.2).

Coverage reports using Twister

Zephyr’s twister script can automatically generate a coverage report from the tests which were executed. You just need to invoke it with the --coverage command line option.

For example, you may invoke:

$ twister --coverage -p qemu_x86 -T tests/kernel

or:

$ twister --coverage -p native_sim -T tests/bluetooth

which will produce twister-out/coverage/index.html report as well as the coverage data collected by gcovr tool in twister-out/coverage.json.

Other reports might be chosen with --coverage-tool and --coverage-formats command line options.

To generate code coverage report including Zephyr sources as well as your application code outside of Zephyr repository (see Application Types) call Twister from your project directory with --coverage-basedir $ZEPHYR_BASE command line option, for example:

$ $ZEPHYR_BASE/scripts/twister --coverage -p native_sim --coverage-basedir $ZEPHYR_BASE -T your_project_dir

Note

By default, Twister calls gcovr tool which filters source files assuming real paths are everywhere with all symlinks resolved, so when your development environment has directories with symlinks then, to avoid incomplete gcovr reports, either your ZEPHYR_BASE should contain a real path, or lcov tool used instead of gcovr with additional Twister command line option --coverage-tool lcov.

The process differs for unit tests, which are built with the host toolchain and require a different board:

$ twister --coverage -p unit_testing -T tests/unit

which produces a report in the same location as non-unit testing.

Per-test coverage matrix

By default coverage is aggregated per test scenario. To attribute coverage down to individual Ztest test cases – for example to answer “which tests exercised line X of foo.c” – pass --coverage-per-test:

$ twister -p mps2/an385 -T tests/kernel --coverage --coverage-tool lcov \
    --coverage-per-test

This enables CONFIG_ZTEST_COVERAGE_PER_TEST, which resets the gcov counters before each test case and dumps an isolated, test-tagged coverage artifact after it. On platforms that support semihosting (ARM, RISC-V and Xtensa targets under QEMU, such as mps2/an385) the per-test data is written straight to the host filesystem, avoiding the large amount of serial console traffic that per-test dumping would otherwise produce; other platforms fall back to the serial console transport.

In addition to the usual aggregated coverage report, this produces:

  • one <scenario>.<test>.info tracefile per test under each build’s coverage/tests/ directory, and

  • twister-out/coverage/test_matrix.json, a machine-readable matrix with a by_line view ({file: {line: [tests]}}) and a by_test view ({test: {file: [lines]}}).

Per-test attribution is carried by the matrix (and its dashboard), not by the aggregated lcov report: each instance’s coverage is collapsed to a single tracefile before aggregation, so the end-of-run reporting stays proportional to the number of instances rather than the total number of test cases.

Note

--coverage-per-test requires the lcov coverage tool, because the matrix relies on lcov’s per-test TN tracefile records; gcovr has no equivalent. It also implies --coverage.

Visualizing the matrix

test_matrix.json can be turned into a self-contained, interactive HTML dashboard with the standalone scripts/gen_test_matrix_dashboard.py script (it has no dependency on Twister and can be run on any previously generated matrix):

$ scripts/gen_test_matrix_dashboard.py -i twister-out/coverage/test_matrix.json

This writes twister-out/coverage/test_matrix.html, which lists – per test – how many files and lines it covers and how many lines it covers uniquely (that no other test reaches, useful for spotting redundant or load-bearing tests), and lets you drill into the files a test covers or look up which tests cover a given file and line.

Using different toolchains

Twister looks at the environment variable ZEPHYR_TOOLCHAIN_VARIANT to check which gcov tool to use by default. The following are used as the default for the Twister --gcov-tool argument default:

Toolchain

--gcov-tool value

host

gcov

llvm

llvm-cov gcov

zephyr

gcov