Skip to content

Get code coverage metrics

Jump to bottom
Günter Wirth edited this page May 12, 2014 · 23 revisions

It is not all the same

Talking about coverage the most important question is: what coverage your are talking about? There are different kinds of coverage measurement and each tool has a slightly different way doing it and displaying the results. If you are looking for a very good introduction read Code Coverage Analysis on the Bullseye page.

SonarQube API provides the possibility to set the covered lines (setHits) and covered conditions (setConditions). All other metrics are derived from these settings:

  • Line coverage: On a given line of code, Line coverage simply answers the following question: Has this line of code been executed during the execution of the unit tests?
  • Conditions by line / Covered conditions by line: These are the number of conditions by line and number of covered conditions by line.
  • Branch coverage: On each line of code containing some boolean expressions, the branch coverage simply answers the following question: 'Has each boolean expression been evaluated both to true and false?'. This is the density of possible branches in flow control structures that have been followed during unit tests execution.
  • Coverage: It is a mix of Line coverage and Branch coverage. Its goal is to provide an even more accurate answer to the following question: How much of the source code has been covered by the unit tests?

For more details and exact calculation see Metric Definitions on the SonarQube page.

Setup the C++ Community Plugin

To collect C++ test coverage metrics with C++ Community Plugin, you have to:

  1. Create a coverage report with an external tool (or many of them). For some examples how to use your favorite coverage tracer see Coverage tracers.
  2. Configure your project to point to the report and rerun the analysis.

Three types of test coverage are supported: Unit, Integration and Overall coverage. Use one of the following configuration properties dependent of the coverage type (see also Supported configuration properties for details):

  • sonar.cxx.coverage.reportPath for unit test coverage
  • sonar.cxx.coverage.itReportPath for integration test coverage
  • sonar.cxx.coverage.overallReportPath for overall test coverage

The C++ Community plugin accepts three different formats for the test coverage reports, which will be recognized automatically:

Notes for BullseyeCoverage users:

SonarQube <3.2 provides metrics for line coverage and branch coverage. Bullseye users have function and branch/decision coverage instead. The C++ Community plugin converts the branch/decision coverage directly into branch coverage however line coverage is far more complex and cannot be correlated directly into function coverage.

Line coverage imported from a bullseye report means than function coverage + line branch coverage (this second occurs since SonarQube will not display branch coverage if there isn't a line it associated with it). This means also that overall coverage will be affected in SonarQube and cannot be compared directly to Bullseye results. The following pictures illustrates this for a small example project.

Coverage visualisation for Bullseye users

Notes for Visual Studio coverage users

TBD guwirth

Prerequisites

Visual Studio provides block or line coverage:

  • block coverage: A block is a piece of code with exactly one entry and exit point. If the program’s control flow passes through a block during a test run, that block is counted as covered. The number of times the block is used has no effect on the result. The block coverage is only on function level available.
  • line coverage: Counts the number of executed lines of code. A line could be covered (covered="yes"), not covered (covered="no") or partial covered (covered="partial").

Mapping this to the SonarQube model this result in:

  • Condition coverage within the code is not available. Block coverage will be set for each function in the first line.
  • SonarQube does not support partial covered lines. Partial covered lines are marked as covered.

See below an example for coverage result calling with myFunc(0, 1):

#include "stdafx.h"

void myFunc(int a, int b)
{
	int x = 0;
	int y = 0;

	if( a ) {
		x = 1;
		y = 1;
	}

	if( b )
	{
		x = 2;
		y = 2;
	}

	if( a &&
		b
		) {
			x = 3;
			y = 3;
	}
}

Resulting Visual Studio XML coverage file:

<?xml version="1.0" encoding="UTF-8" ?>
<results>
  <modules>
    <module name="unittest1.dll" path="unittest1.dll" id="93AA4138F7AFB24C91DC9614B700B83C07000000" block_coverage="72.73" line_coverage="68.75" blocks_covered="8" blocks_not_covered="3" lines_covered="11" lines_partially_covered="1" lines_not_covered="4">
      <functions>
        <function id="77200" name="myFunc" type_name="" block_coverage="62.50" line_coverage="61.54" blocks_covered="5" blocks_not_covered="3" lines_covered="8" lines_partially_covered="1" lines_not_covered="4">
          <ranges>
            <range source_id="0" covered="yes" start_line="4" start_column="0" end_line="4" end_column="0" />
            <range source_id="0" covered="yes" start_line="5" start_column="0" end_line="5" end_column="0" />
            <range source_id="0" covered="yes" start_line="6" start_column="0" end_line="6" end_column="0" />
            <range source_id="0" covered="yes" start_line="8" start_column="0" end_line="8" end_column="0" />
            <range source_id="0" covered="no" start_line="9" start_column="0" end_line="9" end_column="0" />
            <range source_id="0" covered="no" start_line="10" start_column="0" end_line="10" end_column="0" />
            <range source_id="0" covered="yes" start_line="13" start_column="0" end_line="13" end_column="0" />
            <range source_id="0" covered="yes" start_line="15" start_column="0" end_line="15" end_column="0" />
            <range source_id="0" covered="yes" start_line="16" start_column="0" end_line="16" end_column="0" />
            <range source_id="0" covered="partial" start_line="21" start_column="0" end_line="21" end_column="0" />
            <range source_id="0" covered="no" start_line="22" start_column="0" end_line="22" end_column="0" />
            <range source_id="0" covered="no" start_line="23" start_column="0" end_line="23" end_column="0" />
            <range source_id="0" covered="yes" start_line="25" start_column="0" end_line="25" end_column="0" />
          </ranges>
        </function>
      </functions>
      <source_files>
        <source_file id="0" path="x:\coveragetest\source.cpp" checksum_type="MD5" checksum="8A4EC27BE0FE2BCE77401F836581836D">
        </source_file>
      </source_files>
    </module>
  </modules>
</results>

            Wiki Home             Issue Tracker             Releases             Contributors             License

Clone this wiki locally