"Code coverage" is a measure that indicates how much of our library code is exercised by our test suites. We measure code coverage using the Coverlet global tool and use ReportGenerator to visualize that data. Daily/regular reports are currently not available.
This report shows each library currently being tested with code coverage and provides statistics around the quality of the code coverage for the library. It also provides a line-by-line breakdown of what lines are being covered and what lines are not.
The code coverage report provides a percentage value per library of the number of source lines exercised by the tests. There is no hard and fast percentage that must be obtained per library, as every library is unique and comes with its own set of intricacies and constraints. While in some cases it's possible and reasonable to achieve 100% code coverage, this is rare. There are many valid reasons certain pieces of code won't be exercised in tests, e.g.:
- A code file is compiled into multiple projects, and only some of the code is used in each project.
- Code exists to handle rare race conditions too costly to simulate in normal conditions.
- Code exists to handle particular machine/OS configurations that are not used during code coverage runs.
Etc. What's important is that the right set of tests exist to ensure that the code is behaving properly and that regressions in functionality can be caught quickly, and code coverage metrics are a way to help guide us to that end.
Our default, somewhat-arbitrary initial goal for a library is 90% code coverage. That doesn't mean we're done with testing once a library hits 90%, nor does it mean we must keep going with a library until it hits 90%. We use this metric and the associated coverage information to help guide us towards the ideal for a given library.
(Note that we do not want to arbitrarily inflate our code coverage numbers. Tests must provide value in and of themselves and should not simply be written in a haphazard manner meant to execute more lines of code without providing real value.)
Issues are opened for a library when a cursory examination of its code coverage reveal that there are likely still some meaningful gaps that need to be addressed. We welcome contributions to our test suites to help address these gaps and close these issues. Many of these issues are marked as up-for-grabs.
An issue need not be addressed in its entirety. We happily accept contributions that improve our tests and work towards improving code coverage numbers even if they only incrementally improve the situation.
You can perform code coverage runs for the entire repository locally by adding the coverage switch (assuming that source and test assemblies are already built):
build -test -coverage
This runs the tests and generates the full code coverage report. The resulting index.htm file providing the results of the run should be available at:
artifacts\coverage\index.htm
You can also build and test with code coverage for a particular test project rather than for the whole repo with the /p:Coverage=true argument:
dotnet msbuild /t:BuildAndTest /p:Coverage=true
The results for this one library will then show up in the aforementioned index.htm file. For example, to build, test, and get code coverage results for the System.Diagnostics.Debug library, from the root of the repo one can do:
cd src\System.Diagnostics.Debug\tests\
dotnet msbuild /t:BuildAndTest /p:Coverage=true
And then once the run completes:
$(TestPath)\report\index.htm
Note: If you only want to measure the coverage of your local changes (that haven't been pushed to git), run:
dotnet msbuild /t:BuildAndTest /p:Coverage=true /p:CoverageSourceLink=false
Some of the libraries for which contracts and tests live in the corefx repo are actually fully or partially implemented in the core runtime library in another repo, e.g. the implementation that backs the System.Runtime contract is in System.Private.CoreLib.dll in either the coreclr or corert repo. Test projects for code that lives, fully or partially, in System.Private.CoreLib, should have the property TestRuntime set to true in order to obtain proper code coverage reports.
If the test project does not set the property TestRuntime to true and you want to collect code coverage that includes types in System.Private.CoreLib.dll add /p:TestRuntime=true to the coverage build command listed above.
If you want to get coverage report against a private build of System.Private.CoreLib follow the steps outlined at Testing with Private CoreClr Bits.
The build and test projects take care of copying assemblies and PDBs as needed for coverage runs. The resulting code coverage report should now also include details for System.Private.CoreLib.