Deterministic Source Paths Breaks Coverage - VS Coverage format

[iamwyza]

Description

If deterministic source paths are enabled during the builds, the scanner will fail to find the code coverage for the code. (The coverage reports will be indexed, but it will say 0 files matched). This happens by default with the Microsoft.SourceLink package.

Repro steps

Create a dotnet core project with 1 class and 1 unit test project, add the Microsoft.SourceLink package, and then run the sonar scanner against it. It will fail to produce code coverage.

Known workarounds

Turn deterministic source paths off.

Related information

• see SourceLink breaks SonarQube Code Coverage dotnet/sourcelink#615
• see Need to return SourceRoot Items to for referenced projects dotnet/roslyn#43476
• C#/VB.NET Plugins Version - 8.3.1
• Visual Studio Version - N/A
• MSBuild / dotnet version - Core 3.1
• If running through the Scanner for MSBuild, its version - dotnet-scanner 4.8
• Operating System - Win 10.

[andrei-epure-sonarsource]

Also see

• DeterministicSourcePaths can break building if source control information not available dotnet/roslyn#37379
• InstrumentationHelper.HasPdb always returns false for deterministic source paths coverlet-coverage/coverlet#363

[andrei-epure-sonarsource]

Hi @iamwyza

I tried to come up with a reproducer and created as you said a project referencing Microsoft.SourceLink.GitHub - https://github.com/andrei-epure-sonarsource/TestSourceLink .

And it seems code coverage works fine https://sonarcloud.io/dashboard?id=DeterministicSourcePath .

• C# Plugin 8.10 (latest)
• dotnet SDK 3.1.301
• sonar-scanner-msbuild-4.8.0.12008-netcoreapp2.0

The commands I've used:

dotnet "C:\SonarQube\sonar-scanner-msbuild-4.8.0.12008-netcoreapp2.0\SonarScanner.MSBuild.dll" begin /k:"DeterministicSourcePath" /d:sonar.verbose=true /d:sonar.login=<TOKEN> /d:sonar.host.url="https://sonarcloud.io" /o:"andrei-epure-sonarsource-github"  /d:sonar.ws.timeout=300 /d:sonar.cs.opencover.reportsPaths="opencover.xml"

dotnet build .\ClassLibrary3.sln /t:Rebuild /maxcpucount /nr:false /p:DeployExtension=false /warnaserror:AD0001 /clp:Summary

OpenCover.Console.exe -output:"opencover.xml" -register:administrator -target:"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -targetargs:".\UnitTestProject1\bin\Debug\netcoreapp3.1\UnitTestProject1.dll"

dotnet "C:\SonarQube\sonar-scanner-msbuild-4.8.0.12008-netcoreapp2.0\SonarScanner.MSBuild.dll" end /d:sonar.login=<TOKEN>

What code coverage tool have you used?
Could you update to the latest version of SonarCSharp and try to reproduce?

[iamwyza]

I'm using dotnet test, so it'd be the built-in one for dotnet (I think that's just vstest under the covers).

dotnet test somesolution.sln --no-restore --no-build -c Debug --logger trx --results-directory SomeDir\TestResults --collect:"Code Coverage" -s sourceDir\tests\CodeCoverage.runsettings /m:1

A segment of that output would look like:

Test run for SomeProject.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.6.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Microsoft (R) Coverage Collection Tool Version 16.0.30319.3002


Copyright (c) Microsoft Corporation.  All rights reserved.




Results File: c:\SomeDir\TestResults\VssAdministrator-00000_2020-07-01_00_00_00.trx

Attachments:
  c:\SomeDir\TestResults\3abe04cb-ae84-4962-ad8b-50f2b33c4baf\VssAdministrator_00000_2020-07-01.00_00_00.coverage
Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2.6735 Seconds```

[costin-zaharia-sonarsource]

I've added a small project on which we can reproduce this issue: https://github.com/costin-zaharia-sonarsource/Repro3362

Steps: https://github.com/costin-zaharia-sonarsource/Repro3362#steps

[andrei-epure-sonarsource]

Initial investigation

I followed the steps provided by @costin-zaharia-sonarsource in his reproducer.

With DeterministicSourcePaths

Run with dotnet build --no-incremental -p:DeterministicSourcePaths=true

in filename.coveragexml, under module name="repro3362.dll

    <source_file id="0" path="/_/Repro3362/Foo.cs" checksum_type="unknown">

When running in debug mode, we can see the below line (note that it's C:\_\ , not /_/ !!)

DEBUG: Skipping file with path 'C:\_\Repro3362\Foo.cs' because it is not indexed or does not have the supported language.

Unfortunately, in the statistics at the end, we don't mention skipped files (the test file is correctly detected, because it has the full path in the coverage files; only main files seem to have the path mangled).

INFO: Coverage Report Statistics: 1 files, 0 main files, 0 main files with coverage, 1 test files, 0 project excluded files, 0 other language files.

Without DeterministicSourcePaths

in filename.coveragexml, under module name="repro3362.dll

    <source_file id="0" path="C:\Workspace\external-projects\private_customer_reproducers\Repro3362\Repro3362\Foo.cs" checksum_type="unknown">

and coverage works as expected

With DeterministicSourcePaths and SourceRoot

Added the property mentioned in this comment and built with -p:DeterministicSourcePaths=true, but had the same behavior as in "With DeterministicSourcePaths".

Possible plan

Before moving forward:

• I need to test the behavior of DeterministicSourcePaths=true with dotcover and opencover
• however, as an initial baby step, I can only fix it for VS Coverage

Action items:

• count skipped files (due to indexing/language) and print them in the statistics
• add a new analysis parameter to be passed to the plugin via the scanner (e.g. sonar.cs.vscoveragexml.rootfolder or sonar.cs.coverage.rootfolder) and try to use that as a replacement for _ . However, the docs for SourceRoot say that it's

Item group that lists all source roots that the project source files reside under and their mapping to source control server URLs. This includes both source files under source control as well as source files in source packages.

I need to understand what that means, first.

• when seeing the \_\ or /_/ pattern inside a path in org/sonar/plugins/dotnet/tests/VisualStudioCoverageXmlReportParser.java, count them separately, and write a WARN message that DeterministicSourcePaths may be turned on, and that the special parameter must be passed for code coverage to work

[andrei-epure-sonarsource]

Actually, before any action, I need to read what coverlet did to support this coverlet-coverage/coverlet#863 and dotnet/roslyn#43476.

[andrei-epure-sonarsource]

SourceRoot

InitializeSourceRootMappedPaths - This target collects all SourceRoots from various sources.

Work done on Coverlet:

• some comments to get the context
  -- read this comment about possible PathMaps - there can be more than _ e.g. _1, _2
  -- initial idea for coverlet approach (same as mine) and how coverlet works in terms of source paths
  -- how to get the path mappings per referenced project
• this is where the actual support has been added: Add support for deterministic build for msbuild driver #796
• see Coverage.cs#L385 and GetSourceLinkUrl and CoverletSourceRootsMappingTest

Maybe not so relevant:

• Fix deterministic build+source link bug Fix deterministic build+source link bug coverlet-coverage/coverlet#895
• remove the hack for 3.1.300+ SDK Remove workaround for deterministic build for sdk ge 3.1.100 coverlet-coverage/coverlet#965

Other links

• how to enable SourceLink
• pre .NET Core SDK 3.1.300 needs a hack
• interesting comment on what could go wrong when using .NET Framework

[andrei-epure-sonarsource]

hi @iamwyza - did you have the chance to test the Scanner for MSBuild with deterministic source paths enabled for your project(s)?

[TimHess]

@andrei-epure-sonarsource I updated Steeltoe to use the latest package but I'm still seeing 0% coverage in SonarCloud (sample build)

The SonarScanner Context says C# Code Quality and Security 8.16.0.25740 (csharp)

I'd be happy to provide any additional info or try things out

[andrei-epure-sonarsource]

Hi @TimHess , thanks a lot for the feedback. We currently support Deterministic Builds with VS Coverage reports. I updated the release notes for clarity now.

I saw in your logs that you're using OpenCover reports. I've opened #3976 to close this gap. I added a note in the issue to ask you after the release that all works well.

[mrt181]

Hi, I am still facing this issue. See my report here: SourceLink breaks coverage report generation