[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2 #1726

kotsaris · 2025-01-01T11:23:36Z

Describe the bug
I recently upgraded to 6.0.3 and I noticed the pipelines failing. It seems that cobertura.xml comes out empty. I am using a runsettings file with exclude/include. I notice that if I remove these sections the cobertura gets populated.

That said, coverage works fine and cobertura gets entries on 6.0.2 of coverlet.collector nuget package.

To Reproduce
Check out https://github.com/kotsaris/repro-coverlet-empty-cobertura cd into ClassLibrary1 folder and run tests via

dotnet test --collect:"XPlat Code Coverage" --settings .\coverage.runsettings

Expected behavior
It should populate cobertura.xml with something.

Actual behavior
Cobertura.xml is empty.

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1735730899" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

Restore complete (0.3s)
  ClassLibrary1 succeeded (0.1s) → ClassLibrary1\bin\Debug\net9.0\ClassLibrary1.dll
  ClassLibrary1.Tests succeeded (0.0s) → ClassLibrary1.Tests\bin\Debug\net9.0\ClassLibrary1.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0+e341b939fe (64-bit .NET 9.0.0)
[xUnit.net 00:00:00.03]   Discovering: ClassLibrary1.Tests
[xUnit.net 00:00:00.05]   Discovered:  ClassLibrary1.Tests
[xUnit.net 00:00:00.06]   Starting:    ClassLibrary1.Tests
[xUnit.net 00:00:00.08]   Finished:    ClassLibrary1.Tests
  ClassLibrary1.Tests test succeeded (0.7s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 0.7s
Build succeeded in 1.2s

Workload updates are available. Run `dotnet workload list` for more information.

Attachments:
  C:\dev\kotsaris\repro-coverlet-empty-cobertura\ClassLibrary1\ClassLibrary1.Tests\TestResults\5e5adf4e-c517-478b-a81e-ccff2099c832\coverage.cobertura.xml

Configuration (please complete the following information):
Run Settings:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <Exclude>
                        [*]ClassLibrary1.Tests.*
                    </Exclude>
                    <Include>[*]ClassLibrary1*</Include>
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                    <SingleHit>false</SingleHit>
                    <UseSourceLink>false</UseSourceLink>
                    <IncludeTestAssembly>false</IncludeTestAssembly>
                    <SkipAutoProps>true</SkipAutoProps>
                    <DeterministicReport>false</DeterministicReport>
                    <ExcludeAssembliesWithoutSources>MissingAll,MissingAny,None</ExcludeAssembliesWithoutSources>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

Please provide more information on your .NET configuration:
* Which coverlet package and version was used? 6.0.3
* Which version of .NET is the code running on? 9
* What OS and version, and what distro if applicable? Windows 11
* What is the architecture (x64, x86, ARM, ARM64)? x64
* Do you know whether it is specific to that configuration? 6.0.2 does not exhibit this

Additional context
Downgrade to 6.0.2 and the problem goes away.

❗ Please also read Known Issues

sboulema · 2025-01-01T16:13:21Z

I am running into the same problem ☹️

daveMueller · 2025-01-02T01:00:09Z

Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed:
https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357

I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole Include and Exclude filter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli

(btw @kotsaris in your repro it is also sufficient if you only remove the Exclude filter from the runsettings file)

mu88 · 2025-01-06T11:06:37Z

I'm facing the same issue with coverlet.msbuild instead of coverlet.collector: when using v6.0.3 and running dotnet test <all the other things> /p:CoverletOutputFormat=opencover, the output file is empty. Downgrading the package to v6.0.2 resolves the issue.

argium · 2025-01-07T03:14:52Z

+1.

Version 6.0.3 produces a coverage.cobertura.xml without any results. Reverting to 6.0.2 without other changes resolves the issue.

JohnYoungers · 2025-01-08T20:32:48Z

Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed: https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357

I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole Include and Exclude filter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli

(btw @kotsaris in your repro it is also sufficient if you only remove the Exclude filter from the runsettings file)

would it make sense to add that condition back as a 6.0.4 patch? and address the filter logic in a minor bump?

daveMueller · 2025-01-08T23:54:11Z

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect. Just something like [module-names, include-filters, exclude-filters, expected-modules]. We already have the repro @kotsaris provided but some more references would be cool. We just want to make sure that we don't miss anything again that isn't covered by unit test.

@JohnYoungers not sure about a new release just for this, I need to discuss this with the other maintainers. There will be the possibility to consume the fix in our nightly once we have merged it.

argium · 2025-01-09T02:51:19Z

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

Here's what I've got:

          <Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> 
          <Include>*C.D*</Include>

The intention is to include assemblies such as:

A.C.D.E
B.C.D.E
A.C.D.E.F

and obviously exclude the corverlet assemblies. I honestly don't recall why the exclude looks the way it does other than to say that it took longer than I expected and I couldn't spend any more time on it. Without the exclude, coverlet assemblies were throwing off the code coverage percentage.

mu88 · 2025-01-09T08:40:44Z

And we're doing something like this:

dotnet test 
  --configuration Release 
  --no-build 
  --no-restore 
  --logger:trx 
  --verbosity minimal 
  --filter RunOnBuildServer!=false&Category=Unit 
  --results-directory test-outputs/component 
  /p:CollectCoverage=true 
  /p:ExcludeByAttribute="GeneratedCodeAttribute,ExcludeFromUnitTestCoverageAttribute" 
  /p:Exclude=[*]*.Migrations.* 
  /p:CoverletOutputFormat=opencover 
  /p:CoverletOutput=coverage.opencover.Unit.xml 
  Tests/Tests.csproj

JohnYoungers · 2025-01-09T13:10:04Z

@JohnYoungers not sure about a new release just for this, I need to discuss this with the other maintainers. There will be the possibility to consume the fix in our nightly once we have merged it.

I understand the hesitation, but I think as time goes on, this will catch more and more consumers with automated nuget update processes for patch releases.

This would be a sample of our usage:

  /p:CollectCoverage=true \
  /p:CoverletOutputFormat=cobertura%2copencover \
  /p:CoverletOutput=${{ inputs.output_path }}/${{ inputs.test_run_name }} \
  /p:Exclude="Company.App/Subnamespace/**/*,Company.App.Api/Program.cs,database/**" \

JSkimming · 2025-01-09T15:48:15Z

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

We run our test builds from a bash script. Our problem is not specifically cobertura output but any output. Where we previous received coverage, some of our builds get no coverage.

Here's the GitHub actions output where we no longer receive results.

Broken coverage

$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/Abc.MyService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.MyService.Tests.trx \
  --logger html;LogFileName=Abc.MyService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.MyService*]* \
  -p:Exclude="[*.Tests]*,[Abc.*]*.Migrations.*,[Abc.*]*.SourceGenerationContext" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  -p:CoverletOutputFormat="json,opencover,cobertura,lcov"

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.MyService.Tests'
Test run for /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/bin/Release/net9.0/Abc.MyService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.html

Passed!  - Failed:     0, Passed:   687, Skipped:     0, Total:   687, Duration: 1 m 20 s - Abc.MyService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.json'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.opencover.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.cobertura.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.info'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

Though we have another service where we still receive the output.

Still working coverage

$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/Abc.OtherService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/other-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.OtherService.Tests.trx \
  --logger html;LogFileName=Abc.OtherService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.*]* \
  -p:Exclude="[*.Tests]*,[Abc.Testing.*]*,[Abc.Platform.Testing.*]*" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.OtherService.Tests'
Test run for /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/bin/Release/net9.0/Abc.OtherService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.html

Passed!  - Failed:     0, Passed:   776, Skipped:     0, Total:   776, Duration: 50 s - Abc.OtherService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json'

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Abc.OtherService      | 100%   | 100%   | 100%   |
+-----------------------+--------+--------+--------+
| Abc.OtherService.Apps | 99.32% | 94.58% | 100%   |
+-----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 99.36% | 94.76% | 100%   |
+---------+--------+--------+--------+
| Average | 99.66% | 97.29% | 100%   |
+---------+--------+--------+--------+

LeszekKalibrate · 2025-01-09T17:37:54Z

I am running into the same problem ☹️
How this 6.0.3 was release without checking that it still works?

argium · 2025-01-09T21:57:49Z

I understand the hesitation, but I think as time goes on, this will catch more and more consumers with automated nuget update processes for patch releases.

I'll echo this. I don't know how many people are likely to be affected but the impact is significant (it doesn't work at all) so I would probably pull it before too many people upgrade after the holidays.

daveMueller · 2025-01-09T23:57:16Z

Thanks a lot guys 🙏 . I understand it much better now and think my changes should work. Could somebody try out the version I compiled locally?

coverlet.8.0.0-preview.3.g13af8e390b.zip

JohnYoungers · 2025-01-13T13:38:20Z

Sorry for the delay; I meant to follow-up on Friday: I attempted to test the new version, but noticed it was returning coverage results locally on 6.0.3 (on windows opposed to linux), so I couldn't compare with the new version

pinkfloydx33 · 2025-01-13T17:36:38Z

I meant to report this issue months ago and forgot =(. Per my notes it was introduced with 6.0.3-preview.3

You asked what our Include/Excludes look like....

<ExcludeByAttribute>ExcludeFromCodeCoverage,Obsolete,GeneratedCode,DebuggerHidden,DebuggerNonUserCode,Migration</ExcludeByAttribute>
<Exclude>[.*.tests?]*,[*.Tests]*,[xunit.*]*,[*]*.Entities,[My.Company.Analyzers]*,[My.Company.Kubernetes]*,[*]My.Company.Namespace.*</Exclude>

It's been a while, but I believe:

[.*.tests?]* and [*.Tests]* exclude assemblies ending in "test" or "tests"
- I'm not sure why the second one isn't included in the first, but I also think I got these from coverlet documentation at some point
[xunit.*]* - I have no idea why we'd need to exclude xUnit assemblies
that the first two are to exclude test projects from coverage. I'm not sure why xunit is there.
[*]*.Entities - exclude anything in any assembly in a namespace ending in 'Entities'
[My.Company.Analyzers]* and [My.Company.Kubernetes]* - exclude these two assemblies entirely
[*]My.Company.Namespace.* - exclude the My.Company.Namespace in any assembly

Another project has (very similar):

<ExcludeByAttribute>ExcludeFromCodeCoverage,GeneratedCode,DebuggerHidden,DebuggerNonUserCode,Migration,DbContext</ExcludeByAttribute>
<Exclude>[.*.tests?]*,[*.Tests]*,[xunit.*]*,[*]*.Migrations.*,[*.Test.Shared]*</Exclude>

First three same as above
[*]*.Migrations.* - Exclude any namespace with "Migrations" as one of its components
[*.Test.Shared]* - Exclude the entirety of any assembly that ends in ".Test.Shared"

stewartmossSingular · 2025-01-14T13:21:21Z

<?xml version="1.0" encoding="utf-8"?>
  <!-- File name extension must be .runsettings -->
  <RunSettings>
    <DataCollectionRunSettings>
      <DataCollectors>
        <DataCollector friendlyName="XPlat code coverage">
          <Configuration>
            <Exclude>
              [*Migrations]*,
              [*]*.Startup,
              [*]*.StartupExtensions,
              [Bob.Core*]*Bob.Core.Security.Roles*,
              [Bob.Core*]*Bob.Core.Security.*Roles*,
              [Bob.Core.Models]Bob.Core.Models.Identity.*,
            </Exclude>
            <Include>
            </Include>
            <SkipAutoProps>true</SkipAutoProps>
          </Configuration>
        </DataCollector>
      </DataCollectors>
    </DataCollectionRunSettings>
  </RunSettings>

JSkimming · 2025-01-14T20:27:58Z

Thanks a lot guys 🙏 . I understand it much better now and think my changes should work. Could somebody try out the version I compiled locally?

coverlet.8.0.0-preview.3.g13af8e390b.zip

@daveMueller Yes, these updated packages solve the problem for us.

mu88 · 2025-01-16T06:46:25Z

@daveMueller: you closed the issue, but it would be great if you could provide some more information about the next steps 🙂 currently we're using coverlet v6.0.2, but the provided preview package belongs to v8. Therefore, I get the impression that this fix will officially ship with the next major version of coverlet. Is that correct and if yes, when will this major version be released?

Or am I misunderstanding something and you're already in the midst of providing a fix v6.0.4?

daveMueller · 2025-01-16T09:53:53Z

@mu88 Sorry for the misunderstanding. The issue was automatically closed when I merged the PR. We planned some breaking changes for the v8 version but nothing of this is merged yet. So I can easily bump back to e.g. 6.0.4. The plan is most likely to release a 6.0.4 with the fix in the next days. But I need to discuss this with the other maintainers. Also I would appreciate a bit more feedback 😃. As this fix is now merged you can also consume our nightly build to test it.

ps: The nightly build is also called something with 8.0.0-preview... for the same reason as described above. Just ignore that. There is no breaking change yet.

stewartmossSingular · 2025-01-16T10:01:53Z

@daveMueller I have tested the latest nightly build package version 8.0.0-preview.2.ge3717da4a7 and it is working as expected against my exclusion rules.

mu88 · 2025-01-16T12:19:38Z

@daveMueller: I don't find any coverlet.8.0.0-preview.3.g13af8e390b ☹️

stewartmossSingular · 2025-01-16T12:23:13Z

use 8.0.0-preview.2.ge3717da4a7

daveMueller · 2025-01-16T12:32:42Z

@daveMueller: I don't find any coverlet.8.0.0-preview.3.g13af8e390b ☹️

Sorry preview.3.g13af8e390b was a local build. Just use the latest version from our nightly build feed 8.0.0-preview.2.ge3717da4a7.

mu88 · 2025-01-16T14:29:59Z

yes, it is working again with 8.0.0-preview.2.ge3717da4a7

daveMueller · 2025-01-20T13:39:36Z

Good news, we now officially released the fix with version 6.0.4.

JSkimming · 2025-01-20T14:16:58Z

Good news, we now officially released the fix with version 6.0.4.

Nice work @daveMueller. @dependabot updated all our projects last night, and I can confirm the fix has resolved our issue.

github-actions bot added the untriaged To be investigated label Jan 1, 2025

daveMueller added with repro Issue with repro bug Something isn't working and removed untriaged To be investigated labels Jan 2, 2025

karl-sjogren mentioned this issue Jan 7, 2025

Bump coverlet.msbuild from 6.0.2 to 6.0.3 karl-sjogren/Shorthand.HttpArchive#4

Closed

daveMueller mentioned this issue Jan 13, 2025

Exclude filter issue #1728

Merged

stewartmossSingular mentioned this issue Jan 14, 2025

[BUG] with Exclude certain classes in certain assemblies #1729

Closed

daveMueller closed this as completed in #1728 Jan 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2 #1726

[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2 #1726

kotsaris commented Jan 1, 2025 •

edited

Loading

sboulema commented Jan 1, 2025

daveMueller commented Jan 2, 2025 •

edited

Loading

mu88 commented Jan 6, 2025 •

edited

Loading

argium commented Jan 7, 2025

JohnYoungers commented Jan 8, 2025

daveMueller commented Jan 8, 2025

argium commented Jan 9, 2025 •

edited

Loading

mu88 commented Jan 9, 2025

JohnYoungers commented Jan 9, 2025 •

edited

Loading

JSkimming commented Jan 9, 2025

LeszekKalibrate commented Jan 9, 2025

argium commented Jan 9, 2025 •

edited

Loading

daveMueller commented Jan 9, 2025

JohnYoungers commented Jan 13, 2025

pinkfloydx33 commented Jan 13, 2025

stewartmossSingular commented Jan 14, 2025

JSkimming commented Jan 14, 2025

mu88 commented Jan 16, 2025

daveMueller commented Jan 16, 2025

stewartmossSingular commented Jan 16, 2025

mu88 commented Jan 16, 2025

stewartmossSingular commented Jan 16, 2025

daveMueller commented Jan 16, 2025

mu88 commented Jan 16, 2025

daveMueller commented Jan 20, 2025

JSkimming commented Jan 20, 2025

[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2 #1726

[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2 #1726

Comments

kotsaris commented Jan 1, 2025 • edited Loading

sboulema commented Jan 1, 2025

daveMueller commented Jan 2, 2025 • edited Loading

mu88 commented Jan 6, 2025 • edited Loading

argium commented Jan 7, 2025

JohnYoungers commented Jan 8, 2025

daveMueller commented Jan 8, 2025

argium commented Jan 9, 2025 • edited Loading

mu88 commented Jan 9, 2025

JohnYoungers commented Jan 9, 2025 • edited Loading

JSkimming commented Jan 9, 2025

LeszekKalibrate commented Jan 9, 2025

argium commented Jan 9, 2025 • edited Loading

daveMueller commented Jan 9, 2025

JohnYoungers commented Jan 13, 2025

pinkfloydx33 commented Jan 13, 2025

stewartmossSingular commented Jan 14, 2025

JSkimming commented Jan 14, 2025

mu88 commented Jan 16, 2025

daveMueller commented Jan 16, 2025

stewartmossSingular commented Jan 16, 2025

mu88 commented Jan 16, 2025

stewartmossSingular commented Jan 16, 2025

daveMueller commented Jan 16, 2025

mu88 commented Jan 16, 2025

daveMueller commented Jan 20, 2025

JSkimming commented Jan 20, 2025

kotsaris commented Jan 1, 2025 •

edited

Loading

daveMueller commented Jan 2, 2025 •

edited

Loading

mu88 commented Jan 6, 2025 •

edited

Loading

argium commented Jan 9, 2025 •

edited

Loading

JohnYoungers commented Jan 9, 2025 •

edited

Loading

argium commented Jan 9, 2025 •

edited

Loading