GD-133: Test Coverage isn't reported for test cases for Resharper/Rider

[alicerunsonfedora]

The used GdUnit4 version

4.3.3 (Latest Release)

The used Godot version

v4.2.2.stable.mono.official [15073afe3]

Operating System

Windows 11 Home (23H2, OS Build: 22631.3880)
Surface Pro 9, 16GB RAM, Intel Core i5 (i5-1235U)

Describe the bug

In both Rider and Visual Studio (with dotCover/Resharper), test coverage data isn't reported for nodes and other classes that the tests cover.

For example, I have an extension class in my project:

namespace IndexingYourHeart.Utils
{
    public static class CollectionUtils
    {
       public static T RemoveFirst<T>(this List<T> list)
        {
            var firstItem = list[0];
            list.RemoveAt(0);
            return firstItem;
        }
    }
}

With unit tests to cover:

namespace IndexingYourHeart.Tests.Unit
{
    [TestSuite]
    public class CollectionUtilsTests
    {
        [TestCase]
        public void TestRemoveFirst()
        {
            List<string> aList = ["a", "b", "c"];
            var first = aList.RemoveFirst();

            AssertString(first).IsEqual("a");
            AssertInt(aList.Count).IsEqual(2);
        }
    }
}

GdUnit picks up the tests and runs them (yay), and they show in the test explorers for both Visual Studio and Rider. However, coverage data isn't reported. It does report for other C# projects in the solution, but not the one where my game's code is specified.

In VS 2022 Community with Resharper:

In Rider 2024.1.4:

(Yes, I am aware that this is a very silly case, as I could just stick this in a library. But this happens to other tests that I've written using GdUnit).

Steps to Reproduce

1. Write a test to cover some code.
2. Update the .runsettings to point to your Godot EXE file.
3. Run the tests to ensure they're in the test explorer.
4. In either Resharper for Visual Studio or Rider, configure the test runners to read from the .runsettings file for discovery. (Alternatively, import the .DotSettings.user file in the minimal project).
5. In either Resharper for Visual Studio or Rider, run "Cover Tests" (or run the tool to get test coverage).
6. Notice that there is no coverage reported.

Minimal reproduction project

GdUnitCoverageMinimal.zip

[MikeSchulze]

Thanks for reporting this issue, I will take a look into.

[MikeSchulze]

@alicerunsonfedora

At first, you need to enable the test code coverage in your .runsettings (I have actual not documented)

Here is an example I used for VS Code, it generates the code coverage report.

<!-- To enable code coverage -->
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <Format>cobertura</Format>
                    <CoverageFileName>coverage.cobertura.xml</CoverageFileName>
                    <UseSourceLink>true</UseSourceLink>
                    <ExcludeByFile>**/Godot.SourceGenerators/**/**.cs</ExcludeByFile>
                    <SingleHit>false</SingleHit>

                    <CodeCoverage>
                        <ModulePaths>
                            <Include>
                                <ModulePath>.*\.dll$</ModulePath>
                                <ModulePath>.*\.exe$</ModulePath>
                            </Include>
                            <Exclude>
                                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                                <ModulePath>.*Moq.dll</ModulePath>
                                <ModulePath>.*GodotSharp.dll</ModulePath>
                                <ModulePath>.*GodotPlugins.dll</ModulePath>
                                <ModulePath>.*gdUnit4Api.dll</ModulePath>
                                <ModulePath>.*gdUnit4.TestAdapter.dll</ModulePath>
                            </Exclude>
                        </ModulePaths>
                        <Sources>
                            <Exclude>
                                <Source>.*/Godot.SourceGenerators/.*</Source>
                                <Source>.*/gdunit4_testadapter/.*</Source>
                            </Exclude>
                        </Sources>
                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <CollectAspDotNet>False</CollectAspDotNet>
                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>

It should normally work also on Rider and VS.
On Rider it looks like it is ignoring the .runsettings and no coverage report is generated.
For Visual Studio i have no license to test.

@van800 have you any idea why the .runsettings are ignored by the Rider test runner?
I see the correct settings are loaded, but it has no effect.

17:56:43.075 |I| Running up to 1 unit-test runs in parallel
17:56:43.075 |I| Run: 7f6e1a93-45ac-4dda-a5f7-ca453c63aa05 - Starting
17:56:43.075 |V|     Provider: VsTest
    Target Framework: net8.0
    Strategy: VsTestTestRunnerRunStrategy
    Runtime: NetCore
  Host: JetBrains.ReSharper.Plugins.Godot.UnitTesting.GodotGdUnitTestRunnerHost
  Project: Gd Unit Coverage Minimal
  TargetPlatform: AnyCpu
  TargetFrameworkId: net8.0
  HasNativeCode: False
17:56:43.075 |T| Run: 7f6e1a93-45ac-4dda-a5f7-ca453c63aa05 >> RunHostController.PrepareForRun
17:56:43.075 |T| Run: 7f6e1a93-45ac-4dda-a5f7-ca453c63aa05 << RunHostController.PrepareForRun
17:56:43.078 |V| RunSettings: 
<RunSettings><RunConfiguration><MaxCpuCount>1</MaxCpuCount><ResultsDirectory>D:\development\workspace\GdUnitCoverageMinimal\TestResults</ResultsDirectory><TargetFrameworks>net7.0;net8.0</TargetFrameworks><TestSessionTimeout>300000</TestSessionTimeout><TreatNoTestsAsError>true</TreatNoTestsAsError><SolutionDirectory>D:\development\workspace\GdUnitCoverageMinimal</SolutionDirectory></RunConfiguration><LoggerRunSettings><Loggers><Logger friendlyName="console" enabled="True"><Configuration><Verbosity>detailed</Verbosity></Configuration></Logger><Logger friendlyName="html" enabled="True"><Configuration><LogFileName>test-result.html</LogFileName></Configuration></Logger><Logger friendlyName="trx" enabled="True"><Configuration><LogFileName>test-result.trx</LogFileName></Configuration></Logger></Loggers></LoggerRunSettings><!-- To enable code coverage --><DataCollectionRunSettings><DataCollectors><DataCollector friendlyName="XPlat Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"><Configuration><Format>cobertura</Format><CoverageFileName>coverage.cobertura.xml</CoverageFileName><UseSourceLink>true</UseSourceLink><ExcludeByFile>**/Godot.SourceGenerators/**/**.cs,./gdunit4_testadapter/TestAdapterRunner.cs</ExcludeByFile><SingleHit>false</SingleHit><CodeCoverage><ModulePaths><Include><ModulePath>.*\.dll$</ModulePath><ModulePath>.*\.exe$</ModulePath></Include><Exclude><ModulePath>.*CPPUnitTestFramework.*</ModulePath><ModulePath>.*Moq.dll</ModulePath><ModulePath>.*GodotSharp.dll</ModulePath><ModulePath>.*GodotPlugins.dll</ModulePath><ModulePath>.*gdUnit4Api.dll</ModulePath><ModulePath>.*gdUnit4.TestAdapter.dll</ModulePath></Exclude></ModulePaths><Sources><Exclude><Source>.*/Godot.SourceGenerators/.*</Source><Source>.*/gdunit4_testadapter/.*</Source></Exclude></Sources><UseVerifiableInstrumentation>True</UseVerifiableInstrumentation><AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses><CollectFromChildProcesses>True</CollectFromChildProcesses><CollectAspDotNet>False</CollectAspDotNet></CodeCoverage></Configuration></DataCollector></DataCollectors></DataCollectionRunSettings></RunSettings>

[alicerunsonfedora]

It seems this doesn't quite work with VS with Resharper either, so I suspect there's something else going on with JetBrains's tooling specifically. Unfortunately, I don't have an enterprise license of VS 2022, so I can't tell whether coverage reports normally under Microsoft's solution.

I could also have misconfigured my run settings here.

[MikeSchulze]

@van800 do you have knowledge of JetBrains ReSharper and code coverage? I suspect that the problem with code coverage of the executed tests is that they run in a sub-process that is started by the test runner via Godot. So I'm guessing the DLL loaded by the Godot instance is not attached to the ReSharper process.
Have you any idea how to register this DLL's ?

Maybe can we get in contact via Discord?

[MikeSchulze]

@alicerunsonfedora it looks like it is an issue with the Jetbrains ReSharper.

• I got it running inside VS Code with some limitations, see Attempting to parse cobertura.xml fails after converting from visual studio .coverage binary. ryanluker/vscode-coverage-gutters#438
• And also using dotnet test from command line

I suggest doing run the coverage by this batch script as a workaround until I have found the problem with the ReSharper.

runTestCoverage.bat

echo "Start Test Code Coverage"

dotnet test --settings .vscode.runsettings
reportgenerator -reports:./TestResults/*/coverage.cobertura.xml -targetdir:./TestResults/coverage -reporttypes:Html_Dark -sourcedirs:./ -historydir:./TestResults/coverage/history
# opend the report in the standard browser
start "" ./TestResults/coverage\index.html

here the .vscode.runsettings

<RunSettings>
    <RunConfiguration>
		<MaxCpuCount>1</MaxCpuCount>
		<ResultsDirectory>./TestResults</ResultsDirectory>
		<TargetFrameworks>net7.0;net8.0</TargetFrameworks>
		<TestSessionTimeout>300000</TestSessionTimeout>
		<TreatNoTestsAsError>true</TreatNoTestsAsError>
    </RunConfiguration>

    <LoggerRunSettings>
        <Loggers>
            <Logger friendlyName="console" enabled="True">
                <Configuration>
                    <Verbosity>detailed</Verbosity>
                </Configuration>
            </Logger>
            <Logger friendlyName="html" enabled="True">
                <Configuration>
                    <LogFileName>test-result.html</LogFileName>
                </Configuration>
            </Logger>
            <Logger friendlyName="trx" enabled="True">
                <Configuration>
                    <LogFileName>test-result.trx</LogFileName>
                </Configuration>
            </Logger>
        </Loggers>
    </LoggerRunSettings>


	<!-- To enable code coverage -->
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <Format>cobertura</Format>
                    <CoverageFileName>coverage.cobertura.xml</CoverageFileName>
                    <UseSourceLink>true</UseSourceLink>
                    <ExcludeByFile>**/Godot.SourceGenerators/**/**.cs,./gdunit4_testadapter/TestAdapterRunner.cs</ExcludeByFile>
                    <SingleHit>false</SingleHit>

                    <CodeCoverage>
                        <ModulePaths>
                            <Include>
                                <ModulePath>.*\.dll$</ModulePath>
                                <ModulePath>.*\.exe$</ModulePath>
                            </Include>
                            <Exclude>
                                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                                <ModulePath>.*Moq.dll</ModulePath>
                                <ModulePath>.*GodotSharp.dll</ModulePath>
                                <ModulePath>.*GodotPlugins.dll</ModulePath>
                                <ModulePath>.*gdUnit4Api.dll</ModulePath>
                                <ModulePath>.*gdUnit4.TestAdapter.dll</ModulePath>
                            </Exclude>
                        </ModulePaths>
                        <Sources>
                            <Exclude>
                                <Source>.*/Godot.SourceGenerators/.*</Source>
                                <Source>.*/gdunit4_testadapter/.*</Source>
                            </Exclude>
                        </Sources>
                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <CollectAspDotNet>False</CollectAspDotNet>
                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

And you need to complete your project by adding the coverlet.collector

<Project Sdk="Godot.NET.Sdk/4.2.2">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <RootNamespace>GdUnitCoverageMinimal</RootNamespace>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
    <PackageReference Include="gdUnit4.api" Version="4.3.*" />
    <PackageReference Include="gdUnit4.test.adapter" Version="2.*" />

    <PackageReference Include="coverlet.collector" Version="6.0.2">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="ReportGenerator" Version="5.2.4" />
  </ItemGroup>
</Project>

And finally install the global reporting tool
dotnet tool install -g dotnet-reportgenerator-globaltool

here is a very good documentation about https://medium.com/@nocgod/how-to-setup-your-dotnet-project-with-a-test-coverage-reporting-6ff1903f7240

[alicerunsonfedora]

Thanks for the advice! At the moment, I'm not too concerned on test coverage, but I figured I'd report this issue for anyone that does run into the problem with Resharper/Rider at some point.

[van800]

I am back from vacation and passed the investigation to the dotCover team.
https://youtrack.jetbrains.com/issue/DCVR-12691/Support-GdUnit4Net-in-dotCover