/TestCaseFilter does not support ClassName property

[dd-eg]

Expected behavior

The VSTest documentation for the TestCase filter command line option (/TestCaseFilter) describes supported properties for a couple test frameworks. For MSTest, "ClassName" is a supported property and the example provided in the documentation suggests this represents the fully qualified name of the test fixture class, i.e. {Namespace}.{ClassName}.

I'm interested in the ClassName property because I am using the VSTest task in an Azure pipeline to execute tests and am trying to enable rerunning failed test cases through its "rerunFailedTests" property. When I configure my pipeline this way, I see that the task identifies failed test cases and then tries to rerun them using a /TestCaseFilter expression containing ClassName and Name.

(Maybe the VSTest Azure pipeline task should not do be doing this and I should raise a bug there -- I figured I would start here though to understand what properties the NUnit test adapter supports. While reviewing open issues here, I did see #1202, which might ultimately be the same problem, but I wanted to remove Azure pipelines from the scope to make the investigation simpler -- please let me know if I should close this issue and provide information there instead.)

Actual behavior

When running tests with the NUnit framework through VSTest, /TestCaseFilter expressions containing ClassName fail to discover any tests.

For what it's worth, I looked at src/NUnitTestAdapter/VsTestFilter.cs and didn't see ClassName included among the test case properties.

I have tested this with the current latest stable versions of the relevant packages:

• Microsoft.NET.Test.Sdk: 17.12.0
• NUnit: 4.3.2
• NUnit3TestAdapter: 4.6.0
• NUnit.Analyzers: 4.6.0

I have Visual Studio 2022 17.12.0 installed.

Steps to reproduce

Below is a rudimentary .NET 8.0 project that I used to test this behavior.

TestProject1.sln

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 17
VisualStudioVersion = 17.12.35506.116 d17.12
MinimumVisualStudioVersion = 10.0.40219.1
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TestProject1", "TestProject1.csproj", "{AE00BA33-8811-4195-AB93-679B423B3F2D}"
EndProject
Global
	GlobalSection(SolutionConfigurationPlatforms) = preSolution
		Debug|Any CPU = Debug|Any CPU
		Release|Any CPU = Release|Any CPU
	EndGlobalSection
	GlobalSection(ProjectConfigurationPlatforms) = postSolution
		{AE00BA33-8811-4195-AB93-679B423B3F2D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{AE00BA33-8811-4195-AB93-679B423B3F2D}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{AE00BA33-8811-4195-AB93-679B423B3F2D}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{AE00BA33-8811-4195-AB93-679B423B3F2D}.Release|Any CPU.Build.0 = Release|Any CPU
	EndGlobalSection
	GlobalSection(SolutionProperties) = preSolution
		HideSolutionNode = FALSE
	EndGlobalSection
EndGlobal

TestProject1.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
    <PackageReference Include="NUnit" Version="4.3.2" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.6.0" />
    <PackageReference Include="NUnit.Analyzers" Version="4.6.0" />
  </ItemGroup>

</Project>

UnitTest1.cs

using NUnit.Framework;

namespace TestProject1
{
    public class Tests
    {
        [Test]
        public void Test1()
        {
            Assert.Pass();
        }

        [Test]
        public void Test2()
        {
            Assert.Fail();
        }
    }
}

After compiling, I ran the following VSTest command (using the VS2022 Developer Command Prompt) with the expectation that the TestCase filter would run all tests belonging to the specified class (i.e. tests Test1 and Test2):

vstest.console.exe TestProject1.dll /TestCaseFilter:"ClassName=TestProject1.Tests"

However, NUnit3TestExecutor does not discover any matching test cases:

VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter 4.6.0.0: Test execution started
Running selected tests in C:\git\TestProject1\bin\Debug\net8.0\TestProject1.dll
   NUnit3TestExecutor discovered 0 of 0 NUnit test cases using Current Discovery mode, Explicit run
NUnit Adapter 4.6.0.0: Test execution complete
No test matches the given testcase filter `ClassName=TestProject1.Tests` in C:\git\TestProject1\bin\Debug\net8.0\TestProject1.dll

[OsirisTerje]

The properties that are supported by the adapter is:

FullyQualifiedName (which is often used with contains)
Name Which is the same as the method or test name
Priority
TestCategory
Category (The two latest here are the same)

In your case, you could use FullyQualifiedName~TestProject1.Tests.

We'll try to get that MS documentation updated for NUnit too.

PS: Also look up the where filter: https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#where
The syntax for the where filter language is here: https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html

[dd-eg]

Thanks for the response! Good to know the supported properties.

I looked further into the VSTest Azure pipeline task behavior and found that at some point the /TestCaseFilter expression used for rerunning test cases changed from using FullyQualifiedName to ClassName and Name:
microsoft/azure-pipelines-tasks#16990 (comment)

So, I think that's the root of my issue -- I'll need to see whether a change is planned for the pipeline task (since I don't think I can override this behavior, other than specfiying and older version of the task).

[OsirisTerje]

Have you considered switching to use a yaml based pipeline (I assume you're using the classical now, as you refer to the VSTest). Doing that would allow you to use dotnet test instead, which would give you more options on filtering.

Ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/release/from-classic-pipelines?view=azure-devops

[OsirisTerje]

Actually, (just checked) the Name does the same as the ClassName, it actually works with whatever you give it.

If you give it a test method name, it will run all tests with that name

If you give it a test class name, it will run all tests within that class

If you give it a namespace, it will run all tests within that namespace

[dd-eg]

We're using yaml pipelines for builds, but have many release pipelines too (which are stuck on classic for now). I think I'd like to move to dotnet test at some point anyway, but what I was really chasing here was the opportunity to use Azure DevOps's flaky test detection feature in conjunction with the VSTest task's built-in "rerun" properties.

That said, maybe I should just look into dropping the VSTest task and implement my own rerun via dotnet test -- something like (1) dotnet test {initial filter}, (2) extract failures from results (3) dotnet test {failed result filter}. That way I have more direct control over the filtering!