Test discovery fails with exception when test project uses .NET 6+ but references a project using .NET Standard

[dselzle]

I set up a barebones solution with two projects. One was a class library targeting .NET Standard 2.0, and the other was a unit test project targeting .NET 8.0. The test project included a reference to the class library project (<ProjectReference Include="..\path\to\ClassLibrary.csproj" />).

Versions:

• NUnit, version 4.2.2
• NUnit3TestAdapter, version 4.6.0
• Microsoft.NET.Test.Sdk, version 17.11.1
• Visual Studio Professional 2022, version 17.11.4
• Windows 11 Pro, version 10.0.22631

Test discovery fails with the following logs:

NUnit Adapter 4.6.0.0: Test discovery starting
Exception NUnit.Engine.NUnitEngineException, Exception thrown discovering tests in C:\path\to\TestProject\bin\Debug\net8.0\TestProject.dll
An exception occurred in the driver while loading tests.
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
at NUnit.VisualStudio.TestAdapter.NUnitEngine.NUnitEngineAdapter.Explore(TestFilter filter) in C:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\NUnitEngineAdapter.cs:line 94
at NUnit.VisualStudio.TestAdapter.NUnit3TestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger messageLogger, ITestCaseDiscoverySink discoverySink) in C:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestDiscoverer.cs:line 82
InnerException: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at System.Reflection.RuntimeAssembly.GetType(RuntimeAssembly assembly, String name, Boolean throwOnError, Boolean ignoreCase, ObjectHandleOnStack type)
at System.Reflection.RuntimeAssembly.GetType(String name, Boolean throwOnError, Boolean ignoreCase)
at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
NUnit Adapter 4.6.0.0: Test discovery complete
No test is available in C:\path\to\TestProject\bin\Debug\net8.0\TestProject.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

If I change the test project to target .NET Framework 4.6.2+, or if I change the class library to .NET 6+, test discovery works fine. The exception is only thrown when the library project is .NET Standard and the test project is .NET 6+.

I tried deleting bin/ and obj/ folders and restarting Visual Studio, but the behavior didn't change.

[OsirisTerje]

Can you please upload a repro project?

You can a) Add it as a PR to https://github.com/nunit/nunit3-vs-adapter.issues, add it as a separate folder named IssueXXX where the XXX is the issue number, or b) Add it as a zipped attachment here , or c) Add it to your own repo and just post the link.

Make it as small as you can, it should compile "as is".

[dselzle]

@OsirisTerje Pull request is available here.

I discovered that in addition to the .NET Standard/.NET 6+ combination, it also depends on having <GenerateTargetFrameworkAttribute> set to false in the test project. That might mean that this exception is unavoidable under those conditions--if so, I can work around the issue in this case.

[OsirisTerje]

The default value of <GenerateTargetFrameworkAttribute> is true, so I don't really get why it would require this to be explicitly set to true.
It seems to be the engine that reacts to this, which makes sense, as it is the engine that loads the assemblies.

I have been trying this a few times, and if I don't have that attribute set, and clean the solution to get rid of cache issues, then it works.

So it seems I can't repro this.

[dselzle]

I don't think it needs to be set explicitly to true. It just fails if it is explicitly set to false, which is required for some of my use cases.

[OsirisTerje]

@CharliePoole The engine is handling this. I would assume it needs that attribute. Is that correct?

@dselzle Did this work earlier with any other combination of .net frameworks?

[CharliePoole]

Yes, it's how it determines the target framework. If I remember correctly, it assumes .NET Framework 2.0 if the attribute is not found. If the adapter has better information, e.g. from the project file, you could add a setting to the test package to request the use of a certain framework.

[dselzle]

With the class library targeting Standard 2.0, I tried Framework 4.6, 4.6.2, 4.8, and 4.8.1 (all working), and .NET 6, 7, and 8 (all failing) for the test project.

[CharliePoole]

@dselzle Can you explain some use cases where you wouldn't want to have a TargetFrameworkAttribute?

[dselzle]

@CharliePoole A few years ago, we had some builds start failing due to "duplicate" TargetFrameworkAttributes for reasons we could never figure out. We started setting that attribute to false by default for all of our projects and haven't had any more issues.

Like I said, the workaround for this test discovery issue is simple enough in this case.

[CharliePoole]

OK, in that case, I guess I ought to simply document the fact that you can defeat discovery using this setting.