TFS Build won't discover tests using MSTest v2 adapter when using PackageReference

[CoolDadTx]

Description

I have upgraded some of our test projects to use PackageReference instead of the packages.config file for NuGet packages. These projects get built on our TFS 2017 build agent running on premise. Before switching to a PackageReference file the tests were discovered. After switching they weren't.

This only seems to be an issue in TFS. In VS 2017 it detects and runs the tests just fine.

Steps to reproduce

1. Create a unit test project (targeting .NET 4.6.1) and using MSTest v2
2. Change the project to use PackageReference instead of packages.config
3. Set up a build definition to build and run the tests in the project.

Expected behavior

The tests would get run and the TFS build log would display the test results.

Actual behavior

The build log reports there are no tests. Looking at the detailed log, it loads each of the test assemblies but it reports that no test adapters could be found.

Warning: No test is available in {output dll}. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

I wonder if the issue is related to the .targets file. When switching from packages.config to PackageReference, the packages are no longer downloaded and stored in a Packages folder in the solution. There are 2 .targets files added automatically when creating the project. The first 1 Microsoft.TestTools.targets is being pulled from the VSToolsPath (which doesn't seem right on a build server). The second one though (MSTest.TestAdapter.targets) is pulled from the Packages folder. But that folder won't exist in a PackageReference world so it isn't found. When converting from packages.config that import gets removed. Looking at either of the .targets files, it doesn't seem they would break test discovery though.

Environment

VS 2017 Update 3.2 - works fine
Unit Test Project = .NET 4.6.1 targeting MSTest v2 (framework and adapter)
TFS 2017 Update 2
Visual Studio Test task v2.*
Test platform version = Latest

[AbhitejJohn]

@CoolDadTx : Thanks for filing this. We haven't tested this scenario with PackageReference yet though. I imagine this is happening because a PackageReference restores packages to a central user directory as opposed to beside the solution with packages.config. Could you set the globalPackagesFolder relative to the solution via a nuget.config and give this a shot please? That property can be made a relative path off of the solution directory.
/cc: @nigurr , @acesiddhu .

[CoolDadTx]

This sounds like a viable solution for now but there will be issues going forward. 1) I would have to do this for all the solutions and projects (which defeats the purpose of going to a centralized package directory), and 2) when upgrading the package to a newer version I would need to remember to go into the project file(s) and change the .targets path to use the new version number since the package version is part of the path.

I'm curious to know why VS itself has no problems finding the tests but MSBuild does? I assumed they were using the same logic to find the test adapters but evidently they are not.

[CoolDadTx]

Unfortunately I'm unable to get NuGet to properly use the globalPackagesFolder setting. Any relative path given is relative to the .nuget folder. Using a hard-coded path wouldn't work on a build machine. But even trying to use that doesn't do anything. I've got a comment over in the NuGet documentation about this property not behaving correctly but I'm awaiting a response.

[AbhitejJohn]

VS actually reads this from the project references. I do see the point that having a per solution packages folder does not fit in well with a centralized store. In which case you could use the TestAdaptersPaths section in the runsettings to point to a specific version of an adapter. This could be automated via an environment variable or a build script when the adapter is updated.

I'm wondering how this(providing an adapter path if the packages reside in a central location) would work in the build vnext workflows though. @acesiddhu , @nigurr: Thoughts? Maybe we can have the same experience in the TFS Build world.

[nigurr]

@CoolDadTx I have created sample test project using package reference, the build output folder contains Test dll and MSTest.TestAdapter.dll. This should work out of the box using VsTest v2 as long as TestAdapter is present along with test dlls. Can you please share your build log (enable system.debug=true) and can you please confirm whether output directory of msbuild/vsbuild contains TestAdapter

[AbhitejJohn]

@nigurr: Would this work in XAML build as well?

[nigurr]

@AbhitejJohn You can't create XAML build using VS 2017/TFS 2017.

[CoolDadTx]

Test.log

Yes the TestAdapter DLL is in the same folder as the tests when the test task runs. But the warning in the task says it is looking in \s for the test adapter. The build drops the files into the \a folder.

[commonsensesoftware]

Additional Information

I can confirm a similar behavior using the .NET CLI when you skip the build. My build process restores and builds all projects in the solution first. The .NET CLI task configured with:

dotnet test --no-restore --no-build --logger:trx

should be optimal to prevent compilation from happening multiple times. Unfortunately, things do not work with this combination. If I change the arguments to:

dotnet test --logger:trx

then things work as expected.

Workaround

For anyone looking for a workaround until there is a more definitive resolution, the following should work for you:

1. Remove the Visual Studio Test task
2. Add the .NET Core task (I used version 1.*)
   a. Set the command to test
   b. Match the test projects in your repo according to your hierarchy (ex: test\**\*.csproj)
   c. Specify the value --logger:trx in the Arguments section
3. Add the Publish Test Results task (I used version 2.*)
   a. Set the Test Result Format to VSTest
   b. Set the Test Results Files to **\*.trx
   c. Optionally, update the Search Folder; mine was $(System.DefaultWorkingDirectory)\test
   d. Check the Merge Test Results option

Once this is complete, you should be able to build, test, and publish your test results in a manner congruent with the Visual Studio Test task.

[nigurr]

@CoolDadTx if user is having configuration to generate Test binaries in custom location, we expect to mention the same in "Custom Adapter" field.

[CoolDadTx]

If I understand you correctly then you're saying that because I have a custom build location then it wouldn't work but that doesn't really make sense to me. There is no default TFS build location in a vNext build. It completely depends upon the type of app(s) you're building and which build task you use. In the case of web apps that need to publish results you'll likely be building to a single output path (but different subfolders for each project). For desktop/console apps you'll likely want to build to separate directories for each one. This may be in the source folder (like VS does) or it could be elsewhere. In this situation though the project determines the output path, not TFS. I'm not even going to go into the MSBuild task that people probably don't use.

There is no "normal" build output folder. If the test task needs to know the output folder to find things then it should probably be a required argument. Unless you search the standard work folders of a, s and b; the task will miss the adapters in potentially half the builds (assuming a mix of web and non-web solutions). By making the argument explicit then you eliminate this problem. It would be even worse if you have a mix of apps such that some build to s and some build to a. Then you might get a false positive about the adapter being used.

But I really question why you're trying to find a single adapter path anyway. This seems wrong to me. I understand there are some issues around Test Explorer in VS but I don't think the Test task should be following this broken behavior. What happens if I am in the middle of switching test frameworks and I have a mix of NUnit and MSTests. They may or may not be in the same folders. Even worse would be if I run into a compatibility issue with a test framework and have to (or accidentally) use different versions of the same framework. The test task shouldn't be relying on a single test adapter path in my opinion. It should ideally be using the test adapter(s) in the path where the test assemblies reside. Of course this may then cause differences between VS builds and TFS builds.

So, summary, I don't believe the test task can heuristically and reliably find the test adapters in even common cases. So it should probably always require me to specify the path(s) to the adapters. It should then do a recursive search from there (so it finds per-project adapters when I'm building non-web apps.

[AbhitejJohn]

@CoolDadTx : I think we need to think through this a bit. This RFC collects some thoughts on providing adapters but I'm not seeing a definitive guidance for your scenario (This is with the new version of the platform though).
/cc: @codito, @singhsarab, @pvlakshm

[commonsensesoftware]

Maybe I'm missing something. The RFC claims that test adapters provided by NuGet via PackageReference only work in the IDE, but that's not true. The example I gave above works without the IDE. This information can be extracted anytime after dotnet restore in the obj folder. In pure MSBuild parlance, that would be msbuild /t:Restore.

The restore task generates a file called [project].[project ext].nuget.g.props. This file will contain the build extensions for test adapters. In fairness, you won't know which imported build properties contain test adapters without evaluating them, but that should be of minimal consequence. At least you'll know where to look. Even that may not be necessary. At least from what I've seen, the test adapter assemblies are copied to the output bin folder, which is a location I thought was already searched.

At least in my scenario, I'm not customizing the drop location in any way. In this particular case, I'm using MSTest V2 also, but the tests are still not being discovered.

[codito]

@commonsensesoftware the VSTest task in VSTS doesn't support .NET core projects currently. The workaround you mentioned in #286 (comment) is actually the recommended approach to test .NET core projects in CI.

[commonsensesoftware]

I'm not sure I understand why not. It sounds more like the VSTest Task doesn't support MSBuild 15.0. In my case (like @CoolDadTx ) I'm using the new SDK Project model introduced in MSBuild 15.0, but I'm not building a .NET Core application (it's .NET 4.5.1 actually). It's puzzling that version 2.* of the task says that it supports Visual Studio 2017. If that's true, then it should support MSBuild 15.0 too.

Regardless, I submit that the claim in your RFC that states test adapter lookup requires the IDE when referenced via NuGet packages is inaccurate. The CLI command dotnet test proves that tests can be run without the IDE. In the older packages.config model, there may be some limitations at a CLI level that were never addressed, but it is certainly possible to resolve where the request test adapters are if they aren't directly in build output.

With the impact to all the custom test adapters such as Xunit, Nunit, and MSTest V2, I don't understand why this hasn't already been addressed. This seems like it should be a pretty high priority IMHO.

[nigurr]

@CoolDadTx I agree with you partially on this point "So it should probably always require me to specify the path(s) to the adapters. It should then do a recursive search from there".

We do have the Custom Adapter field for the same reason. Additionally, we search for adapters are searched for test binaries folder where most of the default build projects copy the nuget dependencies/assembly references.

We wouldn't be able to make it "always required" for below reasons.

1. VS 2015 ships along with MsTest adapter which is automatically loaded and doesn't require explicit adapter to be shipped with test binaries. Most of this scenarios will break if we make it as required field.
2. Most of the default msbuild projects which uses nuget references instead of package references, copy the test assemblies along along with adapters into single folder. This is the reason behind searching for adapters automatically in specified test assemblies folder.
3. For any reason, customer build project doesn't ship with test adapters, they can choose to mention the path in the task.

We would be happy to understand more of this scenarios to make it more streamlined experience and welcome your suggestions.

[codito]

Regardless, I submit that the claim in your RFC that states test adapter lookup requires the IDE when referenced via NuGet packages is inaccurate. The CLI command dotnet test proves that tests can be run without the IDE. In the older packages.config model, there may be some limitations at a CLI level that were never addressed, but it is certainly possible to resolve where the request test adapters are if they aren't directly in build output.

Let me try to elaborate little more on nuget packages query (as mentioned here: https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0013-Test-Adapter-Lookup.md#nuget). Hopefully I can clarify the scenario more.

1. IDE (VS) uses nuget APIs to query the packages directly (this API is at a higher abstraction, it doesn't care if the nuget package is PackageReference or package.config). In both cases, the adapters are discovered and sent to the runner. Runner is (2) is below.

2. CLI (vstest.console.exe or dotnet vstest) cares only about assemblies (it has no knowledge of build, packaging, project system etc.). You can build with msbuild or cake or nant etc. and still will be able to use the test runner. VSTS task (in my opinion) is an extension of this. It does not understand project or build system.

3. Dotnet CLI (dotnet test) is different from CLI above. It has understanding of project system, dotnet test can take the project as an input, unlike the CLI which takes test.dll. It is actually a superset of CLI - think of it as a layer which understands the project output and invokes CLI. I talked bit more about this in a blog post here.

The tight coupling with Project System, Build in (3) provides certain advantages: it can automatically figure the Target Framework, the exact Test Assembly location, automatically finds out the Test Adapter Path. However being decoupled from project system, build and packaging, the scenarios that (2) can address are a super set of (3). E.g. c++, js, or uwp tests.

In terms of heuristic to find test adapter path in (2), we are adding support for automatically finding test adapters in the same location as the test assembly. E.g. vstest.console.exe c:\test\foo.dll will automatically use any test adapter available in c:\test. This is generic to support C++ and other languages without adding knowledge about project/packaging in each language.

[CoolDadTx]

@nigurr, "We would be happy to understand more of this scenarios "
To me this is straightforward. We need to be able to build and run tests in an MSBuild build just like we can in VS, using the same test adapters that VS would use. Whatever it takes to make that work is what I would want. But, at least right now, it doesn't seem like it is doable because it doesn't appear to do a recursive search right now. So if I am building each test assembly into its own folder then I could only point it to 1 of them. If I had mixed tests (i.e. NUnit and MSTest) then this would fail.

I'll try using the custom adapter path and see if I can get past my issue, which is the whole reason I posted this issue to begin with. If so then I'm fine. Otherwise I'll have to add some additional steps in my build process to move files around so the test task can find things. Given that we have over 40 build definitions then that is a lot of work. Fortunately we are using task groups for the test infrastructure but I don't envy others who aren't.

[nigurr]

@PBoraMSFT

[StingyJack]

The general description of this is very much like one of the side effects of #241 except there is only packages.config

[ShreyasRmsft]

SampleProject.zip

I created this sample project with package references instead of a packages.config file.

Above are the pics of my definition and run on VSTS. Things work out of the box with the default .NET desktop app template.

No custom adapter path was needed here.

Closing the issue. Please reopen if this solution does not work for you and please provide the sample project and the definition you are using.

[vaikzs]

@ShreyasRmsft I agree with your method, everything works fine BUT not xUnit test. I have been struggling for 5 days with this and still not able to figure out a way to execute xUnit with standard template (VS Test) from VSTS marketplace. Please try to suggest possible solutions.

File structure

MainProject
MainProjectUnitTests (Contain xUnit test cases)
MainProject.sln

[ShreyasRmsft]

@vaikzs can you provide the link to a GitHub repo with a sample repro?

I basically need the full debug logs of the vstest task
your project references to see where the xunit adapter and framework dlls are being placed

[ShreyasRmsft]

@vaikzs can you please create a new issue with data i have requested?

[vaikzs]

Sorry for the delayed response. Unfortunately, I do not have the xUnit anymore. I shifted to MSTest. And it worked fine. But I do remember dlls file location is bin\Debug\netcoreapp2.1\MainProjectUnitTests.dll