Test Failures due to auto redirection for nuget packages in VS

[cltshivash]

Details about Problem

NuGet product used : VS 15.6.2 update

VS version (if appropriate):

OS version (i.e. win10 v1607 (14393.321)): N/A

Detailed repro steps so we can see the same problem

Users authoring test projects are running into issues with with the auto redirection being added without the right assemblies being present in the output location. More details below.

Repro (https://github.com/smadala/samples/tree/master/Test%20Explorer%20fails%20to%20show%20tests/ClassLibrary.Tests)
https://github.com/spottedmahn/Experiments/tree/master/Test%20Explorer%20fails%20to%20show%20tests

Detailed Repro Steps

1. Create a simple class library (full clr) and add System.runtime 4.3.0 to the classlibrary
2. There’s no binding redirect added on adding just the system.runtime’s 4.3.0 version to the classlibrary.
3. Create a test project exercising the classlibrary (Test -> Unit Test Project)
4. Execute the test, from the test explorer the tests execute fine with System.RunTime.dll being picked from GAC (4.0.0.0) version (which is the test platform/framework dependency)
5. Add Moq nuget package to the test project (version 4.8.*). When Moq.dll is added to the test project, the app.config of the test project gets updated with the redirect for the System.RunTime to 4.1.1.0. There’s no reference added to the test project appropriate dll (System.RunTime.dll 4.1.1.0 version).
6. Run the tests, the test fails to execute. When the tests execute the configuration provided in dll.config is used to setup an appdomain for test execution. The appbase for the appdomain is the output directory of the test project. Since the System.Runtime.dll (4.1.1.0) is not present in the test project output path the execution fails (the error message doesn't indicate the dll that failed and has been fixed to provide a meaningful message in an upcoming update)
7. Now uninstall Moq.dll (which added the redirection in the first place). The redirection still stays in the test project.
8. The issue doesn’t repro when the test projects target 4.7.1 (looks like some consolidation that’s being talked of here dotnet/corefx#25773 is helping)
9. The issue can be mitigated by either removing the redirects or by adding a reference to the 4.1.1.0 version of the System.Runtime.dll and ensuring that it is present in the test output path.

This is affecting a lot of users (as seen in microsoft/testfx#241). If this isn't the right place to track this issue, do let me know.

[cltshivash]

Any update on this ? This is currently causing one of the top issues users of the test framework are running into.

[zhili1208]

@cltshivash Does this work before? Which VS version it works on? We haven't changed code related to this for a long time. I suspect something changed in VS.

[cltshivash]

@zhili1208 : I will need to check this. The issue repros on the latest 15.6 update. Do you need the exact VS version where it worked ?

[cltshivash]

@zhili1208 : The issue repros on VS 2015 as well. You will need to set the target framework for the classlibrary and the test project as 4.6.2. It doesn't repro if this is set as 4.6.1 or 4.7.*. Hope this helps in narrowing down the cause.

[cltshivash]

@zhili1208 : Any update on this issue ?

[zhili1208]

@cltshivash we are trying to work on this bug this sprint.

[StingyJack]

Related

[spottedmahn]

That's great, thanks for the update @cltshivash!

[rrelyea]

https://developercommunity.visualstudio.com/content/problem/201434/test-explorer-fails-to-show-tests-hresult-0x801310.html?childToView=225207

[mishra14]

@cltshivash Thanks for reporting the issue. I am currently investigating this issue.

Will it be possible for you to unblock yourself by moving the class library to package reference style instead of packages.config?

[StingyJack]

FWIW, this has been happening to my team intermittently over the last year, mostly without including moq. Just using the two MSTest nuget packages that the 4.6.2 (and 4.5.2) unit test project template installs and adding a single unit test to that project and VS will fail to discover that test.

We unblock by removing those two packages entirely and then reffing the old MS quality framework dll. This lets local and server builds always find all the tests and execute them. Changing anything to package reference will not be an option.

[spottedmahn]

Hi @StingyJack 😊

Changing anything to package reference will not be an option.

Wonder why? That's is the new way and where we're all headed...

Migration doc page

[StingyJack]

Hey @spottedmahn - I did the math once for someone on the nuget team who asked the same question . For all the projects and solutions my team has to maintain, it came out to over 4 weeks of time to migrate from package.config to the packageref and that's with only a minimal (smoke) test. I cant make any business argument that justifies spending that kind of time when the potential value add is so trivial.

I actually have several other reasons but dont want to side track getting this very nasty defect fixed. For > 6 months unit tests were failing silently and new bugs were getting into code that was delivered to customers.

[spottedmahn]

For all the projects and solutions my team has to maintain, it came out to over 4 weeks of time to migrate from package.config to the packageref and that's with only a minimal (smoke) test.

Interesting, thanks for sharing! @StingyJack

[mishra14]

@cltshivash I investigated a bit. Here is a rundown -

1. System.Runtime package version 4.3.0 contains the dll with assembly version 4.1.1.0.

2. Moq brings in System.Threading.Tasks.Extensions which has an assembly reference to System.Runtime v4.0.0.0. This causes NuGet to add a binding redirect in the app.config.

3. There is a known issue with removing binding redirects/app.config file on uninstall. We will consider fixing this in short term.

4. I think the following is not correct, because the project bin directory contains System.Runtime.dll and if you look at its assembly version, it is 4.1.1.0

Run the tests, the test fails to execute. When the tests execute the configuration provided in dll.config is used to setup an appdomain for test execution. The appbase for the appdomain is the output directory of the test project. Since the System.Runtime.dll (4.1.1.0) is not present in the test project output path the execution fails (the error message doesn't indicate the dll that failed and has been fixed to provide a meaningful message in an upcoming update)

Update - I have created an issue to track the issue about removing binding redirects on uninstall - #6870.

[cltshivash]

@mishra14 : Could you please clarify on point 4 which project's bin directory ? Below is the content of the testproject's output directory which has no System.RunTime.dll present but the config file (UnitTestProject1.dll.config has the below redirection). Now execution is going to fail as the redirected version will not be found.

[cltshivash]

@mishra14 The test execution uses the testproject's output location as the appbase for an appdomain and tries to invoke the test method. If there's a config file for the test dll , it is added to the appdomain setup info.

[mishra14]

@cltshivash Thanks for giving me more information. In my last pass I was installing Moq on the main project and hence missed the issue.

I investigated this from the test project's point of view. The root cause here is that in Packages.Config, package references do not flow transitively to the test project. Hence System.Runtime is not included in the test project's output directory. You can fix this problem by adding a package reference to System.Runtime v4.3.0 in the test project. This has been by design in Packages.Config type of projects.

Please keep in mind that even if the binding redirect was not added, you still needed a reference to System.Runtime in the test project because the class library project has a runtime dependency on it and since the package reference is not considered while building the test project, the test project will not be able to run.

This specific problem is fixed by design in PackageReference type of projects and is one of the main limitations in Packages.Config.

[StingyJack]

Visual Studio has forever been "helping" me by finding whatever reference or dependency a project needed by looking in all kinds of places. While sometimes this caused frustrating "works on my machine" issues, it was probably overall a good thing. So a project that has references has references that know their references. and so on (for static refs ). Was there a reason why the original design of nuget did not just take advantage of this?

[cltshivash]

@mishra14 Migating the nuget references off package.config to PackageReference helps resolve the issue for the repro shared on this issue (i do see the runtime dll getting copied to the output folder of the test project).

@StingyJack Please do see if the option provided by @mishra14 helps.

[cltshivash]

@spottedmahn It would be helpful if you can validate the above recommendation from @mishra14

[mishra14]

@cltshivash thanks for confirming. I am closing this issue.

[StingyJack]

So if I read this correctly the problem that I originally was discussing - where unit tests fail to be discovered due to an assembly reference issue, and do so silently - is not going to be fixed thanks to some (fairly deft) interdepartmental juggling.

[mishra14]

@StingyJack NuGet has been moving towards PackageReference type projects because it has a much better fundamental design as compared to Packages.Config. Please consider moving to PackageReference, where this issue does not exist.

If you want to keep using Packages.Config then please add all the package references from your product code project into the test project. That will unblock you.

[StingyJack]

I wasn't blocked. Once i realized that half the unit tests were not being executed for gated checkin builds, working around it was done in short order. But its every single test project that gets created that I have to either fix or show someone how to fix, or we are at risk of unintentionally delivering broken code to a customer.

You are leaving me with two choices really; use a package management system that is going to cost me a lot of time, that is buggy, and that has less features, or continue the manual hack I have now in perpetuity, So I can use MSTest and have it discover and run all the tests that are actually present.

Doesnt seem right...