Add additional tests for package namespaces (PackageReference) restore

[kartheekp-ms]

NuGet/NuGet.Client#4149 (review)

Similarly to packages.config, we should a bunch of other scenarios.

For example:

• Add tests for RestoreSources in project file.
• Add tests for dotnet add package, which calls restore itself.
• Add tests restore -source. (This comment will be addressed in Support sourceUri passed on nuget restore/install and dotnet restore package or other equivalent commands #10948)

Update:

• dotnet add package tests including RestoreSources and -SourceUri dotnet add package tests for namespaces feature NuGet.Client#4225
• PMC tests - added one install and one update PMC test in https://github.com/NuGet/NuGet.Client/tree/dev-kartheekp-ms-pmc-tests-pr-namespaces
• Restore Sources tests for nuget.exe and dotnet.exe. - Restore Sources tests for nuget.exe are also added by Erick in Support sourceUri passed on nuget restore/install and dotnet restore or RestoreSources set for msbuild /t:restore command, nuget restore NuGet.Client#4214. I added couple of tests in the PR linked to this issue to tests dotnet.exe restore sources scenario.
• Source option tests for nuget.exe and dotnet.exe. @erdembayar added tests for these scenarios in Support sourceUri passed on nuget restore/install and dotnet restore or RestoreSources set for msbuild /t:restore command, nuget restore NuGet.Client#4214
• End to end tests (EndToEnd or Apex specifically doesn't matter), Implicit restore/solution restore. Instlal/Update are covered in an above point.

[nkolev92]

Moved this to the next sprint.

I would prioritize these tasks for the next sprint @kartheekp-ms

[nkolev92]

Epics shouldn't have estimates themselves, but rather add up from the issues owned by it.

[nkolev92]

@kartheekp-ms What is leftover here?

I tweaked in order to make planning obvious. Adding back the estimate.

[kartheekp-ms]

Made some progress on this task: Added couple of tests for PMC Install/Update package scenarios for source mapping feature https://github.com/NuGet/NuGet.Client/tree/dev-kartheekp-ms-pmc-tests-pr-namespaces

I will create PR's next sprint.

[nkolev92]

Please note that @erdembayar has a task about PMC tests in #11187.

Can you please consolidate and ensure that you're both not doing the same work/have dup issues?

Thanks!

[kartheekp-ms]

@erdembayar here is the progress made on this issue so far.

• dotnet add package tests including RestoreSources and -SourceUri dotnet add package tests for namespaces feature NuGet.Client#4225
• PMC tests - added one install and one update PMC test in https://github.com/NuGet/NuGet.Client/tree/dev-kartheekp-ms-pmc-tests-pr-namespaces - The new 2 tests passed on CI. As Nikolche mentioned above, you can combine this work with [Bug]: Create PMC install -source tests for PackageReferenence type projects #11187
• Restore Sources tests for nuget.exe and dotnet.exe.
• Source URI tests for nuget.exe and dotnet.exe.
• End to end tests (EndToEnd or Apex specifically doesn't matter)

Please let me know if I am missing anything.

[erdembayar]

@kartheekp-ms @nkolev92 @aortiz-msft
I tried to create test cases for PR type for Apex, End2end test.
But it looks we need whole new way of testing for package source mapping. Because each test for PR restore automatically put restored package into GPF but it pollutes restore environment for next package source mapping test.
So we have 3 choices:

• Remove restored packages from GPF at end of test. But there're transitive packages so not straight forward. Most likely not practical.
• Create new GPF with prepopulated necessary packages for each test then clean up at end of each test. But it'll increase our repository size by 100Mbs and it'll slow our tests.
• Each test use new unique packages. Most likely not practical.

Maybe put it on hold for now, we need discussion next week on this.

[kartheekp-ms]

@erdembayar - Can we use a custom globalPackageFolder setting the nuget.config file for every test so that the machines folders are untouched?

https://github.com/NuGet/Samples/blob/main/PackageSourceMappingExample/nuget.config#L6

[erdembayar]

@erdembayar - Can we use a custom globalPackageFolder setting the nuget.config file for every test so that the machines folders are untouched?

https://github.com/NuGet/Samples/blob/main/PackageSourceMappingExample/nuget.config#L6

That is the 2nd option I proposed. In order to build NetCoreConsoleApp we need prepopulate Microsoft.NETCore.App, microsoft.codeanalysis.csharp etc quite many packages.

[erdembayar]

@kartheekp-ms
I noticed allmost all except 1 netcore Apex tests are skipped/disabled due to being flaky test. Not sure if I add more test then it may add more flaky tests.

For example:
https://github.com/NuGet/NuGet.Client/blob/1af032a9a12c530ef434e9ea1a53cd93348e1146/test/NuGet.Tests.Apex/NuGet.Tests.Apex/NuGetEndToEndTests/NuGetConsoleTestCase.cs#L202-L209

https://github.com/NuGet/NuGet.Client/blob/1af032a9a12c530ef434e9ea1a53cd93348e1146/test/NuGet.Tests.Apex/NuGet.Tests.Apex/NuGetEndToEndTests/NetCoreProjectTestCase.cs#L31-L38

[erdembayar]

Maybe we need to find solution leveraging Dartlab(download needed packages during machine provision instead of adding to repository) for this since Apex tests are moving to 1ES.

[nkolev92]

We could consider adding our own templates, potentially ones that target .NET Framework instead.

Updating the used apex version might be a good idea s well.

We could investigate how other people run tests like that.

In general though, we should isolate the tests from impacting the global packages folder.

It really is the same feedback we've often had on the unit/integration tests themselves.