Basic auth issues with NuGet 3.1.1 in Visual Studio 2015

[yishaigalatzer]

From @martinsuchan

Hi, we're using custom NuGet repository secured by basic authentication and we've also started developing UWP Windows 10 apps that use the new project.json NuGet config file. The problem is that **NuGet 3 is unable to successfully restore

packages because the restore process is ignoring provided basic authentication credentials**.

I've created simple repro available here: http://1drv.ms/1hvdfFT
How to use it:
run the 0_InitCustomSource.ps1 file to add reference to test NuGet server located on
http://nugetbug.azurewebsites.net/nuget -UserName "nugetbug" -Password "bugbug"
The server contains two NuGet pacakges, LibCore and LibMain that depends on LibCore, both for Windows 8.1 apps.
The sample also contains UWP project referencing LibMain.

Expected behavior: when I open the project without project.lock.json, click Rebuild - all packages are properly restored, the build works on first try.
Current behavior: The build fails with this error message:

Restoring NuGet packages...
To prevent NuGet from restoring packages during build, open the Visual Studio Options dialog, click on the Package Manager node and uncheck 'Allow NuGet to download missing packages during build.'
Failed to retrieve information from remote source 'http://nugetbug.azurewebsites.net/'.
Response status code does not indicate success: 401 (Unauthorized).
Failed to retrieve information from remote source 'http://nugetbug.azurewebsites.net/'.
Response status code does not indicate success: 401 (Unauthorized).
1>------ Rebuild All started: Project: NuGetTest, Configuration: Debug ARM ------
1>  NuGetTest -> C:\Users\martin.suchan\Desktop\NuGet.bug\NuGetTest\bin\ARM\Debug\NuGetTest.exe

This currently breaks our build environment and it also makes it hard to impossible to do clean checkout and just build our project, this is basically show-stopper right now.
Possible workaround could be disabling the basic authentication on our NuGet server or caching the project.json.lock in source control, none of which is a good solution.

Tested on Windows 10, VS2015 with Win10 SDK, all RTM, and NuGet package manager 3.1.60724.766.

[yishaigalatzer]

@martinsuchan I assume that this works fine for package.config scenarios?

[martinsuchan]

We never had any troubles with older projects using package.config, because we stored all required NuGet packages in our Git repo and the restore feature was not necessary.
This changed with the new .NET Core model where the build process needs about hundred Microsoft packages to build even blank UWP project. The build process also restores all other referenced NuGet packages to the same shared folder including packages from our custom feed.
It would be best if NuGet v3 allowed us to store packages from custom feed in our Git Repo and .NET Core packages in shared .nuget folder, or something like that, or if it just worked with the basic authentication.

[martinsuchan]

By the way, when adding these packages using NuGet Package manager in VS, the behavior is also a bit strange. I got three NuGet sources in settings - default NuGet, our private feed and the test feed. This is how it looks when adding LibMain specifically from our test feed to project.

It makes requests to ALL feeds searching for the package name and does full package restore of all .NET Core packages even when adding package from custom feed with no other dependencies. Not sure, if this is the expected behavior now...

Restoring packages for 'NuGetTest'.
Restoring packages for C:\Users\martin.suchan\Desktop\NuGet.bug\NuGet.bug\NuGetTest\project.json...
Using source http://nugetbug.azurewebsites.net/nuget.
Using source https://api.nuget.org/v3/index.json.
Using source http://[ourfeed].azurewebsites.net/nuget.
Using source https://www.nuget.org/api/v2/curated-feeds/microsoftdotnet/.
Restoring packages for UAP,Version=v10.0...
GET http://nugetbug.azurewebsites.net/nuget/FindPackagesById()?id='LibMain'.
GET http://[ourfeed].azurewebsites.net/nuget/FindPackagesById()?id='LibMain'.
GET https://www.nuget.org/api/v2/curated-feeds/microsoftdotnet/FindPackagesById()?id='LibMain'.
GET https://az320820.vo.msecnd.net/v3-flatcontainer/libmain/index.json.
OK http://nugetbug.azurewebsites.net/nuget/FindPackagesById()?id='LibMain' 216ms
CACHE http://nugetbug.azurewebsites.net/api/v2/package/libmain/1.0.0
Resolving conflicts for UAP,Version=v10.0...
Installing LibMain 1.0.0.
Completed installation of LibMain 1.0.0
Scanning packages for runtime.json files...

[yishaigalatzer]

Yes this is the expected behavior. Since it doesn't know where the package is supposed to come from, it goes out (in parallel) to all feeds.

Suggestion - You want to disable the curated feeds, they are just a duplicate of the main feed and cause extra unnecessary calls.

[yishaigalatzer]

@feiling please validate also package.config scenarios against this server

[martinsuchan]

@yishaigalatzer Since it doesn't know where the package is supposed to come from, it goes out (in parallel) to all feeds.
Uhm I don't agree: I've selected in VS in NuGet Package Manager my private NuGet feed and selected package from that server. At this point it has all information required to check only the specific server for the package, not all sources. Or am I missing something here?

Regarding NuGet Restore in project using package.config, I've just tested it, it works with my private feed using basic authentication. It fails only for the new projects using project.json.

[davidfowl]

@martinsuchan Your dependencies might come from other configured feeds. Are all of your packages on the private feed? If yes then why isn't that configured in a solution level nuget.config file?

[martinsuchan]

It has dependencies on other feeds, but that does not answer the question why it asks for LibMain using all feeds?

...
GET http://nugetbug.azurewebsites.net/nuget/FindPackagesById()?id='LibMain'.
GET http://[ourfeed].azurewebsites.net/nuget/FindPackagesById()?id='LibMain'.
GET https://www.nuget.org/api/v2/curated-feeds/microsoftdotnet/FindPackagesById()?id='LibMain'.
GET https://az320820.vo.msecnd.net/v3-flatcontainer/libmain/index.json.
...

[davidfowl]

Sure that could be an optimization but that's not currently how it's implemented. project.json restore is very different from the way package install used to work.

[yishaigalatzer]

@martinsuchan I see your point. it goes a bit beyond optimization, I assume you are expecting that package to only come from the feed you specified. I'm going to try and explain the process here, I hope it makes it clearer what is going on.

This process applies only to transitive restore or to projects that use project.json.

@davidfowl feel free to edit/correct

Install process

1. You choose a package via search, search is the only thing limited to a feed. Once you picked a package (and version), we attempt to write it to the project.json file, without figuring out any of its dependencies.
2. We start a restore process (details below) if it fails we roll back the changes to project.json
3. We never touch your project files, and we never write into the project.json any of the dependencies.

Restore process

1. We check if a change was made to the project.json file or one of the dependent projects, we start restoring
2. We figure out the dependency graphs for the whole project in one go.
3. We restore missing packages only.
4. We write a lock file specifying the package graph and what dlls are picked from each package
5. Visual Studio/msbuild reads the lock file and adds references in memory.

It is rather complicated to figure out diffs and subgraphs to spend less time on the network, but like @davidfowl says it might be something to look into in the future.

Either way - We are going to work on fixing the basic auth issue shortly, and once this bug is closed feel free to ping me, and I can try and share private bits with you.

[martinsuchan]

I see from your description, that the new package restore mechanism wasn't designed to handle effectively several custom feeds.
What happens for instance if I got LibMain in my private feed, but another different LibMain is also available in official NuGet feed? This is not hypothetical scenario, some teams put all packages they use in their private feeds including common packages like Json.NET, etc.

The package restore should be aware of the source where I selected specific package, but this goes beyond my original problem. For now I'll be happy if I and our TeamCity build server can restore all packages from a clean checkout.

[yishaigalatzer]

This has a similar problem with the existing mechanism, ordering doesn't solve the problem it makes it seem like it does.

The real answer here, is to namespace the in company packages (or similarly the in team packages, when multiple teams share a internal nuget server).

[RanjiniM]

The source is reachable but building takes forever and doesn't finish. Assigning to Fei to take a look

[mathikp]

Hi, I have faced the below issue while install my UWP package from my own private feed.

PM> Install-Package
Restoring packages for 'App4'.
Restoring packages for c:\users\mathi\documents\visual studio 2015\Projects\App4\App4\project.json...
Warning: FindPackagesById:
Response status code does not indicate success: 404 (Not Found).
Warning: FindPackagesById:
Response status code does not indicate success: 404 (Not Found).
NotFound https://api.nuget.org/v3-flatcontainer//index.json 1430ms
Install-Package : Failed to retrieve information from remote source ''.
Response status code does not indicate success: 404 (Not Found).
At line:1 char:1

• Install-Package

• - CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
  - FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
  
  

is this issue fixed in 3.3.0.167 NuGet Package manager version?

[yishaigalatzer]

Hard to tell from this question, why won't you update to 3.3?

[yishaigalatzer]

Mathikp your report seems not related to this issue. Can you please open a new issue and provide a fiddler trace as well as information about the type of private feed you are using and the source you are using?

[qiansen1386]

Warning: FindPackagesById: NuGet.CommandLine
  Response status code does not indicate success: 401 (Unauthorized).
  GET http://nugetrepo.misatravel.net/FindPackagesById()?id='NuGet.CommandLine'
  GET https://www.nuget.org/api/v2/package/NuGet.CommandLine/3.3.0
Warning: FindPackagesById: NuGet.CommandLine
  Response status code does not indicate success: 401 (Unauthorized).
  GET http://nugetrepo.misatravel.net/FindPackagesById()?id='NuGet.CommandLine'
Error: FindPackagesById: NuGet.CommandLine
  Response status code does not indicate success: 401 (Unauthorized).