[Bug]: NuGet package parsing fails

[AaronFriel]

NuGet Product Used

dotnet.exe

Product Version

3.1.x

Worked before?

3.1.x

Impact

It's more difficult to complete my work

Repro Steps & Context

nuget restore on a package using Pulumi, and voila:

• https://github.com/pulumi/pulumi/runs/6872537319?check_suite_focus=true#step:17:38
• https://github.com/pulumi/pulumi-yaml/runs/6869776968?check_suite_focus=true#step:13:1361

I've seen this error cause at least 3 build pipelines to fail. The two above I had in open tabs, there almost certainly are more failures that I haven't directly observed.

dotnet restore dotnet.sln
  Determining projects to restore...
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json'.
  The content at 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json' is not a valid JSON object.
    The JSON document is not an object.
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json'.
  The content at 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json' is not a valid JSON object.
    The JSON document is not an object.
/home/runner/.dotnet/sdk/3.1.419/NuGet.targets(128,5): error : Failed to retrieve information about 'Microsoft.Extensions.Options' from remote source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json'. [/home/runner/work/pulumi/pulumi/sdk/dotnet/dotnet.sln]
/home/runner/.dotnet/sdk/3.1.419/NuGet.targets(128,5): error :   The content at 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json' is not a valid JSON object. [/home/runner/work/pulumi/pulumi/sdk/dotnet/dotnet.sln]
/home/runner/.dotnet/sdk/3.1.419/NuGet.targets(128,5): error :   The JSON document is not an object. [/home/runner/work/pulumi/pulumi/sdk/dotnet/dotnet.sln]

See also:

• https://github.com/NuGet/docs.microsoft.com-nuget/issues/2768

Verbose Logs

No response

[viktorrealt]

I have same problem, when try to restore project in Docker.
An error occurred while sending the request.
Unable to read data from the transport connection: Connection reset by peer.
Connection reset by peer or random 503, 500 errors
Response status code does not indicate success: 502 (Bad Gateway).

[ClaytonHunsinger]

I am also seeing these kinds of errors. Yesterday I had pipelines working no problem, and then suddenly they started failing even though no changes were made to the packages or any other Nuget configurations.
I rarely am getting 5xx responses, but I have had a couple of them. I am mostly seeing my pipelines fail due to 404's.

Here is the pipeline error I am seeing a lot (the package name that fails is random):

/usr/share/dotnet/sdk/3.1.419/NuGet.targets(128,5): error : Failed to retrieve information about 'Humanizer.Core.cs' from remote source 'https://api.nuget.org/v3-flatcontainer/humanizer.core.cs/index.json'. [/path/to/myproject.csproj]
/usr/share/dotnet/sdk/3.1.419/NuGet.targets(128,5): error :   The content at 'https://api.nuget.org/v3-flatcontainer/humanizer.core.cs/index.json' is not a valid JSON object. [/path/to/myproject.csproj]
/usr/share/dotnet/sdk/3.1.419/NuGet.targets(128,5): error :   The JSON document is not an object. [/path/to/myproject.csproj]

Sometimes if I try to follow the URL in my browser I get valid JSON back. Other times I get XML back with a "BlobNotFound" error in the details, but like I said it's totally random.

[jakubadamek]

Same here. Fails quite often but not always.

[mcraiha]

I am also having issues when running in docker. No problems with host. I am getting following error with nslookup

/ # nslookup api.nuget.org
Server:         172.21.112.1
Address:        172.21.112.1:53

Non-authoritative answer:
*** Can't find api.nuget.org: Parse error

Non-authoritative answer:
*** Can't find api.nuget.org: Parse error

[mike-geiger]

I'm getting this only while running from our build server, only for one package. All logged lines for the Hangfire.InMemory package copied below. I'm able to restore this package locally on windows, in a Ubuntu WSL2 after clean and clearing nuget caches. The URL opens just fine for me. We just started seeing this within the last day.

2022-06-14T12:44:59.7429407Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:44:59.8126899Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 12ms
2022-06-14T12:44:59.8130093Z          Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'.
2022-06-14T12:44:59.8131476Z          The content at 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json' is not a valid JSON object.
2022-06-14T12:44:59.8132535Z            The JSON document is not an object.
2022-06-14T12:44:59.8133491Z          Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled.

2022-06-14T12:45:00.8535574Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:45:00.8589007Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 8ms
2022-06-14T12:45:00.8597247Z          Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'.
2022-06-14T12:45:00.8599088Z          The content at 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json' is not a valid JSON object.
2022-06-14T12:45:00.8600153Z            The JSON document is not an object.
2022-06-14T12:45:00.8601236Z          Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled.

2022-06-14T12:45:01.9518272Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:45:01.9623829Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 10ms
2022-06-14T12:45:01.9625637Z          Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'.
2022-06-14T12:45:01.9627874Z          The content at 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json' is not a valid JSON object.
2022-06-14T12:45:01.9629071Z            The JSON document is not an object.
2022-06-14T12:45:01.9630392Z          Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled.

2022-06-14T12:45:03.1064113Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:45:03.1221192Z          Installed System.Collections.Immutable 6.0.0 from https://api.nuget.org/v3/index.json with content hash l4zZJ1WU2hqpQQHXz1rvC3etVZN+2DLmQMO79FhOTZHMn8tDRr+WU287sbomD0BETlmKDn0ygUgVy9k5xkkJdA==.
2022-06-14T12:45:03.1224246Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 17ms
2022-06-14T12:45:03.1306957Z          Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'.
2022-06-14T12:45:03.1308734Z          The content at 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json' is not a valid JSON object.
2022-06-14T12:45:03.1310136Z            The JSON document is not an object.

2022-06-14T12:45:04.2152224Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:45:04.3058425Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 97ms
2022-06-14T12:45:04.3330523Z          Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'.
2022-06-14T12:45:04.3332075Z          The content at 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json' is not a valid JSON object.
2022-06-14T12:45:04.3333022Z            The JSON document is not an object.

2022-06-14T12:45:05.5181161Z            GET https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json
2022-06-14T12:45:05.5275881Z            OK https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json 9ms
2022-06-14T12:45:05.5362350Z      1>/usr/local/ado-agent/_work/54/s/src/Tests/Deal.UnitTests/Deal.UnitTests.csproj : error NU1301: Failed to retrieve information about 'Hangfire.InMemory' from remote source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'. [/usr/local/ado-agent/_work/54/s/src/Eclipse.Deal.sln]
2022-06-14T12:45:05.5366647Z      1>/usr/local/ado-agent/_work/54/s/src/Deal.API/Deal.API.csproj : error NU1301: Failed to retrieve information about 'Hangfire.InMemory' from remote source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'. [/usr/local/ado-agent/_work/54/s/src/Eclipse.Deal.sln]

2022-06-14T12:45:07.3141312Z Build FAILED.
2022-06-14T12:45:07.3141777Z 
2022-06-14T12:45:07.3143519Z        "/usr/local/ado-agent/_work/54/s/src/Eclipse.Deal.sln" (Restore target) (1) ->
2022-06-14T12:45:07.3144888Z        (Restore target) -> 
2022-06-14T12:45:07.3147527Z          /usr/local/ado-agent/_work/54/s/src/Tests/Deal.UnitTests/Deal.UnitTests.csproj : error NU1301: Failed to retrieve information about 'Hangfire.InMemory' from remote source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'. [/usr/local/ado-agent/_work/54/s/src/Eclipse.Deal.sln]
2022-06-14T12:45:07.3151147Z          /usr/local/ado-agent/_work/54/s/src/Deal.API/Deal.API.csproj : error NU1301: Failed to retrieve information about 'Hangfire.InMemory' from remote source 'https://api.nuget.org/v3-flatcontainer/hangfire.inmemory/index.json'. [/usr/local/ado-agent/_work/54/s/src/Eclipse.Deal.sln]
2022-06-14T12:45:07.3153857Z 
2022-06-14T12:45:07.3154776Z     0 Warning(s)
2022-06-14T12:45:07.3155613Z     2 Error(s)
2022-06-14T12:45:07.3156078Z 
2022-06-14T12:45:07.3156926Z Time Elapsed 00:00:16.12

[mkline1554]

Same here. Doesn't always fail but seems to most of the time

[joelverhagen]

Hello, thanks you for taking the time to report this. This is a duplicate of NuGet/NuGetGallery#9140. We'll use that one issue to track the api.nuget.org connectivity problem as well as the JSON corruption issue.

As of now, we have performed a failover for our CDN provider which we expect to mitigate the problem. If you are still facing a problem, please post on the issue I linked above.

[bakgaard]

Hi @joelverhagen
It happenede again yesterday while building on GitHub. Is it the same issue?

I can't get Microsoft, Nuget, or Google packages although they are specified in my <trustedSigners>-section.
Removing this section makes the build run smooth, but it's there for a reason.

[joelverhagen]

Hey @bakgaard, what you're describing appears to be related to the <trustedSigners> config. The issue described, at least to my eyes, appears to be a NuGet.org availability issue. I don't recall the full details since it was a while back but it seems like a CDN problem we had.

Could you open a separate issue on NuGet/Home with your config, the error message, SDK version, etc?

I suggest NuGet/Home not NuGet/NuGetGallery since that's a client feature. Since it's impacting the success of your workflow we should involve the NuGet client team first instead of the NuGet server team.