NuGet restore fails with The repository primary signature validity period has expired

[garath]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2153225

Build leg reported

Build

Pull Request

https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/30063

{
   "ErrorMessage" : "The repository primary signature validity period has expired",
   "BuildRetry": true,
   "ErrorPattern": "",
   "ExcludeConsoleLog": false
}

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
283331	dotnet/runtime	Build HttpStress	Log
2181170	dotnet-diagnostictests	Build - Windows x86 $(NetCoreVersion) on Windows	Log
260336	dotnet/roslyn	Build - Validate determinism	Log	dotnet/roslyn#67807
253865	dotnet/roslyn	Build - Validate determinism	Log
253696	dotnet/roslyn	Build - Validate determinism	Log
253258	dotnet/runtime	Build HttpStress	Log

Build	Definition	Test	Pull Request
270758	dotnet/performance	Partition2.WorkItemExecution	dotnet/performance#3019

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	7

[garath]

ICM on this issue from another team: ICM 380022563.

[dtivel]

@garath, does https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#retry-untrusted-root-failures help?

[dougbu]

I'm going to try something out in dotnet-helix-machines…

[garath]

I'm going to try something out in dotnet-helix-machines…

What is that? Damon's suggestion?

[dougbu]

What is that? Damon's suggestion?

After a conversation w/ @ilyas1974 and @riarenas, our initial thought was to upgrade the .NET SDK (and therefore the dotnet nuget command) used in dotnet-helix-machines. However, a number of the reports above involve very recent SDK versions. An update may solve some issues but likely not this one.

@dtivel's suggestion is about NU3028 but the error is actually https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3037. That said, I'll try it out.

If that doesn't work, the DOTNET_NUGET_SIGNATURE_VERIFICATION=false environment variable setting mentioned in dotnet/sdk#31868 sounds scary but may be necessary.

[dtivel]

@dtivel's suggestion is about NU3028 but the error is actually https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3037. That said, I'll try it out.

The timestamp's trust fails, so the timestamped signature remains expired. A trusted timestamp basically allows a timestamped signature to not expire.

DOTNET_NUGET_SIGNATURE_VERIFICATION has no effect whatsoever on Windows. Also, it's unlikely that this is a problem on Linux. If you find that it is, let me know.

[dougbu]

@garath, does https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#retry-untrusted-root-failures help?

Yes it seems to make a significant difference, at least w/ the 6.0.404 .NET SDK.

@garath thoughts on putting this workaround into job.yml or tools.ps1 and tools.sh❔ Third option would be to pass this success along to the dotnet/performance, roslyn, runtime, and sdk teams.

[dougbu]

@garath, does https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#retry-untrusted-root-failures help?

Yes it seems to make a significant difference, at least w/ the 6.0.404 .NET SDK.

@garath thoughts on putting this workaround into job.yml or tools.ps1 and tools.sh❔ Third option would be to pass this success along to the dotnet/performance, roslyn, runtime, and sdk teams.

/ping @garath. Our customers are still hitting this though helix machines is unblocked w/ the following workaround in place

  # Retry signature validation up to three times, waiting 2 seconds between attempts.
  # See https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#retry-untrusted-root-failures
  - name: NUGET_EXPERIMENTAL_CHAIN_BUILD_RETRY_POLICY
    value: 3,2000

Should we put this into our eng/common/job.yml or reach out to affected customers❔

/cc @dotnet/dnceng for broader visibility since I'll be OOF tomorrow and so will @garath…

[premun]

Should we put this into our eng/common/job.yml or reach out to affected customers❔

I think we'd need to do both. I suspect our job templates might not be utilized in repositories such as runtime.

[garath]

The ICM was closed with no resolution even though @dtivel jumped in and gave a lot of good info.

I'm not sure what the best thing to do here is. From the Known Issue stats it seems like the impact on the builds might not be too bad?

[dougbu]

I'm not sure what the best thing to do here is.

I have a vague feeling that changing job.yml won't hurt anything and might mostly avoid flare-ups like we experienced a month or so ago. the workaround is minimal and has more words in the comments than the actual addition 😀

[garath]

Sounds good to me. PRs welcome!

[dougbu]

Sounds good to me. PRs welcome!

sorry, just one PR: #13659

[dougbu]

/fyi ICM 380022563 was just reopened because another team hit the same issue. turns out NuGet feels they're only able to work around a problem in Windows itself. I believe Windows has no fix (yet❔).

[dougbu]

could someone here please approve #13659❔ would be nice to put this longstanding "FR" issue behind us

[dougbu]

fix is now in (thx @AlitzelMendez). is any validation needed before closing this issue❔

[dougbu]

Completed w/ ab7df88