Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub #42026

Open
akoeplinger opened this issue Jul 8, 2024 · 9 comments
Assignees
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Milestone

Comments

@akoeplinger
Copy link
Member

akoeplinger commented Jul 8, 2024

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build Definition Test Pull Request
928807 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45836
928744 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46063
928709 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46183
928689 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46255
928609 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46281
928598 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46275
928587 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46280
928563 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #44843
928552 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46278
928528 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45419
928513 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46270
928195 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46244
928418 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46254
928112 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46261
928278 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46275
928008 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46249
927996 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46264
927657 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46261
927793 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46244
927012 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46250
927742 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46258
927641 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46184
927625 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46259
927610 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46257
927711 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46262
927509 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46183
927622 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46258
927208 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46238
927121 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46184
926917 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46248
926765 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46101
926897 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45836
926264 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46231
926244 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46184
926219 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46198
926210 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46227
926195 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46226
926156 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45552
925622 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46218
925532 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46216
925397 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45932
925350 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45932
925341 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46211
925289 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46164
925295 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46045
925292 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46210
925259 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45362
925244 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46208
925046 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46183
924829 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline #46196
924849 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46201
924825 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46149
923205 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46141
921210 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46090
920711 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46082
920700 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46067
920705 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline #46081
920682 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #46077
920725 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46034
920695 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46080
920719 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45979
920657 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45980
920684 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #46078
919477 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46036
919432 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #42416
919399 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46031
919342 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46021
919372 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46026
919351 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46022
919377 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45990
919353 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46024
919352 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46023
918778 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46008
918610 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45883
918546 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45986
917903 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45948
917396 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45948
916466 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45946
916462 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45944
916464 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45945
917382 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #44611
916504 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45917
916487 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45948
916480 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45947
916184 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45873
916209 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45932
916203 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45937
916152 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45917
915803 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45345
915795 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45442
915745 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45549
915666 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45906
915374 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
915236 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45923
915191 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #45919
915232 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
915231 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
915197 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #45921
914007 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #45886
914006 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
Displaying 100 of 145 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
27 53 145
@akoeplinger
Copy link
Member Author

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@dotnet/sdk-container-builds-maintainers

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@nagilson
Copy link
Member

Bumping this thread since it's affecting more PRs

@nagilson
Copy link
Member

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

@baronfel
Copy link
Member

Could be fixed at a root cause level by dotnet/dnceng#3389

@MichaelSimons
Copy link
Member

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

@nagilson
Copy link
Member

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

@marcpopMSFT
Copy link
Member

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.

@marcpopMSFT marcpopMSFT added this to the 10.0.1xx milestone Dec 18, 2024
@marcpopMSFT marcpopMSFT added the Area-Containers Related to dotnet SDK containers functionality label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Projects
None yet
Development

No branches or pull requests

6 participants