Docker rate limit hit when using Ubicloud runners

[smcgivern]

We're seeing this when using Ubicloud runners to pull the public Node image from Docker Hub:

 /usr/bin/docker pull node:20.11
  Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

GitHub-hosted runners have an exemption from this limit for Docker Hub: actions/runner-images#1445 (comment)

Is there a way to do something similar with the Ubicloud runners? Or otherwise provide credentials? https://github.com/orgs/community/discussions/76636 suggests we can't provide credentials 😞

[dunklesToast]

Having the exact same issue. For my own docker build commands I've now create an authorized dockerhub account but for container actions this is (as you've said) not possible. It seems that this issue only happens sometimes so I guess retrying until you hit a runner which isn't rate limited or switch to GitHub Actions for the moment are the only workarounds.

All these actions are created from a matrix so they are essentially the same and require this action: https://github.com/appleboy/ssh-action. For some the pull worked, for others it did not.

[fdr]

Hello there,

We've gotten several recent reports of this. Something has changed recently, though we're not sure what. We're currently considering one theory, involving a coarser ipv6 aggregation of quota at #2247.

Our best idea of what to do does involve logging in. We think this works, we'll have to review that thread to see what specifically doesn't work. Expect more updates...

[dunklesToast]

To give more hints what's not working:
If you're using a GitHub Action that's based on a Docker container it needs to be built on job startup. Therefore the runner pulls the image from Docker Hub. Unfortunately there is currently no way to login to those setup builds.

[fdr]

Got it. It looks like our move for this is to disable IPv6 wholesale, because the aggregation of quota for IPv6 is coarser than IPv4 (the on IPv6 it aggregates a whole /64, which all VMs on a machine share).

We might come back to it and try to snip it out just for registry.docker.com so that other connections can be dual-stacked, and then we'll probably have to look into a crash project to enable mirroring.

[enescakir]

We resolved the root cause of DockerHub rate limit issue.

RCA: Docker has a limit of 100 pulls every 6 hours per IP address for anonymous users. The Docker client randomly uses either IPv4 or IPv6. Thus, IPv4 and IPv6 have separate limits. However, Docker only considers the first 64 bits of the IPv6 address for this limit, not the entire 128 bits. As a result, the limit applies to the entire host rather than just the individual virtual machine's IPv6 address on the host (https://www.docker.com/blog/beta-ipv6-support-on-docker-hub-registry/). This makes it easy to exceed the limit when the Docker client uses IPv6. To quickly address this issue, we've disabled IPv6 for our runner virtual machines (#2247). Now that the issue is resolved, we plan to explore disabling IPv6 only for Docker requests.

[fdr]

Update: received word from docker that IPv6 quotas were not enforced as documented until today. So, this was the cause.

[dunklesToast]

Is it possible that this is again an issue? Our builds are now failing due to docker buildx not being able to pull the moby image:

Error response from daemon: No such image: moby/buildkit:buildx-stable-1

running a simple docker pull moby/buildkit:buildx-stable-1 on not-ubicloud machines leads to no issues so this seems to be runner specific. One StackOverflow Comment and a discussion on the setup-buildx-action repo also suggests that this might be related to rate limiting. Unfortunately you cannot auth the docker pull commands for the setups so I cannot really test if the rate limiting is the real cause but it's the most plausble one right now. Is anybody else experiencing this?

[enescakir]

DockerHub had an incident and they say it’s resolved.
https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/677ea803d93e4505cabcb2f6

Are you still experiencing the issue?

[dunklesToast]

DockerHub had an incident and they say it’s resolved. dockerstatus.com/pages/incident/533c6539221ae15e3f000031/677ea803d93e4505cabcb2f6

Are you still experiencing the issue?

Nope just ran a new run and it passed. Sorry for that - dockerhub showed code green when I checked their page and since it worked on another machine I just thought couldn't be them 😅