Error response from daemon: toomanyrequests: Too Many Requests (HAP429)

[asntcrz]

Hi,

Thank you for providing the support to every single case when this appears.

In my case it happened all of the sudden, without any changes on sysadmin so I have no idea how to determine if I have a loose script running wild with the requests, would you please be so kind to provide some stats, or possible causes? Project is hosted on VPS with IP 51.75.74.139.

I can't docker push anything, not even login or pull:

docker login -u ****
Password:
Error response from daemon: login attempt to https://registry-1.docker.io/v2/ failed with status: 429 Too Many Requests

docker pull nginx
Using default tag: latest
Error response from daemon: toomanyrequests: Too Many Requests (HAP429).

Thank you!

[jcarter3]

What VPS provider are you using?

[asntcrz]

Its OVH (located in Frankfurt Germany)

[aep-feha]

I have the same issue with my OVH VPS as well, and it's in the same location, Frankfurt.

[tobsen987]

Do you have IPv6 enabled on your VPS? I guess it has something to do with ubicloud/ubicloud#2244 (comment).

[asntcrz]

ok disabling ipv6 in my host did the trick, followed these steps (https://webshanks.com/how-to-disable-ipv6-on-ubuntu/):

sudo nano /etc/sysctl.conf

Add at the end:

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

And apply:
sudo sysctl -p

But I'm not quite sure of the implications that this will have on our systems...

Ideally docker hub will fix the IPv6 rate limiter to take into account the whole 128 bits and not the first 64, blocking entire hosts, rather than individuals, right? as discussed here ubicloud/ubicloud/discussions/2244

[jcarter3]

This is a delicate balance - rate limiting IPv6 is not straightforward, and we're trying to find the right path. We've seen many network setups where individual machines are granted a whole /64 space, with each individual network request coming from a different IP - this makes the system incredibly fragile to abuse.

[davyhollevoet]

Same issue, also on OVH.

I added one of the IPv4 addresses of registry-1.docker.io to /etc/hosts to work around the issue for now.

[joachimtingvold]

I'm hitting this issue as well. Disabling IPv6 for my OVH VPS is not really an option (as I consume & provide services on IPv6).

As it is now, I cannot use Docker Hub at my OVH VPS.

You should as a minimum separate the rate limit for logins vs. everything else (so that you at least can login from rate-limited networks). For logged in users, the rate-limit should follow the account, and not the IP (i.e. if you log in from a client within a rate-limited /64 prefix, you should not get rate-limited anymore). If you still see abuse and/or high request rates from logged in users, you can more easily ban/rate-limit these specific users, rather than affecting a whole range of innocent users.

Same issue, also on OVH.
I added one of the IPv4 addresses of registry-1.docker.io to /etc/hosts to work around the issue for now.

That doesn't seem to help, as the Docker Engine uses the built-in Go HTTP client that doesn't seem to respect /etc/hosts. At least that's what my current observations are. Even when setting /etc/resolv.conf to a DNS-server that only replies with A-records (a test/lab Power DNS Recursor with filtering to only respond with A-records, that I specifically set up to test this Docker rate-limit issue), it still does all docker login requests towards IPv6 addresses.

edit 1: Ended up using a proxy as the workaround, which seems to work fine.

edit 2: This did not properly work, as setting the daemon proxy also propagates these to the containers (which they then also use). You then have to configure the proxy before doing any login/pull requests, then remove them again before creating containers.

[brain-kode]

Same issue here with an OVS VPS. This fix by @asntcrz worked for a week but now the issue has come back.

ok disabling ipv6 in my host did the trick, followed these steps (https://webshanks.com/how-to-disable-ipv6-on-ubuntu/):

sudo nano /etc/sysctl.conf

Add at the end:

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

And apply: sudo sysctl -p

But I'm not quite sure of the implications that this will have on our systems...

Ideally docker hub will fix the IPv6 rate limiter to take into account the whole 128 bits and not the first 64, blocking entire hosts, rather than individuals, right? as discussed here ubicloud/ubicloud/discussions/2244

[joachimtingvold]

Same issue here with an OVS VPS. This fix by @asntcrz worked for a week but now the issue has come back.

Disabling IPv6 is hardly what I would call a fix.

[brain-kode]

Same issue here with an OVS VPS. This fix by @asntcrz worked for a week but now the issue has come back.

Disabling IPv6 is hardly what I would call a fix.

Sure, it's a temporary workaround but it allowed me to unblock a situation. Now the workaround I'm using is to push all the docker images I need into a private image registry, and I pull them from there.

[JoeAlamo]

Agree with @joachimtingvold - as a paying customer, can't the rate limits be tied to our authenticated account rather than us being collateral damage of IPv6 rate limiting across a wider prefix?

[viceice]

it's weired. i even get it for auth:

>  curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull"
Too Many Requests (HAP429).

disabling IPv6 isn't an options as it would block all IPv6 enabled services 😕

[viceice]

A workaround for bitnami charts and images is to use public.ecr.aws

global:
  # https://github.com/docker/hub-feedback/issues/2421
  imageRegistry: public.ecr.aws

[TheAkio]

Having the same same issue since around mid November. It's extremely unlucky that (apparently) the only "workaround" is completely disabling IPv6 on the system. There should be a way to tell Docker to use IPv4 when making requests to Docker Hub. That would at least be a proper workaround,

EDIT: I found a proper workaround (For me at least) - Blocking the IPv6 Addresses of Docker Hub in the firewall (Yes I'm still using the ip6tables interface, too lazy to switch):

ip6tables -A OUTPUT -d 2600:1f18:2148:bc00::/56 -j DROP

Might be blocking too much and I'm also not sure whether you need to block both input and output but pulling images was successful again after that.

EDIT 2: Blocking only output seems to do the trick

[brain-kode]

Disabling IPv6 doesn't work for me anymore

[Gandalf-Le-Dev]

Same, I don't know what are the options to go further

[xairoo]

Really painful!

[bizhao]

Disabling IPV6 doesn't work for me.
Cannot login because of 429..

ubuntu@node1:~$ docker login --username=bizhao
Password:
Error response from daemon: login attempt to https://registry-1.docker.io/v2/ failed with status: 429 Too Many Requests

[salfter]

Just ran into the same problem while updating a VPS at Linode this morning...the ip6tables block mentioned above worked for me.

[rezbouchabou]

You can use a mirror registry to bypass Docker Hub's rate-limiting.

Add a Docker Registry Mirror:

Edit the Docker daemon configuration file:

sudo nano /etc/docker/daemon.json

Add the following configuration to use a registry mirror:

{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

Restart Docker:

sudo systemctl restart docker

Test the container pull:

sudo docker run hello-world

[GuidoSchnieders]

ok disabling ipv6 in my host did the trick, followed these steps (https://webshanks.com/how-to-disable-ipv6-on-ubuntu/):

sudo nano /etc/sysctl.conf

Add at the end:

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

And apply: sudo sysctl -p

But I'm not quite sure of the implications that this will have on our systems...

Ideally docker hub will fix the IPv6 rate limiter to take into account the whole 128 bits and not the first 64, blocking entire hosts, rather than individuals, right? as discussed here ubicloud/ubicloud/discussions/2244

Worked for me! And after disabling IPv6 in my OVH VPS I was able to run the "docker login", which also failed with IPv6 enabled. With a successful login to the docker HUB I have enabled IPv6 again and now I am still allowed to i.e. pull images. So the limit is now counted against my (free) account and everything works fine!
Thanks Alejandro

[jbjardine]

I'm on OVH VPS (Germany), still have :
Error response from daemon: Get "https://registry-1.docker.io/v2/": toomanyrequests: Too Many Requests (HAP429).
I have done :

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

ip6tables -A OUTPUT -d 2600:1f18:2148:bc00::/56 -j DROP
And I added one of the IPv4 addresses of registry-1.docker.io to /etc/hosts

It's not working anymore 👎
Other solutions? Thanks for help!

[jbjardine]

I resolved the Too Many Requests (HAP429) issue by:

• Disabling IPv6 on my VPS to prevent Docker Hub from using IPv6 connections.
• Attaching an additional IPv4 address from another country (via OVH) and linking it to my VPS.

This setup allowed me to bypass Docker Hub's rate limits successfully.

@ovh

[ScorpioCoding]

Docker issue HAP429 to.
Got a response from docker to disable my IP6 in Daemon.json file

For IPv6 addresses, we rate limit against the first 64 bits in the IPv6 address. If you are still being rate limited even with authenticating and are using IPv6, it is most likely due to a "noisy neighbor" in the same subnet. You can always force IPv4 via Docker Desktop settings or the Docker Engine Daemon file:

Edit dockerd via the Docker Desktop UI:
Open Docker Desktop and navigate to settings > Docker Engine
Add the following line within the JSON structure, ensuring it contains this:
{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "ipv6": false
}
Restart Docker:
Save the changes and restart Docker Desktop to apply the configuration

If your server is running Docker Engine and not Docker Desktop then you will need to edit the daemon.json with the "ipv6": false key and value.

[viceice]

Docker issue HAP429 to.
Got a response from docker to disable my IP6 in Daemon.json file

For IPv6 addresses, we rate limit against the first 64 bits in the IPv6 address. If you are still being rate limited even with authenticating and are using IPv6, it is most likely due to a "noisy neighbor" in the same subnet. You can always force IPv4 via Docker Desktop settings or the Docker Engine Daemon file:

Edit dockerd via the Docker Desktop UI:
Open Docker Desktop and navigate to settings > Docker Engine
Add the following line within the JSON structure, ensuring it contains this:
{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "ipv6": false
}
Restart Docker:
Save the changes and restart Docker Desktop to apply the configuration

If your server is running Docker Engine and not Docker Desktop then you will need to edit the daemon.json with the "ipv6": false key and value.

this doesn't help for kubernetes and is also not an option for a lot people.

they should never block the login in such a hard way. this will block downloads of images which are exempt from rate limits, like bitnami images.

pretty annoying.

[ScorpioCoding]

On ubuntu where do I find the daemon.json file.
in my /etc/docker/ there is no daemon file so do I create one ?
or is it somewhere else?

[SnakeNET64]

I'm experiencing the same issue from the same source (OVH VPS in Frankfurt).
Following your advices, I managed to solve it with this ufw filter : /
ufw deny out from any to 2600:1f18:2148:bc00::/56
Denying access to DockerHub IPs in IPV6, it fail back to ipv4 which work.

[aep-feha]

I'm experiencing the same issue from the same source (OVH VPS in Frankfurt). Following your advices, I managed to solve it with this ufw filter : / ufw deny out from any to 2600:1f18:2148:bc00::/56 Denying access to DockerHub IPs in IPV6, it fail back to ipv4 which work.

It worked perfectly. Thank you!

[Zadigo]

I'm experiencing the same issue from the same source (OVH VPS in Frankfurt). Following your advices, I managed to solve it with this ufw filter : / ufw deny out from any to 2600:1f18:2148:bc00::/56 Denying access to DockerHub IPs in IPV6, it fail back to ipv4 which work.

Thanks for this solution. It is working for me for now because this was getting me mad

[cln-io]

I'm experiencing the same issue from the same source (OVH VPS in Frankfurt). Following your advices, I managed to solve it with this ufw filter : / ufw deny out from any to 2600:1f18:2148:bc00::/56 Denying access to DockerHub IPs in IPV6, it fail back to ipv4 which work.

It worked perfectly. Thank you!

This worked for me (OVH Plesk Debian Buster deploy) thanks!

[thaJeztah]

Documentation on this could probably be improved (at least, I went looking at the documentation for rate-limits, and couldn't directly find information about these cases; https://docs.docker.com/docker-hub/usage/#pull-rate-limit).

I work at docker (but not on the team working on Docker Hub, so don't ask detailed questions 🙈 🙈 ), but I asked on our Slack if our documentation team could have a look at improving the docs (and possibly documenting known situations and workarounds) to help discovery.

[joachimtingvold]

I work at docker (but not on the team working on Docker Hub, so don't ask detailed questions 🙈 🙈 ), but I asked on our Slack if our documentation team could have a look at improving the docs (and possibly documenting known situations and workarounds) to help discovery.

Even if the documentation is made more clear, it does not change these facts;

• Asking to disable IPv6 is not a solution
• Rate-limiting on an entire /64 prefix is questionable
• Rate-limiting logins, effectively making you unable to authenticate in order to bypass rate-limiting is the real problem. Also, authenticated users should not be rate-limited even if the prefix is marked for rate-limit for unauthenticated users. Fix these two things, and the problem goes away.

[thaJeztah]

Rate-limiting logins, effectively making you unable to authenticate in order to bypass rate-limiting is the real problem

I'd have to ask the team, but I think there's separate rate limits in effect; "regular" rate limits on pulls (the one I pointed to in the documentation), and "abuse" limits; the "abuse" limits are much (much) higher, and are in place to protect against DoS attacks and misconfigured systems (e.g. systems with invalid credentials authenticating in a tight loop).

[joachimtingvold]

I'd have to ask the team, but I think there's separate rate limits in effect; "regular" rate limits on pulls (the one I pointed to in the documentation), and "abuse" limits; the "abuse" limits are much (much) higher, and are in place to protect against DoS attacks and misconfigured systems (e.g. systems with invalid credentials authenticating in a tight loop).

Literally every OVH customer with IPv6 enabled can't login/authenticate via CLI due to hitting rate-limits. This is why this GitHub issue was created in the first place (-:

Either this is intended, in which case it's stupid. Or it's not intended. In both cases it should get fixed.

[henk23]

Looks like it's not only OVH.

Have the same problem here, but with Digial Ocean, location is also Frankfurt, IPv6 starting with 2a03:b0c0:3:d0:...

ufw deny out from any to 2600:1f18:2148:bc00::/56

This works like a charm :)

[gaswirth]

I'm experiencing the same issue, from Linode in Newark, NJ, USA. ufw deny out from any to 2600:1f18:2148:bc00::/56 worked for me, too!

[mohammedsaif-fb]

I'm experiencing the same issue, from DigitalOcean in London, UK.

[talex5]

Same here with DigitalOcean. My machine does have its own /64 address. The ufw workaround above got it going.

(I got this when doing docker stack deploy to update some images, and for some reason instead of just doing nothing, it stopped the services that needed updating and then refused to restart them.)

[gaswirth]

I'll add that my experience was similar to @talex5 -- I couldn't update the images, but docker refused to start the ones that were jusr running.

[1player]

This is a delicate balance - rate limiting IPv6 is not straightforward, and we're trying to find the right path. We've seen many network setups where individual machines are granted a whole /64 space, with each individual network request coming from a different IP - this makes the system incredibly fragile to abuse.

This makes no sense — I doubt you're even rate limiting per /64 network. I just created a VPS on OVH, I have an entire /64 at my disposal, I don't have the Privacy Extensions enabled so all connections are made from the same IPv6 address, and yet I am rate limited from the first docker pull I made. How is that even possible, as no one else is sharing that /64 network?

Either the IPv6 rate limiting at Docker Hub is completely borked, or you split it in blocks much larger than /64 (/56? /32?) which is insane and would explain why entire datacentres are receiving 429 errors.

[maps-it]

Here’s my experience:
I noticed that UFW was disabled, so I enabled it:
$ sudo ufw enable

Then, I added a rule to block outgoing connections to the specified IPv6 range:
$ sudo ufw deny out from any to 2600:1f18:2148:bc00::/56

Finally, I reloaded UFW to apply the changes:
$sudo ufw reload

✅ Everything works fine now! 🚀