GHA jobs fail because of too many requests

[jakirkham]

Have started seeing GHA jobs fail because of too many request (for example:

#26 ERROR: toomanyrequests: Too Many Requests (HAP429).

------
 > pushing rapidsai/ci-conda:cuda11.8.0-rockylinux8-py3.10-amd64 with docker:
------

This is the same as issue ( rapidsai/miniforge-cuda#72 ), which already contains suggestions on how this could be resolved

[jameslamb]

I just observed the same thing on https://github.com/rapidsai/ci-imgs/actions/runs/11614334107/job/32342353447

43 of 270 jobs failed because they hit this rate limit. All succeeded on a rebuild. Just posting since this is an old issue, to note that this is still a problem we hit sometimes.

[jameslamb]

I just merged #216, which limits this repo to running 150 jobs at a time within a workflow run. Let's see if that helps.

[vyasr]

@jameslamb what data do we want to see before we close this issue?

[jameslamb]

🤷🏻 I don't know, there are so many factors that affect whether or not we hit this.

Let's say when the next round of renovate PRs go up on January 1st, if we don't see any rate-limiting errors, we close this.

[jameslamb]

In #219 we reduced this further to 50 parallel jobs.

Saw multiple runs on different days where most jobs succeeded and 0 had rate-limiting issues. ref: #219 (comment)

If the job triggered by the merge of #219 also succeeds without any rate-limiting issues, I think we should close this. Build to watch: https://github.com/rapidsai/ci-imgs/actions/runs/12816222911

[jameslamb]

If the job triggered by the merge of #219 also succeeds without any rate-limiting issues, I think we should close this. Build to watch: https://github.com/rapidsai/ci-imgs/actions/runs/12816222911

Pretty similar to what we saw on #219... 265 out of the 270 build jobs succeeded, and that took about 27 minutes.

Importantly, none of the failures were related to rate-limiting

• conda/mamba errors ("Unexpected error 9 on netlink descriptor 9.")
  • https://github.com/rapidsai/ci-imgs/actions/runs/12816222911/job/35736686093#step:8:2512
  • https://github.com/rapidsai/ci-imgs/actions/runs/12816222911/job/35736689657#step:8:2502
  • https://github.com/rapidsai/ci-imgs/actions/runs/12816222911/job/35736888287#step:8:2610
  • https://github.com/rapidsai/ci-imgs/actions/runs/12816222911/job/35737023526#step:8:2860
• "libmamba File not valid: SHA256 doesn't match expectation"
  • https://github.com/rapidsai/ci-imgs/actions/runs/12816222911/job/35736894148#step:8:2683

Just restarted the failed jobs there. I'd like to see all the pushes to DockerHub also succeed without rate-limiting issues before we close this.

[jameslamb]

All the pushes succeeded, with 0 need for restarts!

https://github.com/rapidsai/ci-imgs/actions/runs/12816222911

I think this can be closed. Thanks @bdice for keeping it moving forward with #219