Follow pagination and implement rate throttling

[jacobthemyth]

I was having a problem with images not getting deleted. It turns out the
maximum page size is now 100, down from 500. Since the application I'm
using this for consistently has more than 100 images that don't meet the
delete criteria and they're in the first page, the action would never
fetch the images that needed to be deleted.

This uses the Hypermedia link header
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api#hypermedia)
to traverse all pages. This worked well, but I hit both primary and
secondary rate limits trying to delete all my old images, so I also
added a function to handle rate limiting.

This still needs tests, documentation, and probably some error handling
and print statements, but I wanted to open a PR to see whether you're
okay with this approach and would consider merging it before I spend on
these details.

[codecov]

Codecov Report

Merging #42 (2ff9ce0) into main (cc57103) will increase coverage by 0.24%.
The diff coverage is 75.00%.

❗ Current head 2ff9ce0 differs from pull request most recent head 108bfb6. Consider uploading reports for the commit 108bfb6 to get more accurate results

@@            Coverage Diff             @@
##             main      #42      +/-   ##
==========================================
+ Coverage   89.84%   90.09%   +0.24%     
==========================================
  Files           1        1              
  Lines         197      222      +25     
  Branches       46       51       +5     
==========================================
+ Hits          177      200      +23     
- Misses         14       16       +2     
  Partials        6        6              

Impacted Files	Coverage Δ
main.py	90.09% <75.00%> (+0.24%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[sondrelg]

Traversing backwards and handling rate limits both sound like great ideas, and the current implementation looks good to me at first glance 👏 Let me know when you've added docs and tests, and I'll review right away 👍

[lukasz-mitka]

Please add pagination for list_packages and list_org_packages. Same issue is present there.
https://github.com/snok/container-retention-policy/pull/42/files#diff-b10564ab7d2c520cdd0243874879fb0a782862c3c902ab535faabe57d5a505e1R95

[jacobthemyth]

@lukasz-mitka sure, can do. It will be a few more days until I have time to finish this, but will include those functions.

[sondrelg]

Any updates here @systemist? 🙂

[den-is]

I've encountered the same issue
One ugly configured repository has a Package with 10k untagged and 2k tagged images under it! 🙀
This update and work will be highly appreciated! 🙇
Unfortunately, I lack python asyncio apps development skills.

[lukasz-mitka]

@den-is if you're in a hurry you can try using this this PR directly

uses: systemist/container-retention-policy@pagination-and-rate-limit

Disclaimer: I didn't test it :)

[sondrelg]

@systemist are you still planning on finishing this?

[sondrelg]

Does anyone else fancy finishing the PR? It's 90% ready if I recall correctly?

Would be nice to merge and bump major versions this week or next if possible.

[sondrelg]

I'll merge this, try to add tests and docs, then upgrade today hopefully 🙂

[sondrelg]

The main branch should now be up to date. I've added some tests, and I've added pagination support to the package fetching.

Does anyone have a repo with a package count high enough to test if the main branch currently works? Would be nice to get at least one confirmation before we bump major version, if anyone here is free to try 🙇

[lukasz-mitka]

@sondrelg Thank you for finishing this!
It's working fine in my setup (>>100 versions)

Found the bug though. Step outputs didn't get set: #56

[sondrelg]

Thanks for testing! I'll release a new version today then hopefully 👍