ansible-galaxy doesn't handle rate limiting correctly

[newswangerd]

Summary

ansible-galaxy collection install makes a lot of requests to galaxy.ansible.com and can occasionally hit the rate limit. When that happens, galaxy.ansible.com will return a 520 http code and ansible-galaxy fails to download the collection and exits. Ideally when the client encounters a rate limiting http code (either 520, or 429), it should wait, slow down the request rate and try again rather than exiting.

More information is available here: ansible/galaxy#2429

Issue Type

Bug Report

Component Name

ansible-galaxy

Ansible Version

$ ansible --version
root@ubuntu-s-1vcpu-1gb-nyc3-01:~# ansible --version
ansible 2.10.7
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]

Configuration

$ ansible-config dump --only-changed
root@ubuntu-s-1vcpu-1gb-nyc3-01:~# ansible-config dump --only-changed
root@ubuntu-s-1vcpu-1gb-nyc3-01:~#

OS / Environment

All

Steps to Reproduce

Run ansible-galaxy collection install amazon.aws`. If your internet is fast enough, this will occasionally fail when galaxy.ansible.com returns a 520 error code.

Expected Results

Collection should be installed.

Actual Results

`ansible-galaxy` encounters a 429 or 520 http code and exits.

Code of Conduct

I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• lib/ansible/galaxy

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[awcrosby]

Note: as brought up earlier, a mitigation could be to use the page_size parameter on the community galaxy v2/collections/<ns>/<name>/versions/ endpoint to reduce requests

[sivel]

What recommendation would the galaxy team have for a balanced page_size?

[awcrosby]

I feel like page_size=100 could be a good balance, reducing requests, and there are only 2 keys for each result on the v2 versions/ endpoint, not sure if @newswangerd or @cutwater wants to weigh in

[sivel]

I believe for the page_size=100 adjustment, this is all that is required:

Click for diff...

diff --git a/lib/ansible/galaxy/api.py b/lib/ansible/galaxy/api.py
index 352082ded7..93b6df4114 100644
--- a/lib/ansible/galaxy/api.py
+++ b/lib/ansible/galaxy/api.py
@@ -21,7 +21,7 @@ from ansible.errors import AnsibleError
 from ansible.galaxy.user_agent import user_agent
 from ansible.module_utils.six import string_types
 from ansible.module_utils.six.moves.urllib.error import HTTPError
-from ansible.module_utils.six.moves.urllib.parse import quote as urlquote, urlencode, urlparse
+from ansible.module_utils.six.moves.urllib.parse import quote as urlquote, urlencode, urlparse, parse_qs
 from ansible.module_utils._text import to_bytes, to_native, to_text
 from ansible.module_utils.urls import open_url, prepare_multipart
 from ansible.utils.display import Display
@@ -312,6 +312,7 @@ class GalaxyAPI:
     def _call_galaxy(self, url, args=None, headers=None, method=None, auth_required=False, error_context_msg=None,
                      cache=False):
         url_info = urlparse(url)
+        query = parse_qs(url_info.query)
         cache_id = get_cache_id(url)
         if cache and self._cache:
             server_cache = self._cache.setdefault(cache_id, {})
@@ -342,7 +343,7 @@ class GalaxyAPI:
 
                 return res
 
-            elif not url_info.query:
+            elif 'page' not in query:
                 # The cache entry had expired or does not exist, start a new blank entry to be filled later.
                 expires = datetime.datetime.utcnow()
                 expires += datetime.timedelta(days=1)
@@ -781,7 +782,7 @@ class GalaxyAPI:
             api_path = self.available_api_versions['v2']
             pagination_path = ['next']
 
-        versions_url = _urljoin(self.api_server, api_path, 'collections', namespace, name, 'versions', '/')
+        versions_url = _urljoin(self.api_server, api_path, 'collections', namespace, name, 'versions', '/?page_size=100')
         versions_url_info = urlparse(versions_url)
 
         # We should only rely on the cache if the collection has not changed. This may slow things down but it ensures

With ansible.aws this makes 2 requests instead of 15 to /api/v2/collections/amazon/aws/versions/

Still doesn't directly address retries, but it does speedup install of ansible.aws for me from about 20-30s to 5-10s (with an empty cache).

[bcoca]

we have throttling code already in module_utils, just need to start usinig that