Subscription manager breaks our spawners

[fedora-copr-github-bot]

Original issue: https://pagure.io/copr/copr/issue/2132
Opened: 2022-03-22 16:08:23
Opened by: praiskup

It looks this happens in all the clouds now, but not 100% (occasionally some machine starts, and is successful with the subscription):

2022-03-22 16:08:00,532 [ERROR] subscription-manager:2142:MainThread @cli.py:61 - Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/subscription_manager/cli_command/register.py", line 222, in _do_command
    owner_key = service.determine_owner_key(
  File "/usr/lib64/python3.10/site-packages/rhsmlib/services/register.py", line 180, in determine_owner_key
    owners = self.cp.getOwnerList(username)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 1383, in getOwnerList
    owners = self.conn.request_get(method)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 920, in request_get
    return self._request("GET", method, headers=headers, cert_key_pairs=cert_key_pairs)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 945, in _request
    result = super(Restlib, self)._request(request_type, method,
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 805, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 883, in validateResponse
    raise RemoteServerException(response['status'],
rhsm.connection.RemoteServerException: Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
2022-03-22 16:08:06,492 [ERROR] subscription-manager:2144:MainThread @connection.py:833 - Response: 502
2022-03-22 16:08:06,492 [ERROR] subscription-manager:2144:MainThread @connection.py:834 - JSON parsing error: Expecting value: line 1 column 1 (char 0)
2022-03-22 16:08:06,492 [ERROR] subscription-manager:2144:MainThread @cli.py:60 - Error during registration: Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
2022-03-22 16:08:06,492 [ERROR] subscription-manager:2144:MainThread @cli.py:61 - Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/subscription_manager/cli_command/register.py", line 222, in _do_command
    owner_key = service.determine_owner_key(
  File "/usr/lib64/python3.10/site-packages/rhsmlib/services/register.py", line 180, in determine_owner_key
    owners = self.cp.getOwnerList(username)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 1383, in getOwnerList
    owners = self.conn.request_get(method)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 920, in request_get
    return self._request("GET", method, headers=headers, cert_key_pairs=cert_key_pairs)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 945, in _request
    result = super(Restlib, self)._request(request_type, method,
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 805, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 883, in validateResponse
    raise RemoteServerException(response['status'],
rhsm.connection.RemoteServerException: Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
2022-03-22 16:08:12,462 [ERROR] subscription-manager:2146:MainThread @connection.py:833 - Response: 502
2022-03-22 16:08:12,463 [ERROR] subscription-manager:2146:MainThread @connection.py:834 - JSON parsing error: Expecting value: line 1 column 1 (char 0)
2022-03-22 16:08:12,463 [ERROR] subscription-manager:2146:MainThread @cli.py:60 - Error during registration: Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
2022-03-22 16:08:12,463 [ERROR] subscription-manager:2146:MainThread @cli.py:61 - Server error attempting a GET to /subscription/users/copr-team/owners returned status 502
Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/subscription_manager/cli_command/register.py", line 222, in _do_command
    owner_key = service.determine_owner_key(
  File "/usr/lib64/python3.10/site-packages/rhsmlib/services/register.py", line 180, in determine_owner_key
    owners = self.cp.getOwnerList(username)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 1383, in getOwnerList
    owners = self.conn.request_get(method)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 920, in request_get
    return self._request("GET", method, headers=headers, cert_key_pairs=cert_key_pairs)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 945, in _request
    result = super(Restlib, self)._request(request_type, method,
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 805, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python3.10/site-packages/rhsm/connection.py", line 883, in validateResponse
    raise RemoteServerException(response['status'],
rhsm.connection.RemoteServerException: Server error attempting a GET to /subscription/users/copr-team/owners returned status 502

---

praiskup commented at 2022-03-22 17:16:05:

I filled a Red Hat support case 03178902 for this.

---

praiskup commented at 2022-03-22 17:46:26:

This happened for the second time, and it blocked all users for a long time while only a subset of users depends on it. We should rework it:

1. the subscription management task should be asynchronous from the spawning playbook, it takes an unreasonable amount of time anyway. @msuchy did the discussion with S-M folks went somewhere?
2. only EL8 builds should be "blocked" ... and they could wait till the async process finishes?
3. perhaps fail the whole build if it 2. takes too much time? Or try 3 times, on at most three different builders?

[praiskup]

Triage: We should have "on demand" subscription (if the current task is for RHEL).