Handle connection / timeout requests for manifests

[ColeDCrawford]

A job may ingest assets and manifests and complete, but we should handle the case where a manifest is still unavailable. Not sure if this is a networking issue, but I'm hitting that.

Completed job: https://mps-admin-dev.lib.harvard.edu/admin/ingest/jobstatus/624f10ffbb00ed4e94426f1f/
Failing manifest: https://nrs-dev.lib.harvard.edu/URN-3:AT:GENVUnexC9GeHx3wRM3LNskKu:MANIFEST:3

mapping-color-in-history-web-1         | Exception in thread Thread-5:
mapping-color-in-history-web-1         | Traceback (most recent call last):
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
mapping-color-in-history-web-1         |     conn = connection.create_connection(
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
mapping-color-in-history-web-1         |     raise err
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
mapping-color-in-history-web-1         |     sock.connect(sa)
mapping-color-in-history-web-1         | ConnectionRefusedError: [Errno 111] Connection refused
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | During handling of the above exception, another exception occurred:
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | Traceback (most recent call last):
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
mapping-color-in-history-web-1         |     httplib_response = self._make_request(
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
mapping-color-in-history-web-1         |     self._validate_conn(conn)
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
mapping-color-in-history-web-1         |     conn.connect()
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
mapping-color-in-history-web-1         |     conn = self._new_conn()
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
mapping-color-in-history-web-1         |     raise NewConnectionError(
mapping-color-in-history-web-1         | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0xffffa05f5760>: Failed to establish a new connection: [Errno 111] Connection refused
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | During handling of the above exception, another exception occurred:
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | Traceback (most recent call last):
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
mapping-color-in-history-web-1         |     resp = conn.urlopen(
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
mapping-color-in-history-web-1         |     retries = retries.increment(
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
mapping-color-in-history-web-1         |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
mapping-color-in-history-web-1         | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nrs-dev.lib.harvard.edu', port=443): Max retries exceeded with url: /URN-3:AT:GENVUnexC9GeHx3wRM3LNskKu:MANIFEST:3 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffffa05f5760>: Failed to establish a new connection: [Errno 111] Connection refused'))
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | During handling of the above exception, another exception occurred:
mapping-color-in-history-web-1         | 
mapping-color-in-history-web-1         | Traceback (most recent call last):
mapping-color-in-history-web-1         |   File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
mapping-color-in-history-web-1         |     self.run()
mapping-color-in-history-web-1         |   File "/usr/local/lib/python3.9/threading.py", line 910, in run
mapping-color-in-history-web-1         |     self._target(*self._args, **self._kwargs)
mapping-color-in-history-web-1         |   File "/opt/mcih/src/mcih/scribe/ingest_utility.py", line 126, in ingest_manifest
mapping-color-in-history-web-1         |     work.save()
mapping-color-in-history-web-1         |   File "/opt/mcih/src/mcih/folio/models.py", line 129, in save
mapping-color-in-history-web-1         |     self.set_image_urls()
mapping-color-in-history-web-1         |   File "/opt/mcih/src/mcih/folio/models.py", line 115, in set_image_urls
mapping-color-in-history-web-1         |     data = requests.get(self.iiif_manifest).json()
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/api.py", line 75, in get
mapping-color-in-history-web-1         |     return request('get', url, params=params, **kwargs)
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/api.py", line 61, in request
mapping-color-in-history-web-1         |     return session.request(method=method, url=url, **kwargs)
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
mapping-color-in-history-web-1         |     resp = self.send(prep, **send_kwargs)
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
mapping-color-in-history-web-1         |     r = adapter.send(request, **kwargs)
mapping-color-in-history-web-1         |   File "/opt/mcih/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
mapping-color-in-history-web-1         |     raise ConnectionError(e, request=request)
mapping-color-in-history-web-1         | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='nrs-dev.lib.harvard.edu', port=443): Max retries exceeded with url: /URN-3:AT:GENVUnexC9GeHx3wRM3LNskKu:MANIFEST:3 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffffa05f5760>: Failed to establish a new connection: [Errno 111] Connection refused'))

[ColeDCrawford]

Confirmed this is a network issue.