TimeGate is not reporting the HTTP status correctly

[anjackson]

This is related to #44, #45 but comments there indicate that the TimeGate used to report the status correctly. This seems not to be the case now:

# curl -I https://beta.webarchive.org.uk/wayback/archive/http://www.sahahdsf
HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 02 Mar 2020 16:10:51 GMT
Content-Type: text/html
Content-Length: 3065
Connection: keep-alive
Link: <http://www.sahahdsf>; rel="original", <https://beta.webarchive.org.uk/wayback/archive/http://www.sahahdsf>; rel="timegate", <https://beta.webarchive.org.uk/wayback/archive/timemap/link/http://www.sahahdsf>; rel="timemap"; type="application/link-format"
Vary: accept-datetime
Accept-Ranges: bytes

This causes problems with the Memento Aggregator. I think even if we skip it everywhere else, the TimeGate URI should do a lookup and return the right status code.

[ikreymer]

Currently, the timegate should work at the mp_/ url, eg:

curl -I https://beta.webarchive.org.uk/wayback/archive/mp_/https://webarchive.org.uk/
...
Memento-Datetime: Tue, 12 May 2020 15:00:10 GMT
Link: <https://www.webarchive.org.uk/>; rel="original", <https://beta.webarchive.org.uk/wayback/archive/https://www.webarchive.org.uk/>; rel="timegate", <https://beta.webarchive.org.uk/wayback/archive/timemap/link/https://www.webarchive.org.uk/>; rel="timemap"; type="application/link-format", <https://beta.webarchive.org.uk/wayback/archive/20200512150010mp_/https://www.webarchive.org.uk/>; rel="memento"; datetime="Tue, 12 May 2020 15:00:10 GMT"; collection="archive"
Preference-Applied: rewritten
Vary: accept-datetime, Prefer

However, there is definitely a bug in that it returns the url without mp_/ in ref="timegate".
Would it be better to

1. Fix it so that it returns:

<https://beta.webarchive.org.uk/wayback/archive/mp_/https://www.webarchive.org.uk/>; rel="timegate"

or

2. Make it so the Timegate also works w/o the mp_/ modifier. I suppose this may be clearer, but less consistent as this URL is used for the outer frame, and not for a memento. I suppose in TimeGate query the response is discarded anyway as memento only looks at the headers, so it could still return timegate information with this url, even though the response is the top-frame content.

[anjackson]

The problem with using the mp_/ modifier is that is represents a breaking change of our TimeGate API for all clients (a basic assumption of our work moving from OWB was that switching to pywb should be API compatible). We use the API in various places, as well as it being used by third-parties, and this breaking change has caused a range of problems with both internal and external systems.

This is why I would prefer it if the TimeGate URL https://www.webarchive.org.uk/wayback/archive/<QUERY_URL> behaved in the same way as the mp_/ version.

[ikreymer]

The expected timegate behavior should now work w/o the mp_ modifier. This is also part of the just released pywb 2.4.0