Fix up and return next/prev/first/last Link headers

[ehyche]

Problem Description

In this project, the Swift Package Registry Service List Package Releases endpoint proxies the Github API List Repository Tags endpoint.

The Github API List Repository Tags endpoint is a paginated endpoint which uses the per_page and page URL query arguments to control pagination. The default value for per_page is 30 and the default for page is 1. So by default, you are getting the first 30 tags for a respository.

When the current response for this endpoint does not represent the entire set of tags for the respository, this endpoint will add Link headers to the response. For example, if the request is for the swift-syntax repository:

curl -H "X-GitHub-Api-Version: 2022-11-28" \
        -H "Accept: application/vnd.github+json" \
        -H "Authorization: Bearer XXXXX" \
        --verbose "https://api.github.com/repos/swiftlang/swift-syntax/tags"

Then the response is:

HTTP/2 200 
Date: Wed, 15 Jan 2025 14:43:37 GMT
...
Link: <https://api.github.com/repositories/143079594/tags?per_page=30&page=2>; rel="next", <https://api.github.com/repositories/143079594/tags?per_page=30&page=47>; rel="last"
...
[
  {
    "name": "xcode11-beta1",
    "zipball_url": "https://api.github.com/repos/swiftlang/swift-syntax/zipball/refs/tags/xcode11-beta1",
    "tarball_url": "https://api.github.com/repos/swiftlang/swift-syntax/tarball/refs/tags/xcode11-beta1",
    "commit": {
      "sha": "fdcbd4bfea2e833467e16d7962c4110c14382b56",
      "url": "https://api.github.com/repos/swiftlang/swift-syntax/commits/fdcbd4bfea2e833467e16d7962c4110c14382b56"
    },
    "node_id": "MDM6UmVmMTQzMDc5NTk0OnJlZnMvdGFncy94Y29kZTExLWJldGEx"
  },
  {
    "name": "swift-DEVELOPMENT-SNAPSHOT-2025-01-10-a",
    "zipball_url": "https://api.github.com/repos/swiftlang/swift-syntax/zipball/refs/tags/swift-DEVELOPMENT-SNAPSHOT-2025-01-10-a",
    "tarball_url": "https://api.github.com/repos/swiftlang/swift-syntax/tarball/refs/tags/swift-DEVELOPMENT-SNAPSHOT-2025-01-10-a",
    "commit": {
      "sha": "37fa0c884ac58aded420668786bff4b6d005db27",
      "url": "https://api.github.com/repos/swiftlang/swift-syntax/commits/37fa0c884ac58aded420668786bff4b6d005db27"
    },
    "node_id": "MDM6UmVmMTQzMDc5NTk0OnJlZnMvdGFncy9zd2lmdC1ERVZFTE9QTUVOVC1TTkFQU0hPVC0yMDI1LTAxLTEwLWE="
  },
  ...

So the Link headers are saying that the URL for the next page of tags is https://api.github.com/repositories/143079594/tags?per_page=30&page=2 and the URL for the last page of tags is https://api.github.com/repositories/143079594/tags?per_page=30&page=47.

The problem is that the current implementation of this project does not provide appropriate next/prev/first/last Link headers in the response. So currently, it is essentially only providing the first page of tags in the response.

Fix Description

1. First, we need look in the response of the Github API endpoint to see if there are Link headers. So we use the parse-link-header module to parse the Link header in the response headers.
2. If we find next/prev/first/last Link URLs in the response, then we pass those on to the client, keeping the per_page and page query parameters, but replacing the host and path of the URL appropriately.
3. Then, since those URLs we return are going to be queried back to our service, then we need to support the per_page and page query parameters on the listPackages endpoint. So we parse those out of the URL and pass them along to the fetchTag function.

Testing

Now, with this fix if the request is:

$ curl --verbose "http://127.0.0.1:3001/swiftlang/swift-syntax"

Then the response is:

*   Trying 127.0.0.1:3001...
* Connected to 127.0.0.1 (127.0.0.1) port 3001
> GET /swiftlang/swift-syntax HTTP/1.1
> Host: 127.0.0.1:3001
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
< Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS, HEAD
< Content-Version: 1
< Content-Type: application/json; charset=utf-8
< Link: <https://127.0.0.1:3001/swiftlang/swift-syntax/xcode11-beta1>; rel="latest-version",<https://github.com/swiftlang/swift-syntax>; rel="canonical",<ssh://[email protected]:swiftlang/swift-syntax.git>; rel="alternate",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=2>; rel="next",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=14>; rel="last"
< Content-Length: 13762
< Date: Wed, 15 Jan 2025 14:56:45 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
{
  "releases": {
    "xcode11-beta1": {
      "url": "https://127.0.0.1:3001/swiftlang/swift-syntax/xcode11-beta1"
    },
    "swift-DEVELOPMENT-SNAPSHOT-2025-01-10-a": {
       "url": "https://127.0.0.1:3001/swiftlang/swift-syntax/swift-DEVELOPMENT-SNAPSHOT-2025-01-10-a"
    },
    ...

As you can see the Link header now provides these additional sections:

Link: ...,
    <http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=2>; rel="next",
    <http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=14>; rel="last"

So if we then fetch the next URL:

curl --verbose "http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=2" 

Then the response is:

HTTP/1.1 200 OK
Link: <https://127.0.0.1:3001/swiftlang/swift-syntax/swift-DEVELOPMENT-SNAPSHOT-2024-04-02-a>; rel="latest-version",<https://github.com/swiftlang/swift-syntax>; rel="canonical",<ssh://[email protected]:swiftlang/swift-syntax.git>; rel="alternate",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=1>; rel="prev",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=3>; rel="next",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=14>; rel="last",<http://127.0.0.1:3001/swiftlang/swift-syntax?per_page=100&page=1>; rel="first"
...

{
  "releases": {
    "swift-DEVELOPMENT-SNAPSHOT-2024-04-02-a": {
      "url": "https://127.0.0.1:3001/swiftlang/swift-syntax/swift-DEVELOPMENT-SNAPSHOT-2024-04-02-a"
    },
    "swift-DEVELOPMENT-SNAPSHOT-2024-04-01-a": {
      "url": "https://127.0.0.1:3001/swiftlang/swift-syntax/swift-DEVELOPMENT-SNAPSHOT-2024-04-01-a"
    },
    ...