Serve provenance without requiring the Accept header

[simonw]

What's the problem this feature will solve?

When I visit this page in my browser:

https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance

I get this:

{"message":"Request not acceptable"}

To see the content of that page I have to send an accept header like this:

curl -s \
  -H 'accept: application/vnd.pypi.integrity.v1+json' https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance \
  | jq

Describe the solution you'd like

I'd prefer it if the page served me JSON without me having to send that accept header. That way I could explore and understand the API without needing to fire up a terminal or a custom HTTP client.

Additional context

Here's the implementation:

warehouse/warehouse/api/integrity.py

Lines 62 to 67 in 06a2b58

	def provenance_for_file(file: File, request: Request):
	# Determine our response content-type. For the time being, only the JSON
	# type is accepted.
	request.response.content_type = _select_content_type(request)
	if request.response.content_type != MIME_PYPI_INTEGRITY_V1_JSON:
	return HTTPNotAcceptable(json={"message": "Request not acceptable"})

[simonw]

This is covered by the documentation here (I didn't think to look so I dug around in the source code instead): https://docs.pypi.org/api/integrity/#get-provenance-for-file

Although the docs say:

Example JSON request (default if no Accept header is passed)

Which I think is incorrect documentation - you have to pass the Accept header to see the format shown in that example.

[di]

Your browser is almost certainly sending Accept: text/html or something similar, which is not a bare Accept header. A bare Accept header correctly returns a JSON response:

$ curl -s 
    -H 'accept:' https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance
    | jq

{
  "attestation_bundles": [
    {
       ...

[simonw]

Running curl without sending -H 'accept:' works too:

curl -s https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance

Not great from a developer experience / usability POV though, since I do a lot of my API research these days on a phone with Mobile Safari!

[di]

Running curl without sending -H 'accept:' works too:

That's because by default curl accepts any content type (*/*).

Not great from a developer experience / usability POV though, since I do a lot of my API research these days on a phone with Mobile Safari!

The issue is that we want to respect the Accept header we are being sent: if your browser is indicating that it does not accept application/json for the request, then we don't want to send it application/json.

[SpecLad]

The Accept header handling seems to be broken.

Here I explicitly list the integrity JSON media type as acceptable, yet the server refuses to send it:

$ curl -H 'Accept: text/html, application/vnd.pypi.integrity.v1+json; q=0.9' https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance
{"message":"Request not acceptable"}

And here I don't list it, but the server does send it:

$ curl -H 'Accept: image/png' https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance
{"attestation_bundles":[{"attestations":[{"envelope":{"signature":"MEYCIQCwvxvlD0VNoopId6m0noBJHuJ/Ig6LPV/kckAb3c
...

Even when I expressly prohibit it, the server sends it anyway:

$ curl -H 'Accept: image/png, application/vnd.pypi.integrity.v1+json; q=0' https://pypi.org/integrity/pydantic/1.10.19/pydantic-1.10.19-cp310-cp310-macosx_11_0_arm64.whl/provenance
{"attestation_bundles":[{"attestations":[{"envelope":{"signature":"MEYCIQCwvxvlD0VNoopId6m0noBJHuJ/Ig6LPV/kckAb3c
...

[miketheman]

Not trying to solve this yet, but providing a little more context on why this behaves the current way.

The code here will take in the Accept header if provided, if none match, like the case of image/png, return the "default" content type of application/vnd.pypi.integrity.v1+json.

Then in the view, if any of the options passed isn't the specific header, it's rejected.

So if no Accept header is provided, or an Accept header with something that isn't correct

This logic also denies the ability to pass the q=0 option, as again, the logic in _select_content_type() will return an empty list, so the "default" fallback will be invoked.

This seems slightly incorrect from the original intent to respond with a 404 if missing.

[woodruffw]

I'll take a stab at improving the accept handling today 🙂

Edit: FWICT, this might also be an issue with the Accept: handling for the simple HTML/JSON endpoints: those endpoints use the exact same pattern.

[woodruffw]

Oh, I see what's happening here in the 0.9 case:

With Accept: text/html, application/vnd.pypi.integrity.v1+json; q=0.9 we end up selecting text/html, which is currently listed as a valid offer (even though we reject it later), since it has the highest q-factor.

To fix this, I think we could probably tighten the accept handing: instead of doing the JSON fallback we could reject when no offer is accepted, and we could remove the HTML mimetypes from the offer list until we're actually ready to support them.

In other words, the behavior would become:

1. Unchanged: No Accept: header: assume they accept our default JSON form
2. Unchanged: Accept: application/vnd.pypi.integrity.v1+json: send our default JSON form
3. Everything else (including q=0): reject as unacceptable

Thoughts @miketheman @di?

[SpecLad]

Looks like this is now fixed - all of the test cases from the comments behave as expected.