querier: fix remote read error translation

[dimitarvdimitrov]

Problem

Currently remote-read has conflicting behaviours:

• when returning samples all internal errors are translated to HTTP 400
• when returning chunks all internal errors are translated to HTTP 500

PR description

This PR unifies error handling such that:

• all validation errors will return HTTP 400
• all promQL storage errors and all other errors will return HTTP 500

It also adds tests for this in two places

• integration tests: this comes with moving of some code to query remote read from an endpoint
• unit tests in pkg/querier

The PR also moves the content type string to pkg/querier/api since it's a complicated string that was repeated in a few places.

Note to reviewers

I couldn't really understand the Prometheus docs in terms of the expected return codes and content type. The docs give a JSON example as the structure for the API, but the remote read API is based on protocol buffers. At the same time the protocol buffers do not have fields for errors. So I'm still not sure if we're compliant with the docs page.

Because of the same there are some discrepancies with HTTP status codes on the regular query and query_range APIs - they return HTTP 422 for hit limits whereas remote read returns HTTP 400

[dimitarvdimitrov]

this PR started just as added integration tests, but I ended up finding the bug and fixing it. The description and title are up to date now

[pracucci]

I couldn't really understand the Prometheus docs in terms of the expected return codes and content type. The docs give a JSON example as the structure for the API, but the remote read API is based on protocol buffers. At the same time the protocol buffers do not have fields for errors. So I'm still not sure if we're compliant with the docs page.

Looking at the remote read handler implementation in Prometheus looks like they're just sending errors as plain text messages (see code).

[pracucci]

Great job spotting the bug and fixing it! 👏

[dimitarvdimitrov]

thanks for the review @pracucci!

[dimitarvdimitrov]

i'll leave this open in case anyone else wants to give it a review. Hope to merge it before Friday