Fix reporting of HTTP error messages with binary body content

[pstibrany]

What this PR does

PR #8227 introduced sending of binary HTTP response bodies to OTLP requests. Unfortunately our httpgrpc.Server implementation tries to use such response bodies as error message in status.Status objects, and marshalling such object then fails.

This PR fixes that by adding support for specifying different error message to use in status.Status object (PR in dskit TBD), and fixing OTLP handler to use this support.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[pstibrany]

This interface was extracted to simplify test setup, and is otherwise unrelated to the PR.

[aknuds1]

Great work!! See my comments though; I think there might be a left behind comment, and a missing wg.Add(1).

[pstibrany]

Great work!! See my comments though; I think there might be a left behind comment, and a missing wg.Add(1).

Thanks for review Arve. I've fixed your latest findings.

[grafanabot]

The backport to r293 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-8363-to-r293 origin/r293
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x b9055992dbadb486e0e682bf02a4eebdcdf8774f
# Push it to GitHub
git push --set-upstream origin backport-8363-to-r293
git switch main
# Remove the local backport branch
git branch -D backport-8363-to-r293

Then, create a pull request where the base branch is r293 and the compare/head branch is backport-8363-to-r293.

[grafanabot]

The backport to r294 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-8363-to-r294 origin/r294
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x b9055992dbadb486e0e682bf02a4eebdcdf8774f
# Push it to GitHub
git push --set-upstream origin backport-8363-to-r294
git switch main
# Remove the local backport branch
git branch -D backport-8363-to-r294

Then, create a pull request where the base branch is r294 and the compare/head branch is backport-8363-to-r294.