Fix incorrect OpenMetrics V2 check exposition format HTTP header

[jalaziz]

What does this PR do?

With this change, the openmetrics check should request the appropriate
exposition format fixing "Clashing name" errors.

Motivation

The exposition format to request is set via the "Accept" header. Despite
the OpenMetricsScraper appearing to set the correct header when
use_latest_spec is enabled, it is actually a no-op due to the default
HTTP headers including Accept: */* and the use of setdefault.

Additional Notes

Tests incoming. It appears that there were no existing tests of use_latest_spec or the new exposition format.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #11899 (4eebc0d) into master (aaae622) will not change coverage.
The diff coverage is 75.00%.

Flag	Coverage Δ
arangodb	95.74% <ø> (ø)
avi_vantage	91.92% <ø> (ø)
calico	83.33% <ø> (ø)
citrix_hypervisor	87.50% <ø> (ø)
crio	89.79% <ø> (ø)
ibm_i	81.58% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[ofek]

Thanks! What do you think about always just overriding the header?

[jalaziz]

Thanks! What do you think about always just overriding the header?

That's how I approached it at first, but then realized that it may break workflows where a user sets headers in check config. In fact, that's how I'm working around this issue for now:

      {
        "openmetrics_endpoint": "http://%%host%%:%%port_prometheus%%/metrics",
        "metrics": [".*"],
        "namespace": "foo",
        "use_latest_spec": true,
        "headers": {
          "Accept": "application/openmetrics-text; version=0.0.1; charset=utf-8"
        }
      }

Another option would be to override the header by setting it on config if headers['Accept'] is not set.

[jalaziz]

@ofek Added some really basic tests. Please let me know if anything else is needed.

As for always overriding, happy to switch to that if you think there's never a need to override it via instance configs.

[jalaziz]

@ofek just following up since you requested the changes. Happy to implement "always" override path, but want to confirm that it's okay to break the override path that exists today.

[ofek]

Thanks, great job as always!!!