Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus metrics broken on prom 3.0 #4277

Closed
ishioni opened this issue Dec 8, 2024 · 7 comments · Fixed by #4340
Closed

Prometheus metrics broken on prom 3.0 #4277

ishioni opened this issue Dec 8, 2024 · 7 comments · Fixed by #4340
Assignees
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed

Comments

@ishioni
Copy link

ishioni commented Dec 8, 2024

Describe the bug
Prometheus 3.0 rejects the scrape because dragonfly does not set content-type

To Reproduce
Steps to reproduce the behavior:

  1. curl http://dragonflydb:adminport
  2. note the lack of content-type in the http response header
  3. Prometheus 3.0 complains with "Error scraping target: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target"

Expected behavior
Prometheus metrics should work

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [talos 1.8]
  • Kernel: # 6.6
  • Containerized?: [Kubernetes]
  • Dragonfly Version: [1.25.4]

Additional context
Setting something like
content-type text/plain; version=0.0.4; charset=utf-8; escaping=underscores
should be enough to make it work. Adding a fallback_scrape_protocol should be good as well, but the prometheus operator CRDs don't allow that unfortunately

More context https://prometheus.io/docs/prometheus/3.0/migration/#scrape-protocols

@ishioni ishioni added the bug Something isn't working label Dec 8, 2024
@romange romange assigned adiholden and unassigned adiholden Dec 10, 2024
@romange romange added help wanted Extra attention is needed good first issue Good for newcomers labels Dec 10, 2024
@romange
Copy link
Collaborator

romange commented Dec 10, 2024

@ishioni should be easy to fix. would you like to take a stab on it?

@ishioni
Copy link
Author

ishioni commented Dec 10, 2024

Wish I could - last time I touched cpp was over a decade ago in college

@lnd-d
Copy link

lnd-d commented Dec 11, 2024

I added this to prometheus.yml meanwhile.

global:
 metric_name_validation_scheme: legacy

@VirtualDisk
Copy link

I added this to prometheus.yml meanwhile.

global:
 metric_name_validation_scheme: legacy

How do you enable this in the Prometheus Operator?

@beauwest
Copy link

beauwest commented Dec 18, 2024

A more targeted solution for now is to apply a fallBackScrapeProtocol to the ServiceMonitor or PodMonitor.

Full example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dragonfly-servicemonitor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dragonfly
  endpoints:
    - port: service-port
  fallbackScrapeProtocol: PrometheusText1.0.0

@romange
Copy link
Collaborator

romange commented Dec 18, 2024

How do I reproduce it locally? Running the latest version of prometheus is enough?

@beauwest
Copy link

Yes sir, running Prometheus 3 is enough to reproduce it. The /metrics endpoint does not return a Content-Type header.

From the migration guide

Prometheus v3 is more strict concerning the Content-Type header received when scraping. Prometheus v2 would default to the standard Prometheus text protocol if the target being scraped did not specify a Content-Type header or if the header was unparsable or unrecognised. This could lead to incorrect data being parsed in the scrape. Prometheus v3 will now fail the scrape in such cases.

@romange romange self-assigned this Dec 18, 2024
romange added a commit that referenced this issue Dec 18, 2024
Also, update the local stack to use prometheus 3.0
Finally, hex-escape arguments when logging an error for a command.

Fixes #4277

Signed-off-by: Roman Gershman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants