Prometheus remote write with Thanos out of order metrics stops metrics processing

[nniehoff]

When pushing metrics to a Thanos receive endpoint if the metrics are out of order Thanos will return an HTTP 409 conflict response. Telegraf will then keep retrying the same batch of metrics until action is taken to resolve the issue. Thanos, however, expects the clients to understand 409 is a conflict and should not retry sending the metrics. This will cause new metrics which are then processed by telegraf to fill the buffer and not be delivered to Thanos until the conflict is resolved. See thanos-io/thanos#1509 (comment) where 409 is returned if metrics are out of order and the expected behavior for the remote write client is to not retry.

Relevant telegraf.conf:

[[inputs.tail]]
files = ["/local/sampledata.txt"]
data_format = "influx"
from_beginning = true
precision = "1s"

[[outputs.http]]
  url = "http://thanos-dev-receive.example.com/api/v1/receive"
  data_format = "prometheusremotewrite"

  [outputs.http.headers]
    Content-Type = "application/x-protobuf"
    Content-Encoding = "snappy"
    X-Prometheus-Remote-Write-Version = "0.1.2"

System info:

Telegraf: 1.18.2
OS: Debian based docker container

Docker

Steps to reproduce:

1. Create thanos-receive endpoint and modify the output URL appropriately.
2. Create sampledata.txt:

nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=4 1623440699000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=5 1623440700000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=6 1623440701000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=0 1623440695000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=1 1623440696000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=2 1623440697000000000
nicks_influx_metric,host=fb56ea19c727,nicks_label=0.2.0 value=3 1623440698000000000

3. Run telegraf with the config specified.

Expected behavior:

Ideally, I would love to see all 7 metrics received by Thanos, some subset would also be ok, however, to stop processing metrics and keep retrying prevents any new metrics from reaching Thanos. It would be ok if the conflicting metrics or even batch of metrics were dropped but the pipeline of metrics should not be stopped when a 409 is received.

Actual behavior:

When Thanos responds with an HTTP 409 telegraf will retry to send the batch of metrics. This causes the buffer to fill, preventing any new metrics from being delivered.

Additional info:

[chris-barbour-as]

We've run into the same problem. In our specific case, AWS cloudwatch metrics were being ingested out of order due to the frequency of ingestion. We were able to fix this by collecting more regularly, but we are not convinced that will be a valid solution for all future use cases.

The default FIFO ordering causes cortex to reject samples for the same basic reason Thanos rejects them.

In our case, the metrics buffer eventually fills, causing the offending out of order metrics to be dropped. At that point, delivery of metrics resumes. A certain number of data points will be lost as collateral damage.

Ideally, telegraf should order delivery by metric timestamp instead of FIFO. And it should have the capability of detecting and dropping out of order metrics itself.

Perhaps the ability to detect and drop out of order and duplicate samples could be implemented using a processor? E.g. something similar to dedupe, except that it tracks the latest sample and drops both duplicate and older samples?

As far as I know, remote_write operates on batches of metrics rather than individual metrics. I'm not sure if there is a mechanism to determine what metrics are being delivered out of order and to drop them individually. In our experience, it's pretty common for the entire block of metrics to be dropped.

[Kampe]

We see this same issue with our "edge" compute sites that report into thanos recieve.

[nvinzens]

These out of order metrics will completely break setups where telegraf is remote writing to Thanos with the next release because this got merged thanos-io/thanos#5508.
All metrics with out of order labels will be rejected with a 409 by the thanos-receive component.

To figure out which label is responsible for this we compiled our own thanos with some addditional debug info:

level=debug ts=2022-08-11T14:13:15.200077312Z caller=writer.go:84 component=receive component=receive-writer tenant=firewall msg="Out of order labels in the label set" lset="unsupported value type"
out of order labels detected: current label =  __name__ , previous label =  type

Above shows that the special prometheus label __name__ is sorted at the end of the metric which is wrong according to ASCII since _ < [a-z].

We will try to figure out where this happens but any help is appreciated.

[nniehoff]

FYI There is now an option for the HTTP output which can be used as a workaround:

non_retryable_statuscodes = [409, 413]

[Moep90]

@MyaLongmire does this work for you?

Thanos

image: docker.io/bitnami/thanos:0.28.0-scratch-r0

OS

$ dpkg -l |grep telegraf
ii  telegraf                             1.24.1-1                     amd64        Plugin-driven server agent for reporting metrics into InfluxDB.

Telegraf config

[global_tags]
  hostname         = "myhostname"
  host_ip          = "__ip__"
  host_network     = "__ip__"
  os               = "debian"
  os_major         = "11"
  telegraf_version = "1.24"

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "1s"
  logfile = "/dev/null"
  omit_hostname = false

[[outputs.http]]
  url = "https://thanos-dev-receive.example.com/api/v1/receive"
  non_retryable_statuscodes = [409, 413]
  use_batch_format = false <----- enabled/disabled doesnt matter
  data_format = "prometheusremotewrite"
  [outputs.http.headers]
    Content-Type = "application/x-protobuf"
    Content-Encoding = "snappy"
    X-Prometheus-Remote-Write-Version = "0.1.0"

# -----------------------------------------------
# INPUTS
# -----------------------------------------------
[[inputs.bcache]]
  bcachePath = "/sys/fs/bcache"
[[inputs.bond]]
[[inputs.conntrack]]
  dirs = ["/proc/sys/net/netfilter"]
[[inputs.cpu]]
[[inputs.diskio]]
  devices = ["sd*", "vd*"]
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.ipvs]]
[[inputs.processes]]
[[inputs.mdstat]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.netstat]]
[[inputs.nfsclient]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.ntpq]]
  options = "-p"
[[inputs.kernel_vmstat]]

Telegraf start

2022-09-30T13:37:45Z I! Loaded inputs: bcache bond conntrack cpu disk diskio ipvs kernel_vmstat mdstat mem net netstat nfsclient ntpq processes swap system
2022-09-30T13:37:45Z I! Loaded aggregators: 
2022-09-30T13:37:45Z I! Loaded processors: 
2022-09-30T13:37:45Z I! Loaded outputs: http

Thanos receiver logs

{
  "caller": "writer.go:163",
  "component": "receive-writer",
  "level": "warn",
  "msg": "Error on series with out-of-order labels",
  "numDropped": 825,
  "tenant": "default-tenant",
  "ts": "2022-09-30T13:19:56.282263846Z"
}