store: out-of-order label set

[miekg]

We've ended up with data that can't be verified nor fixed in our data store. I'm currently in the process of delving deeper in this problem with a more up to date Thanos (v0.5). The problem happened while we where running 0.4.

The symptom is:

01D9S32REZD25B65AN54FVX44F: out-of-order label set
{
  <omitted json - available on request>
} for series 813726

We think it may have to do with receiving a metrics that had an_identical_ label with a value only differing in case.

We still have a copy of the data, and, as said, I'm in the process of narrowing down what actually happened.

This looks like #711, which is fixed, so I wonder why we've hit it.

[GiedriusS]

Are you running Prometheus which has this fix prometheus/prometheus#5372 included?

[miekg]

No, this was def. on an older version. From the looks of it, this is included in 2.9.x ?

With that fix in Prometheus, does there still need to be something in Thanos? I.e. I want to re-inject the metrics we have into our metrics store. Is that possible?

[bwplotka]

Well new Thanos gives ability to ignore this error, so it just warns. It is slight inefficiency, but that's it. Are you blocked with this?

Also it looks like compactor warns / errors out if debug.accept-malformed-index is not set to true, not store right?

[miekg]

Thanks. No not blocked. Good to know we can ignore the error, which gives us more defence in depth

Think this can be closed with WAI

[GiedriusS]

@miekg closing as per your request but what's the meaning of WAl in this context? 😄