Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compaction TODO metrics have too high a cardinality #5841

Closed
SuperQ opened this issue Oct 31, 2022 · 8 comments · Fixed by #6049 or #7283
Closed

Compaction TODO metrics have too high a cardinality #5841

SuperQ opened this issue Oct 31, 2022 · 8 comments · Fixed by #6049 or #7283
Labels
component: compact good first issue help wanted

Comments

@SuperQ
Copy link
Contributor

SuperQ commented Oct 31, 2022

Thanos, Prometheus and Golang version used:

Thanos v0.28.1

Object Storage Provider:

N/A

What happened:

The metrics thanos_compact_todo_compactions, thanos_compact_todo_downsample_blocks, etc include a group label that can make the cardinality of the metric reach thousands in large deployments. This label doesn't provide a lot of value.

What you expected to happen:

Remove the group label from compaction TODO metirics.

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:
@yeya24
Copy link
Contributor

yeya24 commented Oct 31, 2022

Almost all the compaction metrics have the label group, right? Why it is an issue for the TODO metrics?

@SuperQ
Copy link
Contributor Author

SuperQ commented Oct 31, 2022

After digging through the metrics we have, I discovered it's actually a problem for all of the metrics that have group on them.

I have a WIP change to remove group label from all of the compactor metrics if people find this acceptable.

@Akshit42-hue
Copy link
Contributor

Working on it :)

@yeya24
Copy link
Contributor

yeya24 commented Oct 31, 2022

Hey @Akshit42-hue, I think we still need some more discussions to see whether we want to do it or not.

@yeya24
Copy link
Contributor

yeya24 commented Nov 1, 2022

I have a WIP change to remove group label from all of the compactor metrics if people find this acceptable.

I think it is still valuable to have a label to identify each compaction group so that we can identify some key metrics like failutres.
Unfortunately the current label group is not a good identifier cause I cannot tell which cluster/tenant it is from the given label hash.

@SuperQ
Copy link
Contributor Author

SuperQ commented Nov 15, 2022

Metrics should identify when there are compaction failures worth investigating. The actual identity of what is failing can be determined from the logs.

@yeya24
Copy link
Contributor

yeya24 commented Jan 17, 2023

Let's do it. I think we should remove this label as the hash doesn't mean anything. However, I am wondering if we should still keep the downsampling information as another label.

SuperQ added a commit to SuperQ/thanos that referenced this issue Jan 17, 2023
@SuperQ
Remove group label from compact metrics
dfd12e2 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Add a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
@SuperQ
Copy link
Contributor Author

SuperQ commented Jan 17, 2023

Downsample resolution sounds like a useful label.

@SuperQ SuperQ mentioned this issue Jan 17, 2023
Merged
2 tasks
SuperQ added a commit to SuperQ/thanos that referenced this issue Jan 17, 2023
@SuperQ
Remove group label from compact metrics
65f9dd2 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Add a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Jan 17, 2023
@SuperQ
Remove group label from compact metrics
dbbd535 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Add a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Feb 22, 2023
@SuperQ
Remove group label from compact metrics
140e6ab 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Add a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Feb 22, 2023
@SuperQ
Remove group label from compact metrics
acf062d 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Add a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Feb 22, 2023
@SuperQ
Replace group with resolution in compact metrics.
0fe838a 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Feb 22, 2023
@SuperQ
Replace group with resolution in compact metrics.
e2a901e 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Mar 1, 2023
@SuperQ
Replace group with resolution in compact metrics.
20c5d01 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Mar 1, 2023
@SuperQ
Replace group with resolution in compact metrics.
a24ce81 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Mar 1, 2023
@SuperQ
Replace group with resolution in compact metrics.
66a9003 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Mar 20, 2023
@SuperQ
Replace group with resolution in compact metrics.
b3b4b0e 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Mar 22, 2023
@SuperQ
Replace group with resolution in compact metrics.
5c8bb5e 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue May 26, 2023
@SuperQ
Replace group with resolution in compact metrics.
522fe2e 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue May 30, 2023
@SuperQ
Replace group with resolution in compact metrics.
7eeb1bc 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
hczhu pushed a commit to databricks/thanos that referenced this issue Jun 27, 2023
@SuperQ
Replace group with resolution in compact metrics.
71c60c0 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
GiedriusS pushed a commit to vinted/thanos that referenced this issue Jul 27, 2023
@SuperQ @GiedriusS
Replace group with resolution in compact metrics.
6725b2a 
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
subbarao-meduri added a commit to stolostron/thanos that referenced this issue Sep 7, 2023
@subbarao-meduri @fpetkovski
@github-actions @alexqyle @Etienne-M @naveadkazi @wallee94 @douglascamata @rabenhorst @GiedriusS @philipgough @damnever @thibaultmg @saswatamcode @shayyxi @rueian @aimuz @yeya24 @ahurtaud @jacobbaungard @pgier @alanprot @PradyumnaKrishna @rgarcia89 @haanhvu @alxric @mhoffm-aiven @4orty @SuperQ @MichaHoffmann @vhbfernandes @jkowall @simonpasquier @Juneezee @jpds @bwplotka @heliapb @sigmaris @dbason @testwill @captncraig @kakkoyun @matej-g @zenbeam @xBazilio @anas-aso @TomMD @jonjohnsonjr
Upgrade to thanos 0.32.1 (#81)
b33e30b 
* Updates busybox SHA (#6283)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287)

* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346

Signed-off-by: Alex Le <[email protected]>

* Reverted test code

Signed-off-by: Alex Le <[email protected]>

* Updated comment

Signed-off-by: Alex Le <[email protected]>

* docs: mismatch in changelog

Signed-off-by: Etienne Martel <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Updates busybox SHA (#6283)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* trigger workflow

Signed-off-by: Alex Le <[email protected]>

* trigger workflow

Signed-off-by: Alex Le <[email protected]>

---------

Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Etienne Martel <[email protected]>
Signed-off-by: GitHub <[email protected]>
Co-authored-by: Etienne Martel <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <[email protected]>

* Add CarTrade Tech as new adopter

Signed-off-by: naveadkazi <[email protected]>

* sort ProxyResponseHeap comparing only diff labels

Signed-off-by: Walther Lee <[email protected]>

* proxy heap: remove store labels before comparing response series

Signed-off-by: Walther Lee <[email protected]>

* tests: Remove custom Between test matcher (#6310)

* Remove custom Between test matcher

The upstream PR to efficientgo/e2e has been merged, so we can use  it from there.

Signed-off-by: Douglas Camata <[email protected]>

* Run go mod tidy

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* Remove testingRespSet and update comparison func

Signed-off-by: Walther Lee <[email protected]>

* Fix tests

Signed-off-by: Walther Lee <[email protected]>

* query frontend, query UI: Native histogram support (#6071)

* Implemented native histogram support for qfe and query UI

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed marshalling for histograms in qfe

Started working on native histogram query ui

Copied histogram implementation for graph

Added query range support for native histograms in qfe

Use prom model (un-)marshal for native histograms in qfe

Use prom model (un-)marshal for native histograms in qfe

Fixed sample and sample stream marshal fn

Extended qfe native histogram e2e tests

Added copyright to qfe queryrange compat

Added query range test fo histograms and try to fix ui tests

Fixed DataTable test

Review feedback

Fixed native histogram e2e test

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Add histogram support for ApplyCounterResetsSeriesIterator

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Add chnagelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed changelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed qfe

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed PrometheusResponse minTime for histograms in qfe

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Updated prometheus common to v0.40.0 and queryrange.Sample fixes

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Updated Readme

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Addressed PR comments

Signed-off-by: Sebastian Rabenhorst <[email protected]>

trigger tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* fixed tsdbutil references

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* fixed imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Enabled pushdown for query native hist test and removed ToDo

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Refactored native histogram query UI

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* store: add streamed snappy encoding for postings list (#6303)

* store: add streamed snappy encoding for postings list

We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:

```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
      flat  flat%   sum%        cum   cum%
         0     0%     0%  1905.67GB 90.20%  golang.org/x/sync/errgroup.(*Group).Go.func1
    2.08GB 0.098% 0.098%  1456.94GB 68.96%  github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
    1.64GB 0.078%  0.18%  1454.87GB 68.86%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
    2.31GB  0.11%  0.29%  1258.15GB 59.55%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
    1.48GB  0.07%  0.36%  1219.67GB 57.73%  github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
 1215.21GB 57.52% 57.87%  1215.21GB 57.52%  github.com/klauspost/compress/s2.Decode
```

This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.

Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.

Benchmark data:

```
name                                                          time/op
PostingsEncodingDecoding/10000/raw/encode-16                  71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16                  76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16            73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16            1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16     111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16    14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16      1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16      14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16               710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16              79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16            719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16           13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16                7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16                81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16          7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16           139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16  11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16  15.5µs ± 4%

name                                                          alloc/op
PostingsEncodingDecoding/10000/raw/encode-16                  13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                   96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16            25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16            11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16    16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16     148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16       148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16       148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16               131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16               96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16            254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16            107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16                1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16                 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16          2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16          1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16  1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16   148kB ± 0%

name                                                          allocs/op
PostingsEncodingDecoding/10000/raw/encode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16              3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16              4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16      4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16      5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16        4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16        5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16                2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16                2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16             3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16             4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16                  2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16                  2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16            3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16            4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16    4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16    5.00 ± 0%
```

Compression ratios are still the same like previously:

```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN   TestDiffVarintCodec/snappy/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44498 bytes
    postings_codec_test.go:81: ratio: 0.012
=== RUN   TestDiffVarintCodec/snappyStreamed/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44670 bytes
    postings_codec_test.go:81: ratio: 0.012
```

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: clean up postings code

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: fix estimation

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: use buffer.Bytes()

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store/postings_codec: reuse extgrpc compressors/decompressors

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: add item

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: clean up whitespace

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: atomically replace no compact marked map (#6319)

With lots of blocks it could take some time to fill this no compact
marked map hence replace it atomically. I believe this leads to problems
in the compaction planner where it picks up no compact marked blocks
because meta syncer does synchronizations concurrently.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Reuse buffers for label comparison

Signed-off-by: Filip Petkovski <[email protected]>

* Use len instead of cap

Signed-off-by: Filip Petkovski <[email protected]>

* Use len instead of cap

Signed-off-by: Filip Petkovski <[email protected]>

* Fixed Jaeger exporter specification link (#6324)

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* receive: Make max backoff configurable (#6163)

Signed-off-by: Philip Gough <[email protected]>

* Replace summary in extprom metrics with histogram (#6327)

* Replaced summary in extprom metrics with histogram

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Added changelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unused parameters from NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Reverted NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Expose info for each TSDB

This commit exposes the label set alongside the min and max time
for each TSDB covered by a Store.

This information is used to scope the min time for a remote query
so that we do not produce partial aggregates in distriuted mode.

Signed-off-by: Filip Petkovski <[email protected]>

* Add test case for proxy store

Signed-off-by: Filip Petkovski <[email protected]>

* Pass engine factory by reference

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Cleanup

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Use NewQueryEngineFactory in tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Avoid expensive log.Valuer evaluation for disallowed levels (#6322)

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Fix inconsistent error for series limits in Store API (#6330)

* store: fix inconsistent error for series limits

Signed-off-by: Thibault Mange <[email protected]>

* update changelog

Signed-off-by: Thibault Mange <[email protected]>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Thibault Mange <[email protected]>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Thibault Mange <[email protected]>

* rename labelValues serires liimiter test function

Signed-off-by: Thibault Mange <[email protected]>

---------

Signed-off-by: Thibault Mange <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* *: remove unmaintained gzip library (#6332)

Switch from nytimes gzip library to the klaustpost's gzip code. The old
gzip HTTP handler shows up a lot in allocs so that's how I ended up
doing this change.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Traces sampler env var (#6306)

* Issue#5947 OTEL_TRACES_SAMPLER env var

Signed-off-by: shayyxi <[email protected]>

* Test correction

Signed-off-by: shayyxi <[email protected]>

* doc failure correction. parse float argument correction.

Signed-off-by: shayyxi <[email protected]>

* added the changelog.

Signed-off-by: shayyxi <[email protected]>

* ran make docs to fix the build failure.

Signed-off-by: shayyxi <[email protected]>

* corrected the incorrect change in tools.md

Signed-off-by: shayyxi <[email protected]>

* fixed review comments.

Signed-off-by: shayyxi <[email protected]>

---------

Signed-off-by: shayyxi <[email protected]>
Signed-off-by: Shazi <[email protected]>
Co-authored-by: shayyxi <[email protected]>

* query: use storepb.SeriesServer (#6334)

Use storepb.SeriesServer instead of the concrete struct. This allows
implementing functionality on top of the proxy.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342)

* use github.com/onsi/gomega/gleak to detect goroutine leak with timeout

Signed-off-by: Rueian <[email protected]>

* fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345)

Signed-off-by: aimuz <[email protected]>

* Return grpc code resource exhausted for byte limit error (#6325)

* return grpc code resource exhausted for byte limit error

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* update partial response strategy

Signed-off-by: Ben Ye <[email protected]>

* fix limit

Signed-off-by: Ben Ye <[email protected]>

* try to fix tests

Signed-off-by: Ben Ye <[email protected]>

* fix test error message

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349)

Signed-off-by: Alban HURTAUD <[email protected]>

* Optimize generating store match reason (#6355)

Removes reason generation when debug logging is not enabled.

Signed-off-by: Filip Petkovski <[email protected]>

* Updates busybox SHA (#6365)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Query: Add +Inf bucket to query duration metrics  (#6358)

* Query: Add +Inf bucket to query duration metrics

For the query duration metrics
(`thanos_store_api_query_duration_seconds`), we record query respond
latency, based on the size of the query (samples/series), and save to a
histogram.

However, when a query is made which exceeds the biggest sample/serie
size, we would prior to this commit, put the request into the largest
bucket.

With this commit, we instead create an `+Inf` bucket, and put requests
which are larger than the biggest defined bucket into that. This gives
more accurate results, and also allow one to see if the bucket sizes are
incorrectly sized.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Tests: Mutex around non-thread safe random source

When creating test blocks, we use a non-thread safe random source, in
multiple goroutines. Due to this, tests would sometime panic.

This commits puts a mutex around calls using the same source, in order
to avoid this.

This should hopefully improve reliability of e2e tests.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Check context when expanding postings (#6363)

* check context when expanding postings

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* ui: only keep name in store_matches param (#6371)

We are doing store matching on the `name` field hence only keep that
field in the URL because otherwise the URL could get quite lengthy with
external labelsets inside of it.

Besides unit tests, I have also tested locally:
- Enable store filtering;
- Select store(-s);
- Copy/paste URL into the new tab and see that the same stores are
  loaded like expected;
- See that URL only has names in them.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* docs: replace --store with --endpoint

Replace deprecated `--store` with `--endpoint` in docs.

Signed-off-by: Paul Gier <[email protected]>

* Optimizing "grafana generated" regex matchers (#6376)

* Opmizing Group Regex

Signed-off-by: Alan Protasio <[email protected]>

* fixing native histogram tests

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Cache: various index cache client improvements (#6374)

* e2e(query): Reproduce dedup issue from #6257

Signed-off-by: Douglas Camata <[email protected]>

* Add dedup e2e test for Receive

With internal and external labels support.

Signed-off-by: Douglas Camata <[email protected]>

* Simplify generated blocks for query test

Signed-off-by: Douglas Camata <[email protected]>

* Improve query dedup test

Signed-off-by: Douglas Camata <[email protected]>

* Write a query test for dedup with sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query dedup test with sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Fix Receive query test

Now it properly ensures the double dedup works (on internal and external labels).

Signed-off-by: Douglas Camata <[email protected]>

* Fix receive drawing

Signed-off-by: Douglas Camata <[email protected]>

* Add one extra test caes for query dedup from store

Signed-off-by: Douglas Camata <[email protected]>

* Complement test for Receive query with dedup

Signed-off-by: Douglas Camata <[email protected]>

* Complement test for Sidecar query dedup

Signed-off-by: Douglas Camata <[email protected]>

* Query Explanation (#6346)

* Return Query Explaination in QueryAPI

A param `explain` is added to QueryAPI, if true then explanation
returned by the `Explain()` method of the query having structure
`ExplainOutputNode` is returned in response.
Query Explanation is added under new field in response that is
`thanosInfo`.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add explain checkbox in thanos UI

A explain checkbox is added to Thanos Query UI, that requests for
query explanation from thanos query api.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add ExpandableNode Component

ExpandableNode component renders Query Explanation in the thanos
UI. Requires a new package `react-accessible-treeview`.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Disable Explain checkbox on prometheus engine

Prometheus engine sends out error if toggle explain button. To
provide better experience, the explain checkbox get disbaled on
switching to prometheus engine and enable back on switching to
thanos engine.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add alert box with horizontal scrolling for Explanation

Signed-off-by: Pradyumna Krishna <[email protected]>

* Remove ExpandableNode and Add ListTree

Updates the design for query explanation box, removes
`ExpandableNode` and the dependency. Builts a new `ListTree` that
does the same using reactstrap and custom css.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor refactor in Query API response

`thanosInfo` is removed from Query reponse and used `explanation`
directly. `disableCheckbox` is also renamed to
`disableExplainCheckbox` in thanos UI.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update UI tests to passing

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor UI changes and test fix

UI improvements and Panel test fix other way around, resetting
the results on panel construction.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update promql-engine to use Explain method

Signed-off-by: Pradyumna Krishna <[email protected]>

* Build UI assets

Build UI assets, that runs new thanos UI with explain button.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Revert proxy url change from package.json

`proxy` was accidently changed and committed with package.json
when removed dependency. Hence, reverting it back.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor changes in UI

Fix requested changes in UI.
- Rename `state` and `setState` to `mapping` and `setMapping`.
- Rename `NodeTree` to `QueryTree`.
- Use unicode characters instead of `-` and `+`.
- Fix blue box on explain button.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update UI assets

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379)

* Implementing Regex optimization on the MatchNotRegexp matcher type

Signed-off-by: Alan Protasio <[email protected]>

* Opmizing MatchNotEqual

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Expected failure of block label query dedup tests

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Put back the correct makefile

Signed-off-by: Douglas Camata <[email protected]>

* add de-cix as adopter (#6386)

Signed-off-by: Raul Garcia Sanchez <[email protected]>

* Remove extra line that broke untouched test

Signed-off-by: Douglas Camata <[email protected]>

* Add back line break at end of makefile

Signed-off-by: Douglas Camata <[email protected]>

* Fix Receive single ingestor test

Signed-off-by: Douglas Camata <[email protected]>

* Reproduce dedup issue in Receive

Signed-off-by: Douglas Camata <[email protected]>

* Add even more test cases for dedup on store gw

Signed-off-by: Douglas Camata <[email protected]>

* Reproduce dedup bug in Sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Reuse nginx image name

Signed-off-by: Douglas Camata <[email protected]>

* Let all users read the metrics file from static metrics server

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* [chore] Updating Query Engine and Prometheus (#6392)

* Updating Query Engine

Signed-off-by: Alan Protasio <[email protected]>

* fix prometheus breaking change

Signed-off-by: Alan Protasio <[email protected]>

* Update prometheus with https://github.com/prometheus/prometheus/pull/12387

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777)

Signed-off-by: haanhvu <[email protected]>

* check context cancel when doing posting batches (#6396)

Signed-off-by: Ben Ye <[email protected]>

* Expose store gateway query stats in series response hints (#6352)

* expose query stats hints

Signed-off-by: Ben Ye <[email protected]>

* update

Signed-off-by: Ben Ye <[email protected]>

* add query stats hints in result

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* add merge method

Signed-off-by: Ben Ye <[email protected]>

* fix unit test

Signed-off-by: Ben Ye <[email protected]>

modify hints proto

Signed-off-by: Ben Ye <[email protected]>

fix unit test

Signed-off-by: Ben Ye <[email protected]>

update format

Signed-off-by: Ben Ye <[email protected]>

* update comments

Signed-off-by: Ben Ye <[email protected]>

* try again

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* receive: make az aware ketama hashring (#6369)

* receive: make az aware ketama hashring

Signed-off-by: Alexander Rickardsson <[email protected]>

* receive: pass endpoints in hashring config as object

Signed-off-by: Michael Hoffmann <[email protected]>

* receive: add some tests for consistent hashing in presence of AZs

Signed-off-by: Michael Hoffmann <[email protected]>

* receive,docs: add migration note for az aware hashring

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Alexander Rickardsson <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* Reformat asciiflow chart

Signed-off-by: Douglas Camata <[email protected]>

* Proposal: query path tenancy  (#6320)

* Add 1st version of query path tenancy proposal

Signed-off-by: Douglas Camata <[email protected]>

* Update proposal after initial feedback

Signed-off-by: Douglas Camata <[email protected]>

* Add cool picture

Signed-off-by: Douglas Camata <[email protected]>

* Include example in cross tenant query complications

Signed-off-by: Douglas Camata <[email protected]>

* Improve reasoning for why not using the QFE

Signed-off-by: Douglas Camata <[email protected]>

* Improve writing in "How" section

Signed-off-by: Douglas Camata <[email protected]>

* Fix owner profile link

Signed-off-by: Douglas Camata <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Address few more PR review comments

Signed-off-by: Douglas Camata <[email protected]>

* Address feedback on flag name text

Signed-off-by: Douglas Camata <[email protected]>

* Update diagram

Signed-off-by: Douglas Camata <[email protected]>

* Improve non-goals text

Signed-off-by: Douglas Camata <[email protected]>

* Update diagram

Signed-off-by: Douglas Camata <[email protected]>

* Update docs/proposals-accepted/202304-query-path-tenancy.md

Co-authored-by: Filip Petkovski <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Clarify scenario for pitfalls of current solution

Signed-off-by: Douglas Camata <[email protected]>

* Clarify that Store doesn't care about tenant label

Signed-off-by: Douglas Camata <[email protected]>

* Add an action plan

Signed-off-by: Douglas Camata <[email protected]>

* Mention alternative idea of modifying Store API

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Co-authored-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Address lots of feedback on the proposal

Signed-off-by: Douglas Camata <[email protected]>

* Format query path tenancy proposal doc

Signed-off-by: Douglas Camata <[email protected]>

* Add a "Tenancy Model" subsection to "Goals"

Signed-off-by: Douglas Camata <[email protected]>

* Mention header semanthics in comparison with gRPC message field

Signed-off-by: Douglas Camata <[email protected]>

* Improve action plan structure and writing

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Filip Petkovski <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>

* Fix double-counting bug in http_request_duration metric (#6399)

* fix double-counting bug in http_request_duration metric

Signed-off-by: 4orty <[email protected]>

* Update Changelog

Signed-off-by: 4orty <[email protected]>

---------

Signed-off-by: 4orty <[email protected]>

* Replace group with resolution in compact metrics.

Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: https://github.com/thanos-io/thanos/issues/5841

Signed-off-by: SuperQ <[email protected]>

* Reuse static metrics server from e2e framework

Signed-off-by: Douglas Camata <[email protected]>

* Run go mod tidy

Signed-off-by: Douglas Camata <[email protected]>

* Updates busybox SHA (#6403)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Fix series stats merge (#6408)

* fix series stats merge

Signed-off-by: Ben Ye <[email protected]>

* update license header

Signed-off-by: Ben Ye <[email protected]>

* use reflect

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: allow unlimited head_series_limit tenants (#6406)

With this commit we now allow to configure tenants with unlimited active
series limit by setting the limit to `0`. Prior to this commit setting a
per tenant limit to `0` would cause the tenant to be unable to write any
metrics at all.

This fixes: https://github.com/thanos-io/thanos/issues/6393

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* expose downloaded data size in query hints (#6409)

Signed-off-by: Ben Ye <[email protected]>

* maintainers: add myself to triagers (#6414)

Signed-off-by: Michael Hoffmann <[email protected]>

* Add `@douglascamata` to triagers (#6418)

Signed-off-by: Douglas Camata <[email protected]>

* Add Blog (#6411)

* Add LFX blog

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add Headers to blog

Signed-off-by: Pradyumna Krishna <[email protected]>

* Lint blog

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* blog: Fix images for LFX post (#6422)

* blog: Fix images for LFX post

Signed-off-by: Saswata Mukherjee <[email protected]>

* fix lint

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Saswata Mukherjee <[email protected]>

* Index Cache: Change cache key for postings (#6405)

* extend postings cache key with codec

Signed-off-by: Ben Ye <[email protected]>

* add changelog

Signed-off-by: Ben Ye <[email protected]>

* update code back

Signed-off-by: Ben Ye <[email protected]>

* add colon

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* fix another test

Signed-off-by: Ben Ye <[email protected]>

* add compression scheme const to remote index cache

Signed-off-by: Ben Ye <[email protected]>

* address required comments

Signed-off-by: Ben Ye <[email protected]>

* fix compression scheme name

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: upgrading logs for failed uploads to error (#6427)

* FIX: upgrading log for failed upload to error

Signed-off-by: Victor Fernandes <[email protected]>

* docs: added changelog entry

Signed-off-by: Victor Fernandes <[email protected]>

---------

Signed-off-by: Victor Fernandes <[email protected]>

* fix postings test

Signed-off-by: Ben Ye <[email protected]>

* Add aiven as adopter... more soon!

Signed-off-by: Jonah Kowall <[email protected]>

* First draft of the Aiven adoption story

Signed-off-by: Jonah Kowall <[email protected]>

* Add aiven as adopter... more soon! (#6430)

Signed-off-by: Jonah Kowall <[email protected]>

* Report gRPC connnection errors to the caller (#6428)

By default `grpc.DialContext()` is non-blocking so any connection issue
will not be surfaced to the user. This change makes it blocking and
configures the gRPC dialer to report the underlying error if any
happens.

Signed-off-by: Simon Pasquier <[email protected]>

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432)

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep

`github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same
dependency. We can keep `github.com/fsnotify/fsnotify` and remove
`gopkg.in/fsnotify.v1`.

Signed-off-by: Eng Zer Jun <[email protected]>

* docs: add changelog

Signed-off-by: Eng Zer Jun <[email protected]>

---------

Signed-off-by: Eng Zer Jun <[email protected]>

* Expose estimated chunk and series size as configurable options (#6426)

* expose estimated chunk and series size as configurable options

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: make tsdb stats limit configurable (#6437)

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <[email protected]>

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Michael Hoffmann <[email protected]>

* *: wire new Engine/Explain fields in query-frontend (#6433)

- Pass Engine/Explain fields in query-frontend codecs
- Add Engine field to QFE cache key
- Add e2e tests for all cases

Signed-off-by: Giedrius Statkevičius <[email protected]>

* index cache: Cache expanded postings (#6420)

* cache expanded postings in index cache

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* fix

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* rebase main and added compression name to key

Signed-off-by: Ben Ye <[email protected]>

* update key

Signed-off-by: Ben Ye <[email protected]>

* add e2e test for memcached

Signed-off-by: Ben Ye <[email protected]>

* fix cache config

Signed-off-by: Ben Ye <[email protected]>

* address review comments

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* add approximate series size to index stats (#6425)

Signed-off-by: Ben Ye <[email protected]>

* index stats: fix chunk size calculation (#6424)

Signed-off-by: Ben Ye <[email protected]>

* Remove some unused Cortex vendored code and metrics (#6440)

* Avoid direct cortex vendor dependency in promclient (#6443)

* use own explanation struct

Signed-off-by: Ben Ye <[email protected]>

* omit empty

Signed-off-by: Ben Ye <[email protected]>

* fix e2e test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351)

Fixes: #6280

Signed-off-by: Jonathan Davies <[email protected]>

* Updates busybox SHA (#6448)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* store: read postings directly into delta encoded format (#6442)

Instead of allocating bytes for raw postings, let's read them directly
into diff varint format to save memory.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: Add Index Stats to block metadata (#6441)

* compactor set index stats on block

Signed-off-by: Ben Ye <[email protected]>

* comment

Signed-off-by: Ben Ye <[email protected]>

* change field

Signed-off-by: Ben Ye <[email protected]>

* update downsample

Signed-off-by: Ben Ye <[email protected]>

* add index stats into compaction

Signed-off-by: Ben Ye <[email protected]>

* fix tests

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* estimate block chunk and series size from metadata (#6449)

Signed-off-by: Ben Ye <[email protected]>

* Adding graphic for blog showing architecture

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing mermaid graph

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing mermaid graph again

Signed-off-by: Jonah Kowall <[email protected]>

* Remove image and fix mermaid for other flowchart

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing prod flowchart

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing prod flowchart title

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing md table

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing md table again

Signed-off-by: Jonah Kowall <[email protected]>

* Adding clarification on cost breakdowns

Signed-off-by: Jonah Kowall <[email protected]>

* Adding clarification on cost breakdowns

Signed-off-by: Jonah Kowall <[email protected]>

* Store: fix crash on empty regex matcher

Signed-off-by: Michael Hoffmann <[email protected]>

* Some corrections on text and links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing diagams

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing currencies

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing diagams

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing images

Signed-off-by: Jonah Kowall <[email protected]>

* Website: Fix blog images width (#6450)

* Set blog post images width to 100%

Blog Post images width isn't set and hence the images doesn't get
fit to the page. Setting width to 100% make sure images fit to
the web page and screen size.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Remove width constraint from blog images

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* Cache calculated mint and maxt for each remote engine (#6458)

The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times
on each remote engine. These calculations can get expensive when an engine
covers a large amount of TSDBs.

This commit introduces caching of the calculated values for a remote
engine's mint, maxt and label sets.

Signed-off-by: Filip Petkovski <[email protected]>

* cache empty expanded postings (#6464)

Signed-off-by: Ben Ye <[email protected]>

* mixin/receive: add limits alerting (#6466)

* add some thanos receive limits alerts

Signed-off-by: Thibault Mange <[email protected]>

* update changelog

Signed-off-by: Thibault Mange <[email protected]>

* fix and generate examples

Signed-off-by: Thibault Mange <[email protected]>

* quick fix

Signed-off-by: Thibault Mange <[email protected]>

* quick fix

Signed-off-by: Thibault Mange <[email protected]>

* fix test

Signed-off-by: Thibault Mange <[email protected]>

* fix invalid changelog

Signed-off-by: Thibault Mange <[email protected]>

---------

Signed-off-by: Thibault Mange <[email protected]>

* Update docs/blog/2023-06-08-thanos-at-aiven.md

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Jonah Kowall <[email protected]>

* Add retention for M3

Signed-off-by: Jonah Kowall <[email protected]>

* Add image versus mermaid diagram. Couldn't comment it out unfortunately

Signed-off-by: Jonah Kowall <[email protected]>

* Adding text around link to fix linter

Signed-off-by: Jonah Kowall <[email protected]>

* make docs

Signed-off-by: Jonah Kowall <[email protected]>

* add alert for tenant reaching head series limit (#6467)

Signed-off-by: Thibault Mange <[email protected]>

* go.mod: bump rest of otel libs (#6447)

* go.mod: bump rest of otel libs

Bump these two OTEL libs to be like the rest of the OTEL libs. We had a
strange issue where `service.name` was not set even though it was in
environment variables. Bumping these libs fixes the problem.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* otlp: only set service.name if provided

Signed-off-by: Giedrius Statkevičius <[email protected]>

* e2e: add test case

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: disable pooling for postings benchmarks (#6473)

Pooling hides the real allocations in postings benchmarks because the
allocation only happens once really. In reality this is different when
multiple readers are created at once. Disable pooling in benchmarks to
understand what is happening.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fixing date and author for Aiven blog (#6476)

* Fixing authors

Signed-off-by: Jonah Kowall <[email protected]>

* make docs and fixing date

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing authors

Signed-off-by: Jonah Kowall <[email protected]>

---------

Signed-off-by: Jonah Kowall <[email protected]>

* Check context when expanding postings 2nd attempt (#6471)

* check context when expanding postings

Signed-off-by: Ben Ye <[email protected]>

* import

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* [FEAT] Add thanos ruler blog post (#6477)

Signed-off-by: Hélia Barroso <[email protected]>

* blog: Small fixes to Aiven article (#6481)

* Correct some links and typos in 2023-06-08-thanos-at-aiven.md

The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links.
Also fixed a few other typos I spotted.

Signed-off-by: Hugh Cole-Baker <[email protected]>

* Update 2023-06-08-thanos-at-aiven.md

Clarify a couple of things in the final paragraph.

Signed-off-by: Hugh Cole-Baker <[email protected]>

---------

Signed-off-by: Hugh Cole-Baker <[email protected]>

* Rename series and postings fetch duration metrics (#6479)

* fix wrong metric names for series and postings fetch duration

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* note breaking change

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Update to prometheus/common v0.44.0 (#6483)

Signed-off-by: Dan Bason <[email protected]>

* go.mod: update rueidis client (#6485)

The latest improvements have great perf improvements. Let's update it.
Thank you to [rueian](https://github.com/rueian) and all of the
contributors to that awesome Redis client!

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Introduce tenancy package (#6482)

This commit introduces a tenancy package in preparation for introduction
of tenancy in the query path. Tenancy related code which can be common
between components are moved out of the receive component into the
tenancy package.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Ignore nginx.com links in mdox (#6489)

Signed-off-by: Saswata Mukherjee <[email protected]>

* blog: Update Aiven blog post (#6487)

Wording changes and better formatting for the tables.

Signed-off-by: Simon Brüggen <[email protected]>

* store: optimized snappy streamed reading (#6475)

Optimize snappy streamed reading by traversing through the byte slice in
advance to determine how big of a buffer we will need. I have opted to
rewrite snappy streamed format decoding because it is a straightforward
format. Complex parts were deferred to klauspost/compress.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Remove Exists call in meta fetcher (#6474)

* Remove Exists call in meta fetcher

The meta fetcher lists the bucket by iterating through
top level paths from the entire keyspace, and then for each key
calls Exists on the meta.json file. This leads to an N+1 amplification
against object storage, manifesting as a very high increase in costs
and occasional throttling by cloud providers.

This commit changes the iteration logic to recursively list all files instead,
which can be done with far fewer API calls to object storage. Meta files
are then identified from the keys returned by the list operation instead
of making one API request from each file.

Signed-off-by: Filip Petkovski <[email protected]>

* Revert e2e test change

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* e2e/store: use now instead of time.Now() each time (#6493)

Consistently use the same timestamp instead of the current time each
time during a test.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* [FIX] Documentation issue (#6494)

Signed-off-by: Hélia Barroso <[email protected]>

* Updates busybox SHA (#6497)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* chore: pkg imported more than once (#6499)

* chore: pkg imported more than once

Signed-off-by: guoguangwu <[email protected]>

* fix: remove pkg alias

Signed-off-by: guoguangwu <[email protected]>

---------

Signed-off-by: guoguangwu <[email protected]>

* *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496)

* Fix and improve PathContentReloader tests

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Add changelog entry

Signed-off-by: Douglas Camata <[email protected]>

* Fix table tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Co-authored-by: Filip Petkovski <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>
Co-authored-by: Filip Petkovski <[email protected]>

* add context check when decoding cached postings (#6506)

Signed-off-by: Ben Ye <[email protected]>

* *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503)

* Add fallback config reload for symlinks

Signed-off-by: Douglas Camata <[email protected]>

* Improve pooling config reload

Signed-off-by: Douglas Camata <[email protected]>

* Make tests more reliable

Signed-off-by: Douglas Camata <[email protected]>

* Improve tests again

Assert on runutil.Repeat output

Signed-off-by: Douglas Camata <[email protected]>

* Pass the debounce/reload time to each test

Signed-off-by: Douglas Camata <[email protected]>

* Add comments and ensure interfaces are implemented

Signed-off-by: Douglas Camata <[email protected]>

* Make the limit configuration reload time customizable

Signed-off-by: Douglas Camata <[email protected]>

* goimports file

Signed-off-by: Douglas Camata <[email protected]>

* Fix lint warning

Signed-off-by: Douglas Camata <[email protected]>

* Improve log for polling engine

Signed-off-by: Douglas Camata <[email protected]>

* Fix tests

Signed-off-by: Douglas Camata <[email protected]>

* Make linter happy

Signed-off-by: Douglas Camata <[email protected]>

* Extract symlink identification to a separate function

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix links to go-grpc-middleware after v2 merge into main

Signed-off-by: Douglas Camata <[email protected]>

* Add period to make linter happy.

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog entry

Signed-off-by: Douglas Camata <[email protected]>

* Replace the fsnotify based engine from `PathContentReloader` with the polling one

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Remove check for empty filePath

Signed-off-by: Douglas Camata <[email protected]>

* Unexpose pollingEngine.Start

Signed-off-by: Douglas Camata <[email protected]>

* Fix pollingEngine doc-comment

Signed-off-by: Douglas Camata <[email protected]>

* Log file path when config is reloaded

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* receive: add float histogram support (#6323)

* Added receive float histogram support

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Added comments for fns copied from Prometheus

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Improved comment

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unnecessary if

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed native histogram proto conversion in remote engine

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unused histogram conversion from remote engine

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fix and renaming in native_histograms_test.go

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Trigger Build

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* go.mod: bump e2e framework version (#6516)

Bump e2e framework version and add a note about E2E_DOCKER_CPUS.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Add changelog for #6474 (#6518)

Signed-off-by: Filip Petkovski <[email protected]>

* cortex/redisclient: use rueidis client (#6520)

* cortex/redisclient: use rueidis client

Use the same rueidis client in query-frontend. Solves
https://github.com/thanos-io/thanos/issues/6094.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cortex/cache: gofumpt + fix errors.Errorf

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cacheutil/docs: clean up more old stuff

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* index header: Remove memWriter from fileWriter (#6509)

* index header: remove memWriter from fileWriter

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* refactor

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* update comment

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Wrap object store Prometheus registry (#6152)

* Wrap object store Prometheus registery

In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26.

Signed-off-by: Douglas Camata <[email protected]>

* Update objstore to the latest version

This version removes the `thanos_` prefix from metrics, which is the
reason for wrapping the objstore's metrics registry in the first place.

Signed-off-by: Douglas Camata <[email protected]>

* Fix modules

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Wrap metrics registerer for objstore bucket

Signed-off-by: Douglas Camata <[email protected]>

* Remove prefix from Thanos Store metrics

Signed-off-by: Douglas Camata <[email protected]>

* Fix goimports

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Put back upgraded objstore dep

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Move to more recent ref of thanos/objstore

Signed-off-by: Douglas Camata <[email protected]>

* Ignore OBS in objstore tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix linting error on main test after objstore upgrade

Signed-off-by: Douglas Camata <[email protected]>

* Skip OCS objstore test in circle ci

Signed-off-by: Douglas Camata <[email protected]>

* Fix echo in makefile

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* e2e/store: try to fix Series() limit test again (#6522)

I finally managed to reproduce this failure locally with
https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb.
The added t.Logf() showed that is the problem is that with a lower bytes
limit, it might hit the series or chunks part first. I have bumped the
bytes limit. I calculated the new bytes limit by checking how much bytes
are allocated before sending the last chunk.

I have also noticed that one block is created without a delay. Update it
so that it would be like the others.

Include objstore@main update with
https://github.com/thanos-io/objstore/pull/62/files so that Iter() would
always return an error on a timeout.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* pkg/reloader: use watchInterval timeout for initial apply (#6519)

* pkg/reloader: use watchInterval timeout for initial apply

Signed-off-by: Craig Peterson <[email protected]>

* changelog

Signed-off-by: Craig Peterson <[email protected]>

* use distinct context for initial sync

Signed-off-by: Craig Peterson <[email protected]>

---------

Signed-off-by: Craig Peterson <[email protected]>

* Standardize index cache metrics (#6523)

* Add histogram metrics for index cache item size (#6528)

* add histogram metrics for index cache item size

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Upgrade objstore (#6507)

* Upgrade objstore

Signed-off-by: Kemal Akkoyun <[email protected]>

* Update with the latest APIs

Signed-off-by: Kemal Akkoyun <[email protected]>

---------

Signed-off-by: Kemal Akkoyun <[email protected]>
Co-authored-by: Matej Gera <[email protected]>

* Make compact lifecycle more flexible to be overridden for sharded compaction (#5964)

* Updates busybox SHA (#6534)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* BucketedBytes to buffer byte slices when decoding postings from cache (#6531)

* BucketedBytes to buffer byte slices when deconding postings from cache

Signed-off-by: Alan Protasio <[email protected]>

* Fixing Lint

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Shipper: change upload compacted type from bool to a function (#6526)

* change shipper upload compacted type from bool to a function

Signed-off-by: Ben Ye <[email protected]>

* add default to false

Signed-off-by: Ben Ye <[email protected]>

* reset uploadedCompacted to 0

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Query: Forward tenant information via StoreAPI (#6530)

* Querier: Forward tenant information downstream

With this commit we attach tenant information to each query request and
forward it via the StoreAPI to any downstream Store Gateways and
Queriers.

We add the following command lines options which mimics the tenant
functionality in Receive. The options are currently hidden, as they
provide no real functionality yet. This will come in future steps.

--query.tenant-header
--query.default-tenant
--query.tenant-certificate

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Receive: Use CertificateField from Tenancy pkg

These consts are now defined in the Tenancy package, so we should use
those instead.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Deduplicate matchers in posting group (#6532)

* deduplicate matchers in posting group

Signed-off-by: Ben Ye <[email protected]>

* don't use map for deduplication

Signed-off-by: Ben Ye <[email protected]>

* lint

Signed-off-by: Ben Ye <[email protected]>

* address comments

Signed-off-by: Ben Ye <[email protected]>

* fi…
subbarao-meduri added a commit to stolostron/thanos that referenced this issue Sep 21, 2023
@subbarao-meduri @junotx
@PradyumnaKrishna @Etienne-M @fpetkovski @github-actions @alexqyle @naveadkazi @wallee94 @douglascamata @rabenhorst @GiedriusS @philipgough @damnever @thibaultmg @saswatamcode @shayyxi @rueian @aimuz @yeya24 @ahurtaud @jacobbaungard @pgier @alanprot @rgarcia89 @haanhvu @alxric @mhoffm-aiven @4orty @SuperQ @MichaHoffmann @vhbfernandes @jkowall @simonpasquier @Juneezee @jpds @bwplotka @heliapb @sigmaris @dbason @testwill @captncraig @kakkoyun @matej-g @zenbeam @xBazilio @anas-aso @TomMD @jonjohnsonjr @dependabot @ritaCanavarro
[release-2.9] Upgrade to thanos 0.32.3 (#84)
8727d9f 
* fix 0.31 changelog (#6278)

Signed-off-by: junot <[email protected]>

* Query: Switch Multiple Engines (#6234)

* Query: Switch engines using `engine` param

Thanos query has two engine, prometheus (default) and thanos.
A single engine runs through thanos query command at a time, and
have to re run the command to switch between.

This commit adds a functionality to run multiple engines at once
and switch between them using `engine` query param inq query api.

To avoid duplicate matrics registration, the thanos engine is
provided with a different registerer having prefix `tpe_` (not
been finalized yet).

promql-engine command line flag has been removed that specifies
the query engine to run.

Currently this functionality not implemented on GRPCAPI.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add multiple engine support to GRPCAPI

Fix build fail for thanos, adds support for multiple engine in
GRPCAPI.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Create QueryEngineFactory to create engines

QueryEngineFactory makes a collection for all promql engines used
by thanos and returns it. Any engine can be created and returned
using `GetXEngine` method.

It is currently limited to 2 engines prometheus and thanos engines
that get created on the first call.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Use QueryEngineFactory in query API

thanos query commands pass `QueryEngineFactory` to query apis
that will use engine based on query params. It will provide more
flexibility to create multiple engines in thanos.

Adds `defaultEngine` CLI flag, A default engine to use if not
specified with query params.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update Query API tests

Fixes breaking tests

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor changes and Docs fixes

* Move defaultEngine argument to reduce diff.
* Generated Docs.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add Engine Selector/ Dropdown to Query UI

Engine Selector is a dropdown that sets an engine to be used to
run the query. Currently two engines `thanos` and `prometheus`.

This dropdown sends a query param `engine` to query api, which
runs the api using the engine provided. Provided to run query
using multiple query engines from Query UI.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Move Engine Selector to Panel

Removes Dropdown component, and renders Engine Selector directly.
Receives defaultEngine from `flags` API.
Updates parseOptions to parse engine query param and updates test
for Panel and utils.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Upgrade promql-engine dependency

Updates promql-engine that brings functionality to provide
fallback engine using enigne Opts.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add MinT to remote client

MinT method was missing from Client due to updated promql-engine.
This commits adds mint to the remote client.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Use prometheus fallback engine in thanos engine

Thanos engine creates a fallback prometheus engine that conflicts
with another prometheus engine created by thanos, while
registering metrics. To fix this, provided created thanos engine
as fallback engine to thanos engine in engine Opts.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Use enum for EngineType in GRPC

GRPC is used for communication between thanos components and
defaultEngine was a string before. Enum makes more sense, and
hence the request.Enigne type has been changed to
querypb.EngineType.
Default case is handled with another default value provided in
the enum.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update query UI bindata.go

Compile react app using `make assets`.

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* docs: mismatch in changelog

Signed-off-by: Etienne Martel <[email protected]>

* Updates busybox SHA (#6283)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287)

* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346

Signed-off-by: Alex Le <[email protected]>

* Reverted test code

Signed-off-by: Alex Le <[email protected]>

* Updated comment

Signed-off-by: Alex Le <[email protected]>

* docs: mismatch in changelog

Signed-off-by: Etienne Martel <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Updates busybox SHA (#6283)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* trigger workflow

Signed-off-by: Alex Le <[email protected]>

* trigger workflow

Signed-off-by: Alex Le <[email protected]>

---------

Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Etienne Martel <[email protected]>
Signed-off-by: GitHub <[email protected]>
Co-authored-by: Etienne Martel <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <[email protected]>

* Add CarTrade Tech as new adopter

Signed-off-by: naveadkazi <[email protected]>

* sort ProxyResponseHeap comparing only diff labels

Signed-off-by: Walther Lee <[email protected]>

* proxy heap: remove store labels before comparing response series

Signed-off-by: Walther Lee <[email protected]>

* tests: Remove custom Between test matcher (#6310)

* Remove custom Between test matcher

The upstream PR to efficientgo/e2e has been merged, so we can use  it from there.

Signed-off-by: Douglas Camata <[email protected]>

* Run go mod tidy

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* Remove testingRespSet and update comparison func

Signed-off-by: Walther Lee <[email protected]>

* Fix tests

Signed-off-by: Walther Lee <[email protected]>

* query frontend, query UI: Native histogram support (#6071)

* Implemented native histogram support for qfe and query UI

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed marshalling for histograms in qfe

Started working on native histogram query ui

Copied histogram implementation for graph

Added query range support for native histograms in qfe

Use prom model (un-)marshal for native histograms in qfe

Use prom model (un-)marshal for native histograms in qfe

Fixed sample and sample stream marshal fn

Extended qfe native histogram e2e tests

Added copyright to qfe queryrange compat

Added query range test fo histograms and try to fix ui tests

Fixed DataTable test

Review feedback

Fixed native histogram e2e test

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Add histogram support for ApplyCounterResetsSeriesIterator

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Add chnagelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed changelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed qfe

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed PrometheusResponse minTime for histograms in qfe

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Updated prometheus common to v0.40.0 and queryrange.Sample fixes

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Updated Readme

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Addressed PR comments

Signed-off-by: Sebastian Rabenhorst <[email protected]>

trigger tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Made assets

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* fixed tsdbutil references

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* fixed imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Enabled pushdown for query native hist test and removed ToDo

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Refactored native histogram query UI

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* store: add streamed snappy encoding for postings list (#6303)

* store: add streamed snappy encoding for postings list

We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:

```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
      flat  flat%   sum%        cum   cum%
         0     0%     0%  1905.67GB 90.20%  golang.org/x/sync/errgroup.(*Group).Go.func1
    2.08GB 0.098% 0.098%  1456.94GB 68.96%  github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
    1.64GB 0.078%  0.18%  1454.87GB 68.86%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
    2.31GB  0.11%  0.29%  1258.15GB 59.55%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
    1.48GB  0.07%  0.36%  1219.67GB 57.73%  github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
 1215.21GB 57.52% 57.87%  1215.21GB 57.52%  github.com/klauspost/compress/s2.Decode
```

This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.

Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.

Benchmark data:

```
name                                                          time/op
PostingsEncodingDecoding/10000/raw/encode-16                  71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16                  76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16            73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16            1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16     111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16    14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16      1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16      14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16               710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16              79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16            719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16           13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16                7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16                81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16          7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16           139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16  11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16  15.5µs ± 4%

name                                                          alloc/op
PostingsEncodingDecoding/10000/raw/encode-16                  13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                   96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16            25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16            11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16    16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16     148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16       148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16       148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16               131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16               96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16            254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16            107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16                1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16                 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16          2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16          1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16  1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16   148kB ± 0%

name                                                          allocs/op
PostingsEncodingDecoding/10000/raw/encode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16              3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16              4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16      4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16      5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16        4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16        5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16                2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16                2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16             3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16             4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16                  2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16                  2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16            3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16            4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16    4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16    5.00 ± 0%
```

Compression ratios are still the same like previously:

```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN   TestDiffVarintCodec/snappy/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44498 bytes
    postings_codec_test.go:81: ratio: 0.012
=== RUN   TestDiffVarintCodec/snappyStreamed/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44670 bytes
    postings_codec_test.go:81: ratio: 0.012
```

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: clean up postings code

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: fix estimation

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: use buffer.Bytes()

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store/postings_codec: reuse extgrpc compressors/decompressors

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: add item

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: clean up whitespace

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: atomically replace no compact marked map (#6319)

With lots of blocks it could take some time to fill this no compact
marked map hence replace it atomically. I believe this leads to problems
in the compaction planner where it picks up no compact marked blocks
because meta syncer does synchronizations concurrently.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Reuse buffers for label comparison

Signed-off-by: Filip Petkovski <[email protected]>

* Use len instead of cap

Signed-off-by: Filip Petkovski <[email protected]>

* Use len instead of cap

Signed-off-by: Filip Petkovski <[email protected]>

* Fixed Jaeger exporter specification link (#6324)

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* receive: Make max backoff configurable (#6163)

Signed-off-by: Philip Gough <[email protected]>

* Replace summary in extprom metrics with histogram (#6327)

* Replaced summary in extprom metrics with histogram

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Added changelog

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unused parameters from NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Reverted NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Expose info for each TSDB

This commit exposes the label set alongside the min and max time
for each TSDB covered by a Store.

This information is used to scope the min time for a remote query
so that we do not produce partial aggregates in distriuted mode.

Signed-off-by: Filip Petkovski <[email protected]>

* Add test case for proxy store

Signed-off-by: Filip Petkovski <[email protected]>

* Pass engine factory by reference

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Cleanup

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Use NewQueryEngineFactory in tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Avoid expensive log.Valuer evaluation for disallowed levels (#6322)

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Fix inconsistent error for series limits in Store API (#6330)

* store: fix inconsistent error for series limits

Signed-off-by: Thibault Mange <[email protected]>

* update changelog

Signed-off-by: Thibault Mange <[email protected]>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Thibault Mange <[email protected]>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Thibault Mange <[email protected]>

* rename labelValues serires liimiter test function

Signed-off-by: Thibault Mange <[email protected]>

---------

Signed-off-by: Thibault Mange <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* *: remove unmaintained gzip library (#6332)

Switch from nytimes gzip library to the klaustpost's gzip code. The old
gzip HTTP handler shows up a lot in allocs so that's how I ended up
doing this change.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Traces sampler env var (#6306)

* Issue#5947 OTEL_TRACES_SAMPLER env var

Signed-off-by: shayyxi <[email protected]>

* Test correction

Signed-off-by: shayyxi <[email protected]>

* doc failure correction. parse float argument correction.

Signed-off-by: shayyxi <[email protected]>

* added the changelog.

Signed-off-by: shayyxi <[email protected]>

* ran make docs to fix the build failure.

Signed-off-by: shayyxi <[email protected]>

* corrected the incorrect change in tools.md

Signed-off-by: shayyxi <[email protected]>

* fixed review comments.

Signed-off-by: shayyxi <[email protected]>

---------

Signed-off-by: shayyxi <[email protected]>
Signed-off-by: Shazi <[email protected]>
Co-authored-by: shayyxi <[email protected]>

* query: use storepb.SeriesServer (#6334)

Use storepb.SeriesServer instead of the concrete struct. This allows
implementing functionality on top of the proxy.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342)

* use github.com/onsi/gomega/gleak to detect goroutine leak with timeout

Signed-off-by: Rueian <[email protected]>

* fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345)

Signed-off-by: aimuz <[email protected]>

* Return grpc code resource exhausted for byte limit error (#6325)

* return grpc code resource exhausted for byte limit error

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* update partial response strategy

Signed-off-by: Ben Ye <[email protected]>

* fix limit

Signed-off-by: Ben Ye <[email protected]>

* try to fix tests

Signed-off-by: Ben Ye <[email protected]>

* fix test error message

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349)

Signed-off-by: Alban HURTAUD <[email protected]>

* Optimize generating store match reason (#6355)

Removes reason generation when debug logging is not enabled.

Signed-off-by: Filip Petkovski <[email protected]>

* Updates busybox SHA (#6365)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Query: Add +Inf bucket to query duration metrics  (#6358)

* Query: Add +Inf bucket to query duration metrics

For the query duration metrics
(`thanos_store_api_query_duration_seconds`), we record query respond
latency, based on the size of the query (samples/series), and save to a
histogram.

However, when a query is made which exceeds the biggest sample/serie
size, we would prior to this commit, put the request into the largest
bucket.

With this commit, we instead create an `+Inf` bucket, and put requests
which are larger than the biggest defined bucket into that. This gives
more accurate results, and also allow one to see if the bucket sizes are
incorrectly sized.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Tests: Mutex around non-thread safe random source

When creating test blocks, we use a non-thread safe random source, in
multiple goroutines. Due to this, tests would sometime panic.

This commits puts a mutex around calls using the same source, in order
to avoid this.

This should hopefully improve reliability of e2e tests.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Check context when expanding postings (#6363)

* check context when expanding postings

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* ui: only keep name in store_matches param (#6371)

We are doing store matching on the `name` field hence only keep that
field in the URL because otherwise the URL could get quite lengthy with
external labelsets inside of it.

Besides unit tests, I have also tested locally:
- Enable store filtering;
- Select store(-s);
- Copy/paste URL into the new tab and see that the same stores are
  loaded like expected;
- See that URL only has names in them.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* docs: replace --store with --endpoint

Replace deprecated `--store` with `--endpoint` in docs.

Signed-off-by: Paul Gier <[email protected]>

* Optimizing "grafana generated" regex matchers (#6376)

* Opmizing Group Regex

Signed-off-by: Alan Protasio <[email protected]>

* fixing native histogram tests

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Cache: various index cache client improvements (#6374)

* e2e(query): Reproduce dedup issue from #6257

Signed-off-by: Douglas Camata <[email protected]>

* Add dedup e2e test for Receive

With internal and external labels support.

Signed-off-by: Douglas Camata <[email protected]>

* Simplify generated blocks for query test

Signed-off-by: Douglas Camata <[email protected]>

* Improve query dedup test

Signed-off-by: Douglas Camata <[email protected]>

* Write a query test for dedup with sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query dedup test with sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Fix Receive query test

Now it properly ensures the double dedup works (on internal and external labels).

Signed-off-by: Douglas Camata <[email protected]>

* Fix receive drawing

Signed-off-by: Douglas Camata <[email protected]>

* Add one extra test caes for query dedup from store

Signed-off-by: Douglas Camata <[email protected]>

* Complement test for Receive query with dedup

Signed-off-by: Douglas Camata <[email protected]>

* Complement test for Sidecar query dedup

Signed-off-by: Douglas Camata <[email protected]>

* Query Explanation (#6346)

* Return Query Explaination in QueryAPI

A param `explain` is added to QueryAPI, if true then explanation
returned by the `Explain()` method of the query having structure
`ExplainOutputNode` is returned in response.
Query Explanation is added under new field in response that is
`thanosInfo`.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add explain checkbox in thanos UI

A explain checkbox is added to Thanos Query UI, that requests for
query explanation from thanos query api.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add ExpandableNode Component

ExpandableNode component renders Query Explanation in the thanos
UI. Requires a new package `react-accessible-treeview`.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Disable Explain checkbox on prometheus engine

Prometheus engine sends out error if toggle explain button. To
provide better experience, the explain checkbox get disbaled on
switching to prometheus engine and enable back on switching to
thanos engine.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add alert box with horizontal scrolling for Explanation

Signed-off-by: Pradyumna Krishna <[email protected]>

* Remove ExpandableNode and Add ListTree

Updates the design for query explanation box, removes
`ExpandableNode` and the dependency. Builts a new `ListTree` that
does the same using reactstrap and custom css.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor refactor in Query API response

`thanosInfo` is removed from Query reponse and used `explanation`
directly. `disableCheckbox` is also renamed to
`disableExplainCheckbox` in thanos UI.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update UI tests to passing

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor UI changes and test fix

UI improvements and Panel test fix other way around, resetting
the results on panel construction.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update promql-engine to use Explain method

Signed-off-by: Pradyumna Krishna <[email protected]>

* Build UI assets

Build UI assets, that runs new thanos UI with explain button.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Revert proxy url change from package.json

`proxy` was accidently changed and committed with package.json
when removed dependency. Hence, reverting it back.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Minor changes in UI

Fix requested changes in UI.
- Rename `state` and `setState` to `mapping` and `setMapping`.
- Rename `NodeTree` to `QueryTree`.
- Use unicode characters instead of `-` and `+`.
- Fix blue box on explain button.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Update UI assets

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379)

* Implementing Regex optimization on the MatchNotRegexp matcher type

Signed-off-by: Alan Protasio <[email protected]>

* Opmizing MatchNotEqual

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Expected failure of block label query dedup tests

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Put back the correct makefile

Signed-off-by: Douglas Camata <[email protected]>

* add de-cix as adopter (#6386)

Signed-off-by: Raul Garcia Sanchez <[email protected]>

* Remove extra line that broke untouched test

Signed-off-by: Douglas Camata <[email protected]>

* Add back line break at end of makefile

Signed-off-by: Douglas Camata <[email protected]>

* Fix Receive single ingestor test

Signed-off-by: Douglas Camata <[email protected]>

* Reproduce dedup issue in Receive

Signed-off-by: Douglas Camata <[email protected]>

* Add even more test cases for dedup on store gw

Signed-off-by: Douglas Camata <[email protected]>

* Reproduce dedup bug in Sidecar

Signed-off-by: Douglas Camata <[email protected]>

* Reuse nginx image name

Signed-off-by: Douglas Camata <[email protected]>

* Let all users read the metrics file from static metrics server

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* [chore] Updating Query Engine and Prometheus (#6392)

* Updating Query Engine

Signed-off-by: Alan Protasio <[email protected]>

* fix prometheus breaking change

Signed-off-by: Alan Protasio <[email protected]>

* Update prometheus with https://github.com/prometheus/prometheus/pull/12387

Signed-off-by: Alan Protasio <[email protected]>

---------

Signed-off-by: Alan Protasio <[email protected]>

* Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777)

Signed-off-by: haanhvu <[email protected]>

* check context cancel when doing posting batches (#6396)

Signed-off-by: Ben Ye <[email protected]>

* Expose store gateway query stats in series response hints (#6352)

* expose query stats hints

Signed-off-by: Ben Ye <[email protected]>

* update

Signed-off-by: Ben Ye <[email protected]>

* add query stats hints in result

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* add merge method

Signed-off-by: Ben Ye <[email protected]>

* fix unit test

Signed-off-by: Ben Ye <[email protected]>

modify hints proto

Signed-off-by: Ben Ye <[email protected]>

fix unit test

Signed-off-by: Ben Ye <[email protected]>

update format

Signed-off-by: Ben Ye <[email protected]>

* update comments

Signed-off-by: Ben Ye <[email protected]>

* try again

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* receive: make az aware ketama hashring (#6369)

* receive: make az aware ketama hashring

Signed-off-by: Alexander Rickardsson <[email protected]>

* receive: pass endpoints in hashring config as object

Signed-off-by: Michael Hoffmann <[email protected]>

* receive: add some tests for consistent hashing in presence of AZs

Signed-off-by: Michael Hoffmann <[email protected]>

* receive,docs: add migration note for az aware hashring

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Alexander Rickardsson <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* Reformat asciiflow chart

Signed-off-by: Douglas Camata <[email protected]>

* Proposal: query path tenancy  (#6320)

* Add 1st version of query path tenancy proposal

Signed-off-by: Douglas Camata <[email protected]>

* Update proposal after initial feedback

Signed-off-by: Douglas Camata <[email protected]>

* Add cool picture

Signed-off-by: Douglas Camata <[email protected]>

* Include example in cross tenant query complications

Signed-off-by: Douglas Camata <[email protected]>

* Improve reasoning for why not using the QFE

Signed-off-by: Douglas Camata <[email protected]>

* Improve writing in "How" section

Signed-off-by: Douglas Camata <[email protected]>

* Fix owner profile link

Signed-off-by: Douglas Camata <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Address few more PR review comments

Signed-off-by: Douglas Camata <[email protected]>

* Address feedback on flag name text

Signed-off-by: Douglas Camata <[email protected]>

* Update diagram

Signed-off-by: Douglas Camata <[email protected]>

* Improve non-goals text

Signed-off-by: Douglas Camata <[email protected]>

* Update diagram

Signed-off-by: Douglas Camata <[email protected]>

* Update docs/proposals-accepted/202304-query-path-tenancy.md

Co-authored-by: Filip Petkovski <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Clarify scenario for pitfalls of current solution

Signed-off-by: Douglas Camata <[email protected]>

* Clarify that Store doesn't care about tenant label

Signed-off-by: Douglas Camata <[email protected]>

* Add an action plan

Signed-off-by: Douglas Camata <[email protected]>

* Mention alternative idea of modifying Store API

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Co-authored-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Address lots of feedback on the proposal

Signed-off-by: Douglas Camata <[email protected]>

* Format query path tenancy proposal doc

Signed-off-by: Douglas Camata <[email protected]>

* Add a "Tenancy Model" subsection to "Goals"

Signed-off-by: Douglas Camata <[email protected]>

* Mention header semanthics in comparison with gRPC message field

Signed-off-by: Douglas Camata <[email protected]>

* Improve action plan structure and writing

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Filip Petkovski <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>

* Fix double-counting bug in http_request_duration metric (#6399)

* fix double-counting bug in http_request_duration metric

Signed-off-by: 4orty <[email protected]>

* Update Changelog

Signed-off-by: 4orty <[email protected]>

---------

Signed-off-by: 4orty <[email protected]>

* Replace group with resolution in compact metrics.

Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.

Fixes: https://github.com/thanos-io/thanos/issues/5841

Signed-off-by: SuperQ <[email protected]>

* Reuse static metrics server from e2e framework

Signed-off-by: Douglas Camata <[email protected]>

* Run go mod tidy

Signed-off-by: Douglas Camata <[email protected]>

* Updates busybox SHA (#6403)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Fix series stats merge (#6408)

* fix series stats merge

Signed-off-by: Ben Ye <[email protected]>

* update license header

Signed-off-by: Ben Ye <[email protected]>

* use reflect

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: allow unlimited head_series_limit tenants (#6406)

With this commit we now allow to configure tenants with unlimited active
series limit by setting the limit to `0`. Prior to this commit setting a
per tenant limit to `0` would cause the tenant to be unable to write any
metrics at all.

This fixes: https://github.com/thanos-io/thanos/issues/6393

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* expose downloaded data size in query hints (#6409)

Signed-off-by: Ben Ye <[email protected]>

* maintainers: add myself to triagers (#6414)

Signed-off-by: Michael Hoffmann <[email protected]>

* Add `@douglascamata` to triagers (#6418)

Signed-off-by: Douglas Camata <[email protected]>

* Add Blog (#6411)

* Add LFX blog

Signed-off-by: Pradyumna Krishna <[email protected]>

* Add Headers to blog

Signed-off-by: Pradyumna Krishna <[email protected]>

* Lint blog

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* blog: Fix images for LFX post (#6422)

* blog: Fix images for LFX post

Signed-off-by: Saswata Mukherjee <[email protected]>

* fix lint

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Saswata Mukherjee <[email protected]>

* Index Cache: Change cache key for postings (#6405)

* extend postings cache key with codec

Signed-off-by: Ben Ye <[email protected]>

* add changelog

Signed-off-by: Ben Ye <[email protected]>

* update code back

Signed-off-by: Ben Ye <[email protected]>

* add colon

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* fix another test

Signed-off-by: Ben Ye <[email protected]>

* add compression scheme const to remote index cache

Signed-off-by: Ben Ye <[email protected]>

* address required comments

Signed-off-by: Ben Ye <[email protected]>

* fix compression scheme name

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: upgrading logs for failed uploads to error (#6427)

* FIX: upgrading log for failed upload to error

Signed-off-by: Victor Fernandes <[email protected]>

* docs: added changelog entry

Signed-off-by: Victor Fernandes <[email protected]>

---------

Signed-off-by: Victor Fernandes <[email protected]>

* fix postings test

Signed-off-by: Ben Ye <[email protected]>

* Add aiven as adopter... more soon!

Signed-off-by: Jonah Kowall <[email protected]>

* First draft of the Aiven adoption story

Signed-off-by: Jonah Kowall <[email protected]>

* Add aiven as adopter... more soon! (#6430)

Signed-off-by: Jonah Kowall <[email protected]>

* Report gRPC connnection errors to the caller (#6428)

By default `grpc.DialContext()` is non-blocking so any connection issue
will not be surfaced to the user. This change makes it blocking and
configures the gRPC dialer to report the underlying error if any
happens.

Signed-off-by: Simon Pasquier <[email protected]>

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432)

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep

`github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same
dependency. We can keep `github.com/fsnotify/fsnotify` and remove
`gopkg.in/fsnotify.v1`.

Signed-off-by: Eng Zer Jun <[email protected]>

* docs: add changelog

Signed-off-by: Eng Zer Jun <[email protected]>

---------

Signed-off-by: Eng Zer Jun <[email protected]>

* Expose estimated chunk and series size as configurable options (#6426)

* expose estimated chunk and series size as configurable options

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Receive: make tsdb stats limit configurable (#6437)

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <[email protected]>

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Michael Hoffmann <[email protected]>

* *: wire new Engine/Explain fields in query-frontend (#6433)

- Pass Engine/Explain fields in query-frontend codecs
- Add Engine field to QFE cache key
- Add e2e tests for all cases

Signed-off-by: Giedrius Statkevičius <[email protected]>

* index cache: Cache expanded postings (#6420)

* cache expanded postings in index cache

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* fix

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* rebase main and added compression name to key

Signed-off-by: Ben Ye <[email protected]>

* update key

Signed-off-by: Ben Ye <[email protected]>

* add e2e test for memcached

Signed-off-by: Ben Ye <[email protected]>

* fix cache config

Signed-off-by: Ben Ye <[email protected]>

* address review comments

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* add approximate series size to index stats (#6425)

Signed-off-by: Ben Ye <[email protected]>

* index stats: fix chunk size calculation (#6424)

Signed-off-by: Ben Ye <[email protected]>

* Remove some unused Cortex vendored code and metrics (#6440)

* Avoid direct cortex vendor dependency in promclient (#6443)

* use own explanation struct

Signed-off-by: Ben Ye <[email protected]>

* omit empty

Signed-off-by: Ben Ye <[email protected]>

* fix e2e test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351)

Fixes: #6280

Signed-off-by: Jonathan Davies <[email protected]>

* Updates busybox SHA (#6448)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* store: read postings directly into delta encoded format (#6442)

Instead of allocating bytes for raw postings, let's read them directly
into diff varint format to save memory.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: Add Index Stats to block metadata (#6441)

* compactor set index stats on block

Signed-off-by: Ben Ye <[email protected]>

* comment

Signed-off-by: Ben Ye <[email protected]>

* change field

Signed-off-by: Ben Ye <[email protected]>

* update downsample

Signed-off-by: Ben Ye <[email protected]>

* add index stats into compaction

Signed-off-by: Ben Ye <[email protected]>

* fix tests

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* estimate block chunk and series size from metadata (#6449)

Signed-off-by: Ben Ye <[email protected]>

* Adding graphic for blog showing architecture

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing mermaid graph

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing mermaid graph again

Signed-off-by: Jonah Kowall <[email protected]>

* Remove image and fix mermaid for other flowchart

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing prod flowchart

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing prod flowchart title

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing md table

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing md table again

Signed-off-by: Jonah Kowall <[email protected]>

* Adding clarification on cost breakdowns

Signed-off-by: Jonah Kowall <[email protected]>

* Adding clarification on cost breakdowns

Signed-off-by: Jonah Kowall <[email protected]>

* Store: fix crash on empty regex matcher

Signed-off-by: Michael Hoffmann <[email protected]>

* Some corrections on text and links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing links

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing diagams

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing currencies

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing diagams

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing images

Signed-off-by: Jonah Kowall <[email protected]>

* Website: Fix blog images width (#6450)

* Set blog post images width to 100%

Blog Post images width isn't set and hence the images doesn't get
fit to the page. Setting width to 100% make sure images fit to
the web page and screen size.

Signed-off-by: Pradyumna Krishna <[email protected]>

* Remove width constraint from blog images

Signed-off-by: Pradyumna Krishna <[email protected]>

---------

Signed-off-by: Pradyumna Krishna <[email protected]>

* Cache calculated mint and maxt for each remote engine (#6458)

The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times
on each remote engine. These calculations can get expensive when an engine
covers a large amount of TSDBs.

This commit introduces caching of the calculated values for a remote
engine's mint, maxt and label sets.

Signed-off-by: Filip Petkovski <[email protected]>

* cache empty expanded postings (#6464)

Signed-off-by: Ben Ye <[email protected]>

* mixin/receive: add limits alerting (#6466)

* add some thanos receive limits alerts

Signed-off-by: Thibault Mange <[email protected]>

* update changelog

Signed-off-by: Thibault Mange <[email protected]>

* fix and generate examples

Signed-off-by: Thibault Mange <[email protected]>

* quick fix

Signed-off-by: Thibault Mange <[email protected]>

* quick fix

Signed-off-by: Thibault Mange <[email protected]>

* fix test

Signed-off-by: Thibault Mange <[email protected]>

* fix invalid changelog

Signed-off-by: Thibault Mange <[email protected]>

---------

Signed-off-by: Thibault Mange <[email protected]>

* Update docs/blog/2023-06-08-thanos-at-aiven.md

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Jonah Kowall <[email protected]>

* Add retention for M3

Signed-off-by: Jonah Kowall <[email protected]>

* Add image versus mermaid diagram. Couldn't comment it out unfortunately

Signed-off-by: Jonah Kowall <[email protected]>

* Adding text around link to fix linter

Signed-off-by: Jonah Kowall <[email protected]>

* make docs

Signed-off-by: Jonah Kowall <[email protected]>

* add alert for tenant reaching head series limit (#6467)

Signed-off-by: Thibault Mange <[email protected]>

* go.mod: bump rest of otel libs (#6447)

* go.mod: bump rest of otel libs

Bump these two OTEL libs to be like the rest of the OTEL libs. We had a
strange issue where `service.name` was not set even though it was in
environment variables. Bumping these libs fixes the problem.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* otlp: only set service.name if provided

Signed-off-by: Giedrius Statkevičius <[email protected]>

* e2e: add test case

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* store: disable pooling for postings benchmarks (#6473)

Pooling hides the real allocations in postings benchmarks because the
allocation only happens once really. In reality this is different when
multiple readers are created at once. Disable pooling in benchmarks to
understand what is happening.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fixing date and author for Aiven blog (#6476)

* Fixing authors

Signed-off-by: Jonah Kowall <[email protected]>

* make docs and fixing date

Signed-off-by: Jonah Kowall <[email protected]>

* Fixing authors

Signed-off-by: Jonah Kowall <[email protected]>

---------

Signed-off-by: Jonah Kowall <[email protected]>

* Check context when expanding postings 2nd attempt (#6471)

* check context when expanding postings

Signed-off-by: Ben Ye <[email protected]>

* import

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* [FEAT] Add thanos ruler blog post (#6477)

Signed-off-by: Hélia Barroso <[email protected]>

* blog: Small fixes to Aiven article (#6481)

* Correct some links and typos in 2023-06-08-thanos-at-aiven.md

The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links.
Also fixed a few other typos I spotted.

Signed-off-by: Hugh Cole-Baker <[email protected]>

* Update 2023-06-08-thanos-at-aiven.md

Clarify a couple of things in the final paragraph.

Signed-off-by: Hugh Cole-Baker <[email protected]>

---------

Signed-off-by: Hugh Cole-Baker <[email protected]>

* Rename series and postings fetch duration metrics (#6479)

* fix wrong metric names for series and postings fetch duration

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* note breaking change

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Update to prometheus/common v0.44.0 (#6483)

Signed-off-by: Dan Bason <[email protected]>

* go.mod: update rueidis client (#6485)

The latest improvements have great perf improvements. Let's update it.
Thank you to [rueian](https://github.com/rueian) and all of the
contributors to that awesome Redis client!

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Introduce tenancy package (#6482)

This commit introduces a tenancy package in preparation for introduction
of tenancy in the query path. Tenancy related code which can be common
between components are moved out of the receive component into the
tenancy package.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Ignore nginx.com links in mdox (#6489)

Signed-off-by: Saswata Mukherjee <[email protected]>

* blog: Update Aiven blog post (#6487)

Wording changes and better formatting for the tables.

Signed-off-by: Simon Brüggen <[email protected]>

* store: optimized snappy streamed reading (#6475)

Optimize snappy streamed reading by traversing through the byte slice in
advance to determine how big of a buffer we will need. I have opted to
rewrite snappy streamed format decoding because it is a straightforward
format. Complex parts were deferred to klauspost/compress.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Remove Exists call in meta fetcher (#6474)

* Remove Exists call in meta fetcher

The meta fetcher lists the bucket by iterating through
top level paths from the entire keyspace, and then for each key
calls Exists on the meta.json file. This leads to an N+1 amplification
against object storage, manifesting as a very high increase in costs
and occasional throttling by cloud providers.

This commit changes the iteration logic to recursively list all files instead,
which can be done with far fewer API calls to object storage. Meta files
are then identified from the keys returned by the list operation instead
of making one API request from each file.

Signed-off-by: Filip Petkovski <[email protected]>

* Revert e2e test change

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* e2e/store: use now instead of time.Now() each time (#6493)

Consistently use the same timestamp instead of the current time each
time during a test.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* [FIX] Documentation issue (#6494)

Signed-off-by: Hélia Barroso <[email protected]>

* Updates busybox SHA (#6497)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* chore: pkg imported more than once (#6499)

* chore: pkg imported more than once

Signed-off-by: guoguangwu <[email protected]>

* fix: remove pkg alias

Signed-off-by: guoguangwu <[email protected]>

---------

Signed-off-by: guoguangwu <[email protected]>

* *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496)

* Fix and improve PathContentReloader tests

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Add changelog entry

Signed-off-by: Douglas Camata <[email protected]>

* Fix table tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Co-authored-by: Filip Petkovski <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>
Co-authored-by: Filip Petkovski <[email protected]>

* add context check when decoding cached postings (#6506)

Signed-off-by: Ben Ye <[email protected]>

* *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503)

* Add fallback config reload for symlinks

Signed-off-by: Douglas Camata <[email protected]>

* Improve pooling config reload

Signed-off-by: Douglas Camata <[email protected]>

* Make tests more reliable

Signed-off-by: Douglas Camata <[email protected]>

* Improve tests again

Assert on runutil.Repeat output

Signed-off-by: Douglas Camata <[email protected]>

* Pass the debounce/reload time to each test

Signed-off-by: Douglas Camata <[email protected]>

* Add comments and ensure interfaces are implemented

Signed-off-by: Douglas Camata <[email protected]>

* Make the limit configuration reload time customizable

Signed-off-by: Douglas Camata <[email protected]>

* goimports file

Signed-off-by: Douglas Camata <[email protected]>

* Fix lint warning

Signed-off-by: Douglas Camata <[email protected]>

* Improve log for polling engine

Signed-off-by: Douglas Camata <[email protected]>

* Fix tests

Signed-off-by: Douglas Camata <[email protected]>

* Make linter happy

Signed-off-by: Douglas Camata <[email protected]>

* Extract symlink identification to a separate function

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix links to go-grpc-middleware after v2 merge into main

Signed-off-by: Douglas Camata <[email protected]>

* Add period to make linter happy.

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog entry

Signed-off-by: Douglas Camata <[email protected]>

* Replace the fsnotify based engine from `PathContentReloader` with the polling one

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Remove check for empty filePath

Signed-off-by: Douglas Camata <[email protected]>

* Unexpose pollingEngine.Start

Signed-off-by: Douglas Camata <[email protected]>

* Fix pollingEngine doc-comment

Signed-off-by: Douglas Camata <[email protected]>

* Log file path when config is reloaded

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* receive: add float histogram support (#6323)

* Added receive float histogram support

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Added comments for fns copied from Prometheus

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Improved comment

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unnecessary if

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed native histogram proto conversion in remote engine

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Removed unused histogram conversion from remote engine

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fix and renaming in native_histograms_test.go

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Trigger Build

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* go.mod: bump e2e framework version (#6516)

Bump e2e framework version and add a note about E2E_DOCKER_CPUS.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Add changelog for #6474 (#6518)

Signed-off-by: Filip Petkovski <[email protected]>

* cortex/redisclient: use rueidis client (#6520)

* cortex/redisclient: use rueidis client

Use the same rueidis client in query-frontend. Solves
https://github.com/thanos-io/thanos/issues/6094.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cortex/cache: gofumpt + fix errors.Errorf

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cacheutil/docs: clean up more old stuff

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* index header: Remove memWriter from fileWriter (#6509)

* index header: remove memWriter from fileWriter

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* refactor

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

* update comment

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Wrap object store Prometheus registry (#6152)

* Wrap object store Prometheus registery

In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26.

Signed-off-by: Douglas Camata <[email protected]>

* Update objstore to the latest version

This version removes the `thanos_` prefix from metrics, which is the
reason for wrapping the objstore's metrics registry in the first place.

Signed-off-by: Douglas Camata <[email protected]>

* Fix modules

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Wrap metrics registerer for objstore bucket

Signed-off-by: Douglas Camata <[email protected]>

* Remove prefix from Thanos Store metrics

Signed-off-by: Douglas Camata <[email protected]>

* Fix goimports

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Put back upgraded objstore dep

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Rerun CI

Signed-off-by: Douglas Camata <[email protected]>

* Move to more recent ref of thanos/objstore

Signed-off-by: Douglas Camata <[email protected]>

* Ignore OBS in objstore tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix linting error on main test after objstore upgrade

Signed-off-by: Douglas Camata <[email protected]>

* Skip OCS objstore test in circle ci

Signed-off-by: Douglas Camata <[email protected]>

* Fix echo in makefile

Signed-off-by: Douglas Camata <[email protected]>

* Fix typo

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* e2e/store: try to fix Series() limit test again (#6522)

I finally managed to reproduce this failure locally with
http…
SuperQ added a commit to SuperQ/thanos that referenced this issue Apr 16, 2024
@SuperQ
Compact: Replace group with resolution in ownsample metrics
c1872c7 
Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
@SuperQ SuperQ mentioned this issue Apr 16, 2024
Merged
2 tasks
SuperQ added a commit to SuperQ/thanos that referenced this issue Apr 16, 2024
@SuperQ
Compact: Replace group with resolution in ownsample metrics
854c095 
Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
SuperQ added a commit to SuperQ/thanos that referenced this issue Apr 16, 2024
@SuperQ
Compact: Replace group with resolution in ownsample metrics
408a07b 
Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
GiedriusS pushed a commit that referenced this issue Apr 18, 2024
@SuperQ
Compact: Replace group with resolution in ownsample metrics (#7283)
f7ba140 
Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: #5841

Signed-off-by: SuperQ <[email protected]>
jnyi pushed a commit to jnyi/thanos that referenced this issue Jun 1, 2024
@SuperQ @jnyi
Compact: Replace group with resolution in ownsample metrics (thanos-i…
7e62627 
…o#7283)

Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
jnyi pushed a commit to jnyi/thanos that referenced this issue Jun 4, 2024
@SuperQ @jnyi
Compact: Replace group with resolution in ownsample metrics (thanos-i…
1f66f43 
…o#7283)

Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: thanos-io#5841

Signed-off-by: SuperQ <[email protected]>
coleenquadros added a commit to stolostron/thanos that referenced this issue Jul 24, 2024
@acm-observability-bot @MichaHoffmann
@GiedriusS @yeya24 @MeenuyD @saswatamcode @matej-g @bwplotka @rikhil-s @martinhaus @fpetkovski @simonpasquier @mercxry @wenxu1024 @kartikaysaxena @github-actions @sagnik3788 @danielblando @JHeilCoveo @lpreethvika @rabenhorst @Vanshikav123 @pawarpranav83 @xBazilio @tasrieit @jacobbaungard @Pratham1812 @pedro-stanaka @alecrajeev @harsh-ps-2003 @Player256 @nicolastakashi @alexqyle @douglascamata @hanyuting8 @cincinnat @jakekeeys @bavarianbidi @chetanpdeshmukh @sdufel @damnever @munir131 @danielmellado @payalraviya @TheSpiritXIII @heliapb @thibaultmg @outofrange @roth-wine @kakkoyun @suhas-chikkanna @suhas-chikkanna-shield @NeerajNagure @jnyi @tizki @NotAFile @yj-yoo @magiceses @SuperQ @guillaumelecerf @coleenquadros
[ACM Obs bot] Bump stolostron/thanos to v0.35.1 (#128)
e0165a7 
* CI: increase e2e test parallelism (#6875)

Signed-off-by: Michael Hoffmann <[email protected]>

* compact: hook nodownsamplemarkfilter into filters chain (#6893)

We have a NoDownsampleMarkFilter that we were not using before in the
compactor for some reason. Hook it into the filters chain if
downsampling is enabled and then trim matching ULIDs from the
downsampling process. Add a test to cover this scenario.

Fixes https://github.com/thanos-io/thanos/issues/6179.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* e2e/compact: fix assertion values (#6909)

I think the original values were simply too small there - if the runner
is slow then it can see smaller metrics values. If the runner works well
then the values can go to these bigger values and then the test fails.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* exposes asyncOperationProcessor (#6908)

expose struct



fix lint

Signed-off-by: Ben Ye <[email protected]>

* Removed the gitpod from the contribution.md (#6907)

Signed-off-by: MeenuyD <[email protected]>

* .circleci: cache tool deps (#6910)

I have noticed that downloading tool deps takes ~5 min in CI each time.
We should be able to also cache those tool deps.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* MAINTAINERS.md: add myself as maintainer (#6911)

Signed-off-by: Michael Hoffmann <[email protected]>

* blog: Introducing ThanosCon in Paris KubeCon EU 2024 (#6912)

* blog: Introducing ThanosCon in Paris KubeCon EU 2024

Signed-off-by: Saswata Mukherjee <[email protected]>

* Add image

Signed-off-by: Saswata Mukherjee <[email protected]>

* make docs

Signed-off-by: Saswata Mukherjee <[email protected]>

* Apply suggestions

Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>

* Apply suggestions

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>

* Add link to slack channel

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>

* Update 2023-20-11-thanoscon.md (typo) (#6915)

Signed-off-by: Bartlomiej Plotka <[email protected]>

* Added info about ThanosCon to website and Readme. (#6917)

* Added info about ThanosCon to website and Readme.

Signed-off-by: bwplotka <[email protected]>

* Update website/layouts/index.html

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Bartlomiej Plotka <[email protected]>

---------

Signed-off-by: bwplotka <[email protected]>
Signed-off-by: Bartlomiej Plotka <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* Objstore: Bump Objstore for Azure Workload Identity support (#6891)

* Bump objstore for Azure Workload Identity support

Signed-off-by: Rikhil Shah <[email protected]>

* make check-docs

Signed-off-by: Rikhil Shah <[email protected]>

* Add changelog entry

Signed-off-by: Rikhil Shah <[email protected]>

* Update Azure client docs

Signed-off-by: Rikhil Shah <[email protected]>

* make check-docs

Signed-off-by: Rikhil Shah <[email protected]>

* Move changelog entry to 0.33.0 release

Signed-off-by: Rikhil Shah <[email protected]>

* Move changelog entry

Signed-off-by: Rikhil Shah <[email protected]>

---------

Signed-off-by: Rikhil Shah <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>

* update compactor backlog doc for checking halt (#6906)

Signed-off-by: Ben Ye <[email protected]>

* Update LabyrintLabs logo

Signed-off-by: Martin Hauskrecht <[email protected]>

* Support reload using signal (#6453)

* Support reload using signal

Signed-off-by: Simon Pasquier <[email protected]>

* Add --reloader.method option to sidecar

This option allows to tell the sidecar to send a SIGHUP signal to the
monitored process to reload its configuration instead of the default
HTTP-based method.

Signed-off-by: Simon Pasquier <[email protected]>

* Update docs and CHANGELOG.md

Signed-off-by: Simon Pasquier <[email protected]>

---------

Signed-off-by: Simon Pasquier <[email protected]>

* Support float histogram in store gateway (#6925)

* support float histogram in store gateway

Signed-off-by: Ben Ye <[email protected]>

* fix lint

Signed-off-by: Ben Ye <[email protected]>

* fix e2e test

Signed-off-by: Ben Ye <[email protected]>

* update prompb

Signed-off-by: Ben Ye <[email protected]>

* add changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* chore: add truelayer to the adopters (#6933)

Signed-off-by: Matteo Martellini <[email protected]>

* Fetcher: Add a BlockIDsFetcher Interface to BaseFetcher (#6902)

* add BlockIDsFetcher to BaseFetcher

Signed-off-by: Wen Xu <[email protected]>

* fix lint

Signed-off-by: Wen Xu <[email protected]>

* use chan in the interface method to accept active block ids

Signed-off-by: Wen Xu <[email protected]>

* fix comments

Signed-off-by: Wen Xu <[email protected]>

* fix lint

Signed-off-by: Wen Xu <[email protected]>

* add description of active and parital blocks and modify changelog

Signed-off-by: Wen Xu <[email protected]>

* fix interface description

Signed-off-by: Wen Xu <[email protected]>

* remove entry in changelog

Signed-off-by: Wen Xu <[email protected]>

---------

Signed-off-by: Wen Xu <[email protected]>

* Update hugo file (#6927)

* Update hugo file #6889

Signed-off-by: Kartikay <[email protected]>

* Version 0.33 removed

Signed-off-by: Kartikay <[email protected]>

---------

Signed-off-by: Kartikay <[email protected]>

* Updates busybox SHA (#6897) (#6937)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <[email protected]>

* Fix hotstar logo (#6938)

* Fix hotstar logo

Signed-off-by: Kartikay <[email protected]>

* Delete website/.hugo_build.lock

Signed-off-by: Kartikay <[email protected]>

---------

Signed-off-by: Kartikay <[email protected]>
Signed-off-by: Kartikay <[email protected]>

* errutil: fix deadlock

Fix deadlock in the Add() function - it accidentally was calling itself
recursively. This struct wraps a `MultiError` so we need to call `Add()`
on that `MultiError` inside.

This deadlock can manifest in Receive - million+ leaking goroutines, all
stuck on the MultiTSDB lock.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* update runtime/CloseWithErrCapture comment (#6952)

* update runtime/CloseWithErrCapture comment

Signed-off-by: Sagnik Das <[email protected]>

* fix lint

Signed-off-by: Sagnik Das <[email protected]>

---------

Signed-off-by: Sagnik Das <[email protected]>

* Added a new flag for maximum retention bytes for thanos (#6944)

* changes

Signed-off-by: Kartikay <[email protected]>

* Changed Int64 to BytesVar

Signed-off-by: Kartikay <[email protected]>

* Updated Changelog for new flag in receive

Signed-off-by: Kartikay <[email protected]>

---------

Signed-off-by: Kartikay <[email protected]>
Signed-off-by: Kartikay <[email protected]>

* fix store gateway span name (#6953)

Signed-off-by: Ben Ye <[email protected]>

* Upgrade prometheus to latest main (#6950)

* upgrade prometheus to latest main

Signed-off-by: Ben Ye <[email protected]>

* upgrade again

Signed-off-by: Ben Ye <[email protected]>

* fix test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Improve filtered index cache filtering (#6955)

* improve filtered index cache filtering

Signed-off-by: Ben Ye <[email protected]>

* make inline

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* store: fix misc label_value issues related to external labels (#6879)

Signed-off-by: Michael Hoffmann <[email protected]>

* receive: fix adding fields to logger

Fix adding fields to the logger because otherwise it looks like
"[tenant foo]: (UNDEFINED)".

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Copy labels coming from remote engines (#6957)

* Copy labels coming from remote engines

When running in distributed mode, the remote engine will use an unsafe
cast from ZLabels to Prometheus labels to avoid making new allocations.
This makes it hard to use the new gRPC shared buffer pool for receiving
and decompressing messages since memory gets retained beyond the scope
of a Recv() call.

This commit removes the unsafe cast and makes an explicit memory copy
of received series labels. Since remote queries are already aggregated
series, the amount of data we receive should be small anyway, and the copies
on average should have a small impact.

Signed-off-by: Filip Petkovski <[email protected]>

* Use clone on strings

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* Add tracing index cache (#6954)

* add tracing index cache

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Change order of log for finished compacting blocks (#6966)

Signed-off-by: Daniel Deluiggi <[email protected]>

* Update MAINTAINERS.md (#6967)

Signed-off-by: Matej Gera <[email protected]>

* Add `keep_firing_for` support to alerting rule (#6943)

* Add keep_firing_for to alert rules

Signed-off-by: JHeil <[email protected]>

* Updating changelog

Signed-off-by: JHeil <[email protected]>

* Re-order keepFiringFor at the end of the struct

Signed-off-by: JHeil <[email protected]>

---------

Signed-off-by: JHeil <[email protected]>
Signed-off-by: JHeil <[email protected]>

* Use registerer for gRPC opts and endpoints (#6971)

The EndpointSet and StoreGRPC options accept concrete *prometheus.Registry type.
This makes it hard to create multiple instances of them because they will try
to register the same metrics and the registry will panic.

This commit changes the type to a prometheus.Registerer which allows clients
to inject a prefixed registry and get metrics for each instance.

Signed-off-by: Filip Petkovski <[email protected]>

* receive: upload compacted blocks if OOO enabled (#6974)

* Add streaming series limit at block series client (#6972)

* add series limit that is applied when streaming using block series client

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

* add unit tests

Signed-off-by: Ben Ye <[email protected]>

* address comments

Signed-off-by: Ben Ye <[email protected]>

* fix comment

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Updated the defaultChecked property in checkbox element (#6976)

Signed-off-by: lpreethvika-samsung <[email protected]>

* Update prometheus to d0c2d9c (#6978)

* Update prometheus to d0c2d9c

This commit updates Prometheus to commit d0c2d9c which contains
the loser-tree based postings merge. Note that we cannot update to
latest main since there is currently a bug in PromQL when querying
native histograms introduced by https://github.com/prometheus/prometheus/pull/13276.
This issue should be fixed by https://github.com/prometheus/prometheus/pull/13289.

This commit also updates the Thanos PromQL engine to latest main.

Signed-off-by: Filip Petkovski <[email protected]>

* Fix handler_test.go

Signed-off-by: Filip Petkovski <[email protected]>

* Fix manager_test.go

Signed-off-by: Filip Petkovski <[email protected]>

* Use empty registry for file discovery

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* Updates busybox SHA (#6982)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* Updates busybox SHA (#6982)

Signed-off-by: GitHub <[email protected]>
Co-authored-by: fpetkovski <[email protected]>

* VERSION: Bump version for release

Signed-off-by: Michael Hoffmann <[email protected]>

* rule: native histogram support (#6390)

* Added native histogram support for ruler

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Formatted imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Formated imports

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed native histogram tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

Fixed receiver type

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fix for rebase

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Added docs for query endpoints differences

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed comments and naming

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* made HTTPConfig optional

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* made HTTPConfig optional

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Reverted and added check

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixes from comments

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* renamed queryconfig to clientconfig

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* common prepareEndpointSet

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* fixed lint

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed sidecar

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* Fixed tests

Signed-off-by: Sebastian Rabenhorst <[email protected]>

---------

Signed-off-by: Sebastian Rabenhorst <[email protected]>

* mdox ignore checking twitter urls (#7001)

Signed-off-by: Ben Ye <[email protected]>

* Updated Grofers logo (#7006)

Signed-off-by: Kartikay <[email protected]>

* TraceID : Fetching TraceID (#6973)

* docs: add promcon 2023 thanos talks

Signed-off-by: Michael Hoffmann <[email protected]>

* Added website page for companies who offer consultancy and enterprise… (#7000)

* Added website page for companies who offer consultancy and enterprise support for Thanos

Signed-off-by: Kartikay <[email protected]>

* adopters.yml revert

Signed-off-by: Kartikay <[email protected]>

* retrigger checks

Signed-off-by: Kartikay <[email protected]>

* added a new line in welcome.md

Signed-off-by: Kartikay <[email protected]>

* retrigger checks

Signed-off-by: Kartikay <[email protected]>

---------

Signed-off-by: Kartikay <[email protected]>

* Lazy downloaded index header (#6984)

* lazy downloaded index header

Signed-off-by: Ben Ye <[email protected]>

* update tests

Signed-off-by: Ben Ye <[email protected]>

* address comments

Signed-off-by: Ben Ye <[email protected]>

* address comments

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* tests: use remote write in query frontend tests (#6998)

* query-frontend: Added support of auto_discovery for memcached (#7004)

* query-frontend: Added support of auto_discovery for memcached

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* adjustments to build on main branch

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* CHANGELOG.md

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* typo fixed

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* minor fixex after review

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

---------

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* Fix URI encoding of strings

Signed-off-by: Kartikay <[email protected]>

* ui: enable partial response strategy by default

Rebuild

Signed-off-by: Vanshika <[email protected]>

* Added negative offset check for caching queries (#7011)

Signed-off-by: pawarpranav83 <[email protected]>
Co-authored-by: pawarpranav83 <[email protected]>

* Update MAINTAINERS.md (Bartek's company)

Should be done in January last year...

Signed-off-by: Bartlomiej Plotka <[email protected]>

* store: add chunksize tests to acceptance tests

* add chunk size tests to acceptance tests
* refactor acceptance tests slightly

Signed-off-by: Michael Hoffmann <[email protected]>

* added tasrie it services as support partner (#7023)

* added tasrie it services as support partner

Signed-off-by: Tasrie IT Services <[email protected]>

* added the link in the list.html page

Signed-off-by: Tasrie IT Services <[email protected]>

* fixed the logo resolution to 190x70

Signed-off-by: Tasrie IT Services <[email protected]>

* Remove extraneous text

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Tasrie IT Services <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* Query Frontend: Add tenant label to metrics (#6887)

This commit adds a tenant label to the HTTP metrics which are exported
by the Query Frontend.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* tests: use remote write in query frontend tests (#7017)

* resolved issue #7024

Signed-off-by: Pratham Agarwal <[email protected]>

* QueryFrontend|Query: Create new arg to enable extended functions (#7028)

* Adding new parameter for extended functions in querier

Signed-off-by: Pedro Tanaka <[email protected]>

* Adding new flag for QFE

Signed-off-by: Pedro Tanaka <[email protected]>

* improve argument passing in query side

Signed-off-by: Pedro Tanaka <[email protected]>

* Adding changelog

Signed-off-by: Pedro Tanaka <[email protected]>

* Adding e2e test for query

Signed-off-by: Pedro Tanaka <[email protected]>

* undoing uneeded changes

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing docs

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing e2e tests

Signed-off-by: Pedro Tanaka <[email protected]>

* Fixing backward compat test

Signed-off-by: Pedro Tanaka <[email protected]>

* changes from CR

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
Signed-off-by: Pedro Tanaka <[email protected]>

* receive: respect forward timeout in http handler buckets (#7030)

Forward timeout is what ultimately decides how long a HTTP request might
take so it doesn't make sense to hard-code max value of 5. In this pull
request I propose respecting the configured forward timeout and adding
extra buckets if the default buckets don't cover it completely.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive: do not leak grpc connections (#7031)

* receive: do not leak grpc connections

Prevent a leak in gRPC connections by garbage collecting old ones when
the hashring changes. For that purpose, I propose adding a `Nodes()
string` method so that it would be possible to know what nodes do not
exist in the hashring anymore.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive: change order of operations

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* fix multi-tenancy link (#7032)

Signed-off-by: Alec Rajeev <[email protected]>

* removing todo comments from query docs

Signed-off-by: Harsh Pratap Singh <[email protected]>

* Query: add optional tenancy enforcement (#6756)

* Query: add optional tenancy enforcement

With this commit it's now possible to enable enforcement of tenancy. If
tenancy is enabled, a tenant label will be added to queries based on the
tenant information provided by the tenant header, and the
tenant-label-name.

The implementation for query APIs are done by using prom-label-proxy as
library, while the implementation for non-query APIs are written from
scratch.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Add changelog entry

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Query: Add non-default tenant testcase

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Test: make query a constant to make linter happy

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Address review comments

- Remove empty lines
- If multiple tenant matchers are found in the original query, we only
  replace the first one with the header provided tenant, and remove any
  subsequent ones.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Address review comments

- `--enable-tenancy` -> `--enforce-tenancy`
- Create `RewritePromQL` and `RewriteLabelMatchers` to clean up code in
  query api. Also move getLabelMatchers to tenancy pkg.
- Use prom-label-proxys `EnforceMatchers` to rewrite labels on non-query
  APIs instead of own solution
- Don't specifically handle `illegalLabelMatcherError`

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Re-arrage go.mod to make linter happy.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Address review comments

Minor changes to CLI docs, code-comments and changelog.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Query: Add tenancy docs

This commit adds documentation for the tenancy features.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Update docs/components/query.md

Review comment

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Jacob Baungård Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>
Signed-off-by: Jacob Baungård Hansen <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* Tests: unique docker env name for tenant tests (#7038)

The e2e tests would occasionally fail due to non-unqiue docker environment
names. With this commit the tests are environments are given unique names
to avoid these failures.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* changed default 3m to 15m

Signed-off-by: Player256 <[email protected]>

* Fix docs (#7039)

Fix docs post #6539 merge.

Signed-off-by: Filip Petkovski <[email protected]>

* Revert "receive: upload compacted blocks if OOO enabled (#6974)" (#7053)

This reverts commit 7b8eb86c0ff3b1144aa1cea392806afa40a4cdf8.

Proper way to handle this is to disable vertical compaction. I am trying
to add this functionality here:
https://github.com/prometheus/prometheus/pull/13393

Signed-off-by: Giedrius Statkevičius <[email protected]>

* [CHORE] adding thanos upload-snapshot command (#6884)

Signed-off-by: Nicolas Takashi <[email protected]>

* fix docs error

Signed-off-by: Kartikay <[email protected]>

* cmd: make bucket upload command take lset from flags (#7059)

Signed-off-by: Michael Hoffmann <[email protected]>

* CHANGELOG: mark 0.34 as in progress

Signed-off-by: Michael Hoffmann <[email protected]>

* VERSION: cut release 0.34.0-rc.0

Signed-off-by: Michael Hoffmann <[email protected]>

* UI: Don't always force tracing (#7062)

Forced tracing was.. Forced true always, even if the checkbox in the UI
to enable tracing was not actually checked.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* go.mod: update Prometheus version (#7047)

Update Prometheus version to include
https://github.com/prometheus/prometheus/pull/13242 which is important
for me - it unblocks further postings work.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Make RetryError and HaltError able to be fetched for root cause (#7043)

* Make RetryError and HaltError able to be fetched for root cause

Signed-off-by: Alex Le <[email protected]>

* Added unit test

Signed-off-by: Alex Le <[email protected]>

* fix lint

Signed-off-by: Alex Le <[email protected]>

* fixed IsRetryError and IsHaltError functions

Signed-off-by: Alex Le <[email protected]>

---------

Signed-off-by: Alex Le <[email protected]>

* receive: disable overlapping compaction

Use the new TSDB flag to disable overlapping compaction to fix OOO
samples handling in the Receive component.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CI: Ensure static react-app is checked in (#7063)

* CI: Ensure static react-app is checked in

With this commit the CI system should fail if changes to the react-app
has been made without checking in the changes.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Add `react-app` as dependency `check-react-app`

To ensure the react-app is rebuilt before checking for changes.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Receive: refactor handler for improved readability and organization (#6898)

* [wip] First checkpoint

Signed-off-by: Douglas Camata <[email protected]>

* [wip] Second checkpoint

All tests passing, unit and e2e.

Signed-off-by: Douglas Camata <[email protected]>

* Small random refactors

Signed-off-by: Douglas Camata <[email protected]>

* Add some useful trace tags

Signed-off-by: Douglas Camata <[email protected]>

* Concurrent and traced local writes

Signed-off-by: Douglas Camata <[email protected]>

* Improve variable names in remote writes

Signed-off-by: Douglas Camata <[email protected]>

* Rename `newFanoutForward` function

Signed-off-by: Douglas Camata <[email protected]>

* More refactors

Signed-off-by: Douglas Camata <[email protected]>

* Fix linting issue

Signed-off-by: Douglas Camata <[email protected]>

* Add a quorum test with sloppy quorum

Signed-off-by: Douglas Camata <[email protected]>

* [wip] Try to make retries work

Signed-off-by: Douglas Camata <[email protected]>

* [wip] Checkpoint: wait group still hanging

Signed-off-by: Douglas Camata <[email protected]>

* Some refactors

Signed-off-by: Douglas Camata <[email protected]>

* Add some commented code so I don't lose it

Signed-off-by: Douglas Camata <[email protected]>

* Adapt tests

Signed-off-by: Douglas Camata <[email protected]>

* Remove sloppy quorum code

Signed-off-by: Douglas Camata <[email protected]>

* Move some code around

Signed-off-by: Douglas Camata <[email protected]>

* Remove even more leftover of sloppy quorum

Signed-off-by: Douglas Camata <[email protected]>

* Extract a type to hold function params

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused struct field

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless variable

Signed-off-by: Douglas Camata <[email protected]>

* Remove type that wasn't used enough

Signed-off-by: Douglas Camata <[email protected]>

* Delete function to tighten up max buffered responses

Signed-off-by: Douglas Camata <[email protected]>

* Add comments to some functions

Signed-off-by: Douglas Camata <[email protected]>

* Fix peer up check

Signed-off-by: Douglas Camata <[email protected]>

* Fix size of replication tracking slices

Signed-off-by: Douglas Camata <[email protected]>

* Rename context

Signed-off-by: Douglas Camata <[email protected]>

* Don't do local writes concurrently

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra error logging

Signed-off-by: Douglas Camata <[email protected]>

* Fix syntax after merge

Signed-off-by: Douglas Camata <[email protected]>

* Add missing methods to peersContainer

Signed-off-by: Douglas Camata <[email protected]>

* Fix handler test

Signed-off-by: Douglas Camata <[email protected]>

* Reset peers state on hashring changes

Signed-off-by: Douglas Camata <[email protected]>

* Handle PR comment regarding waitgroup

Signed-off-by: Douglas Camata <[email protected]>

* Set span tags to help debug

Signed-off-by: Douglas Camata <[email protected]>

* Fix concurrency issue

We close the request as soon as quorum is reached and leave a few Go routines running to finish replication and so cleanups.

This means that the context from the HTTP request is cancelled... which ends up also cancelling the pending replication requests.

Signed-off-by: Douglas Camata <[email protected]>

* Fix request ID middleware

Signed-off-by: Douglas Camata <[email protected]>

* Fix `distributeTimeseriesToReplicas` comment

Signed-off-by: Douglas Camata <[email protected]>

* Extract var with 1-indexed replication index

Signed-off-by: Douglas Camata <[email protected]>

* Rename methods in peersContainer interface

Signed-off-by: Douglas Camata <[email protected]>

* Make peerGroup `getConnection` check if peers are up

Signed-off-by: Douglas Camata <[email protected]>

* Remove yet one more not useful log

Signed-off-by: Douglas Camata <[email protected]>

* Remove logger from `h.sendWrites`

Signed-off-by: Douglas Camata <[email protected]>

---------

Signed-off-by: Douglas Camata <[email protected]>

* Upgrade grpc to 1.57.2 (#7078)

1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2
2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail

Signed-off-by: hanyuting8 <[email protected]>

* Store: acceptance test for proxy store (#7084)

* Add basic acceptance tests for proxy store
* Fix bug where invalid requests got ignored because of partial response
  strategy

Signed-off-by: Michael Hoffmann <[email protected]>

* Fix lazy postings with zero length (#7083)

* fix lazy postings with zero length

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

* unit tests

Signed-off-by: Ben Ye <[email protected]>

* fix doc

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Store: fix label values edge case (#7082)

If the requested label is an external label and we have series matchers
we should only return results if the series matchers actually match a
series.

Signed-off-by: Michael Hoffmann <[email protected]>

* VERSION: cut release 0.34.0-rc.1

Signed-off-by: Michael Hoffmann <[email protected]>

* receive: race condition in handler Close() when stopped early (#7087)

Receiver hangs waiting for the HTTP Hander to shutdown if an error occurs
before Handler is initialized. This might happen, for example, if the hashring
is too small for a given replication factor.

Signed-off-by: Mikhail Nozdrachev <[email protected]>

* receive: use async remote writing (#7045)

* CHANGELOG: cut release 0.34 (#7095)

Signed-off-by: Michael Hoffmann <[email protected]>

* Stores: convert tests to not rely on slice labels (#7098)

Signed-off-by: Michael Hoffmann <[email protected]>

* all: get rid of query pushdown to simplify query path (#7014)

Signed-off-by: Michael Hoffmann <[email protected]>

* Store: dont rely on slice labels continued

Signed-off-by: Michael Hoffmann <[email protected]>

* Update prometheus/prometheus (#7096)

* Update prometheus/prometheus

This commit updates prometheus/prometheus to latest main (60b6266e).

Signed-off-by: Filip Petkovski <[email protected]>

* Fix file discovery

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* receive/handler: fix locking twice (#7112)

Fix bug introduced in https://github.com/thanos-io/thanos/pull/6898: we
were RLock()ing twice. This leads to a deadlock in some situations.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* fix minio store gateway err (#7114)

Signed-off-by: Kartikay <[email protected]>

* receive/handler: do not double lock (#7124)

markPeerUnavailable was always taking a lock and in one case we were
calling it with a lock already taken. Fix this.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* default to alertmanager v2 api (#7123)

Signed-off-by: Jake Keeys <[email protected]>

* Receive: dont rely on slice labels (#7100)

Signed-off-by: Michael Hoffmann <[email protected]>

* Upgrade grpc to 1.57.2 (#7078)

1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2
2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail

Signed-off-by: hanyuting8 <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>

* VERSION: cut release 0.34.1

Signed-off-by: Michael Hoffmann <[email protected]>

* docs: update helm installation instruction

the prometheus helm chart is a community maintained chart since a few
years. With that, the old example pointed to an old chart and the
provided example values aren't also working anymore.

This update the documentation.

Signed-off-by: Mario Constanti <[email protected]>

* docs: fix link (#7129)

The link has moved to another since Cisco bought Banzai Cloud.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* docs: run make docs for helm installation instruction

Signed-off-by: Mario Constanti <[email protected]>

* Fixing log line for remote engine in debug mode (#7133)

Signed-off-by: Pedro Tanaka <[email protected]>

* Adding new method on BucketedBytes to expose used memory (#7137)

* Adding new method on bucketed bytes to expose used

Signed-off-by: Pedro Tanaka <[email protected]>

* Removing interface, using RWMutex

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>

* Adding InfraCloud as Enterprise support partner (#7141)

* adding InfraCloud as Enterprise support partner

Signed-off-by: Chetan Deshmukh <[email protected]>

* replaced svg file to match layout

Signed-off-by: Chetan Deshmukh <[email protected]>

* added alt-text and horizontal image

Signed-off-by: Chetan Deshmukh <[email protected]>

---------

Signed-off-by: Chetan Deshmukh <[email protected]>

* docs: fix link (#7129)

The link has moved to another since Cisco bought Banzai Cloud.

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>

* cache: attach object storage hash to iter key (#6880)

Attach object storage hash to the iter key so that it would be possible
to reuse the same cache storage e.g. Redis for different buckets.
Without this, the results are funny to say the least if you accidentally
attempt to do that. Thus, let's add the hash to reduce the possibility
of an accident for our users.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Add support for extended promql functions in rule (#7105)

Adds a flag to register the extended promql functions supported by the thanos
query engine when running the rule component.  This will allow rule config
files containing query expressions with (xrate / xincrease / xdelta) to pass
validation.  This will only work if the query endpoint in use is running the
thanos engine.

Signed-off-by: Samuel Dufel <[email protected]>

* Extended func support - doc update (#7161)

* Add support for extended promql functions in rule

Adds a flag to register the extended promql functions supported by the thanos
query engine when running the rule component.  This will allow rule config
files containing query expressions with (xrate / xincrease / xdelta) to pass
validation.  This will only work if the query endpoint in use is running the
thanos engine.

Signed-off-by: Samuel Dufel <[email protected]>

* Update rendered docs with added flag

Signed-off-by: Samuel Dufel <[email protected]>

---------

Signed-off-by: Samuel Dufel <[email protected]>

* Copy labels from remote instant queries (#7151)

Similar to https://github.com/thanos-io/thanos/pull/6957, we should copy
labels from remote instant queries so that memory does not get overwritten
when processing series in a central engine.

Signed-off-by: Filip Petkovski <[email protected]>

* e2e/query_frontend: add tests for explain/analyze (#7160)

Adding tests for explain/analyze with QFE. Will add fixes as separate
PR.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cache/caching_bucket: add path to hash (#7158)

Add path to the hash. This allows identifying difference instances by
different config paths.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Allow using different listing strategies (#7134)

* Allow using different listing strategies

Signed-off-by: Filip Petkovski <[email protected]>

* Expose flags for block list strategy

Signed-off-by: Filip Petkovski <[email protected]>

* Run make docs

Signed-off-by: Filip Petkovski <[email protected]>

* Fix whitespace

Signed-off-by: Filip Petkovski <[email protected]>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* cache: implement the circuit breaker pattern for asynchronous set operations in the cache client (#7010)

* Implement the circuit breaker pattern for asynchronous set operations in the cache client

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Add feature flag for circuitbreaker

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Sync docs

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Skip configuration validation if the circuit breaker is disabled

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Make lint happy

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* Abstract the logic of the circuit breaker

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

---------

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>

* queryfrontend: fix analysis after API changes

Fix the analysis functionality with query-frontend after the recent
changes. Added tests for this.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* bugfix: lazy posting optimization with wrong cardinality for estimation (#7122)

* bugfix: catch lazy posting optimization using wrong cardinality for estimation

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* *: properly treat native histogram deduplication in chunk series merger

We have detected a problem in the chunk seriers merger where it will
panic in case it encounters native histogram chunks.
I am using thanos as a library for a project and wanted to use the
penalty function to dedup blocks from Prometheus instances.

Signed-off-by: Pedro Tanaka <[email protected]>

* Query UI: Add tenant box (#6867)

* Query UI: Add tenant box

With this commit as tenant box is added to the query UI. It can be used
to specify which tenant to use when making a query.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Re-compile static react app

Recompiles the static react app as now needed following:
https://github.com/thanos-io/thanos/pull/6900

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Move changelog item to appropiate future release

After merging it was under the 0.34 release.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Move query path tenancy proposal to done

Signed-off-by: Jacob Baungard Hansen <[email protected]>

---------

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* receive: add support for globbing tenant specifiers (#7155)

We want to be able to route all tenants which begin with certain letters
to some receivers so we need to have some kind of globbing/regex support
in the hashring. This PR adds that functionality. We've been using this
in prod successfully.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* .circleci: bump setup_docker_version version (#7177)

The current image is deprecated. See
https://discuss.circleci.com/t/remote-docker-image-deprecations-and-eol-for-2024/50176.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Unhide distributed execution mode

This commit exposes the distributed query execution mode to end-users by unhiding the
flag used to toggle this feature.

The commit also adds documentation on when the mode is appropriate to be used.

Signed-off-by: Filip Petkovski <[email protected]>

* Add changelog entry

Signed-off-by: Filip Petkovski <[email protected]>

* cortex/querier: fix analysis merging (#7179)

We were not merging analysis properly - mergo was overwriting data.
Instead of using a whole library for this, just write two small
functions and use them. Add test to cover this.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix docs

Fixes docs formatting and updates the distributed execution link to the done proposal.

Signed-off-by: Filip Petkovski <[email protected]>

* Added platformengineers

Signed-off-by: Munir Khakhi <[email protected]>

* Added platformengineers (#7181)

Signed-off-by: Munir Khakhi <[email protected]>

* Bump google.golang.org/protobuf to v1.33.0 (#7191)

This PR bumps the version of google.golang.org/protobuf to v1.33.0 fix a
potential vulnerability in the protojson.Unmarhsl function [1] that can
occure when unmarshaling a message with a protobuf value.

Even if the function isn't used directly in Thanos it would be safer to
just bump it directly.

[1] https://pkg.go.dev/vuln/GO-2024-2611

Signed-off-by: Daniel Mellado <[email protected]>

* fix: make responsive support page and change size of platform-engineer logo

fix: add anchor tag to all images
Signed-off-by: Payal17122000 <[email protected]>

* downsample: retry objstore related errors

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* downsample: retry objstore related errors

Signed-off-by: Vasiliy Rumyantsev <[email protected]>

* rule: do not turn off if resolving fails (#7192)

Do not turn off Ruler if resolving fails. We can still (try to) evaluate
rules even if Alertmanager is not available.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Query UI: Only show tenant box with enforcement on (#7186)

With this commit we only show the tenant-ui box when enforcement of
tenancy is on, as it is not needed otherwise.

Signed-off-by: Jacob Baungard Hansen <[email protected]>

* Reloader: Add support for watching and decompressing Prometheus configuration directories (#7199)

Signed-off-by: Daniel Hrabovcak <[email protected]>

* [FEAT] Adding blog post (#7202)

Signed-off-by: Helia Barroso <[email protected]>
Co-authored-by: Helia Barroso <[email protected]>

* storepb: make ServerAsClient channels unbuffered

Signed-off-by: Michael Hoffmann <[email protected]>

* Add support for TSDB selector in querier (#7200)

* Add support for TSDB selector in querier

This PR allows using the query distributed mode against a set of multi-tenant receivers
as described in https://github.com/thanos-io/thanos/blob/main/docs/proposals-done/202301-distributed-query-execution.md#distributed-execution-against-receive-components.

The feature is enabled by a selector.relabel-config flag in the Query component
which allows it to select a subset of TSDBs to query based on their external labels.

Signed-off-by: Filip Petkovski <[email protected]>

* Add CHANGELOG entry and fix docs

Signed-off-by: Filip Petkovski <[email protected]>

* Fix tests

Signed-off-by: Filip Petkovski <[email protected]>

* Add comments

Signed-off-by: Filip Petkovski <[email protected]>

* Add test case for MatchersForLabelSets

Signed-off-by: Filip Petkovski <[email protected]>

* Fix failing test

Signed-off-by: Filip Petkovski <[email protected]>

* Use an unbuffered channel

Signed-off-by: Filip Petkovski <[email protected]>

* Change flag description

Signed-off-by: Filip Petkovski <[email protected]>

* Remove parameter from ServerAsClient

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* Update thanos-io/promql-engine (#7215)

* Update thanos-io/promql-engine

This commit updates the promql-engine module to latest main and modifies
to remote engine based on the breaking change.

Signed-off-by: Filip Petkovski <[email protected]>

* Fix lint

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* expose NewPromSeriesSet (#7214)

Signed-off-by: Ben Ye <[email protected]>

* [CHORE] adding auto GOMEMLIMIT flag (#7223)

Co-authored-by: Filip Petkovski <[email protected]>

* bump objstore package version to latest main (#7228)

Signed-off-by: Ben Ye <[email protected]>

* Query-frontend: fix missing redis username config (#7224)

* add username cfg to rueidis client

Signed-off-by: Thibault Mange <[email protected]>

* update changelog

Signed-off-by: Thibault Mange <[email protected]>

---------

Signed-off-by: Thibault Mange <[email protected]>

* UI: Showing Block Size Stats (#7233)

* feat(ui): added BlockSizeStats calculation to blocks page

A block can have a list of contained files set in `.thanos.files`.
If the `files` array is set, all referenced files with `size_bytes` set are counted:
- sum of all `chunk/*` file sizes
- size of index file
- total size (sum of both)

Shows statistics about the selected block in the block details view:
- Total size of block
- Size of index (and percentage of total)
- Size of all chunks (and percentage of total)
- Daily growth, based on total size and block duration

Output is humanized up to Pebibytes and fixed to two decimal places;
raw bytes are accessible through mouse over / title text.

Signed-off-by: Markus Möslinger <[email protected]>

* feat(ui): added aggregated BlockSizeStats to blocks row title

Added total size of all blocks from a source to the row title, beneath the source name.

The shown total size is humanized up to pebibytes and fixed to two decimal places;
raw bytes value is accessible through mouse over / title text.

The shown value will refresh with selected compaction levels, but doesn't take block filter into account.

I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions.

Signed-off-by: Markus Möslinger <[email protected]>

* chore(docs): added UI block size PR to CHANGELOG.md

Signed-off-by: Markus Möslinger <[email protected]>

* chore(ui): removed comments

Automatic code formatting duplicated some comments near import statements.

Signed-off-by: Markus Möslinger <[email protected]>

---------

Signed-off-by: Markus Möslinger <[email protected]>

* Fix lazy expanded postings cache and bug of non equal matcher (#7220)

* fix lazy expanded postings cache and bug of non equal matcher with non existent values

Signed-off-by: Ben Ye <[email protected]>

* test case for remove keys noop

Signed-off-by: Ben Ye <[email protected]>

* add promqlsmith fuzz test

Signed-off-by: Ben Ye <[email protected]>

* update

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

* fix go mod

Signed-off-by: Ben Ye <[email protected]>

* rename test

Signed-off-by: Ben Ye <[email protected]>

* fix series request timestamp

Signed-off-by: Ben Ye <[email protected]>

* skip e2e test

Signed-off-by: Ben Ye <[email protected]>

* handle non lazy expanded case

Signed-off-by: Ben Ye <[email protected]>

* update comment

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Bump Prometheus to include new label regex optimization (#7232)

* bump Prometheus version to include new label matcher regex value optimization

Signed-off-by: Ben Ye <[email protected]>

* update

Signed-off-by: Ben Ye <[email protected]>

* fix again

Signed-off-by: Ben Ye <[email protected]>

* include latest fix

Signed-off-by: Ben Ye <[email protected]>

* update go mod

Signed-off-by: Ben Ye <[email protected]>

* fix explain test

Signed-off-by: Ben Ye <[email protected]>

* fix test again

Signed-off-by: Ben Ye <[email protected]>

* update again

Signed-off-by: Ben Ye <[email protected]>

* update

Signed-off-by: Ben Ye <[email protected]>

* fix tests so far

Signed-off-by: Ben Ye <[email protected]>

* fix compactor tests

Signed-off-by: Ben Ye <[email protected]>

* use own out of order chunk index

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* update docs for receive routing only with limits (#7241)

Signed-off-by: Alec Rajeev <[email protected]>

* docs: add thanoscon 2024 talks (#7243)

Signed-off-by: Michael Hoffmann <[email protected]>

* remove write method from Compactor interface (#7246)

Signed-off-by: Ben Ye <[email protected]>

* change shipper to not overwrite all external labels (#7247)

Signed-off-by: Ben Ye <[email protected]>

* fix(changelog): fix GOMEMLIMIT pull request reference

Signed-off-by: roth-wine <[email protected]>

* Update kakkoyun's affiliation (#7251)

* Added Shield in adopters (#7254)

* Added Shield in adopters

Signed-off-by: suhas.chikkanna.shield <[email protected]>

* Upload compatible image 

Signed-off-by: suhas-chikkanna <[email protected]>

---------

Signed-off-by: suhas.chikkanna.shield <[email protected]>
Signed-off-by: suhas-chikkanna <[email protected]>
Co-authored-by: suhas.chikkanna.shield <[email protected]>

* Tracing: added missing sampler types (#7231)

* added missing sampler types

Signed-off-by: Neeraj Nagure <[email protected]>

* added changelog entry

Signed-off-by: Neeraj Nagure <[email protected]>

* fixed changelog entry

Signed-off-by: Neeraj Nagure <[email protected]>

* Fixed changelog entry conflict

Signed-off-by: Neeraj Nagure <[email protected]>

---------

Signed-off-by: Neeraj Nagure <[email protected]>

* fix query_test when --race enabled (#7258)

Signed-off-by: Yi Jin <[email protected]>

* go.mod: bump promql-engine (#7263)

Bump promql-engine version to include samples counting.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix 7244 error targets page (#7245)

* added UNKNOWN to TargetHealth_value at targets proto

Signed-off-by: Tidhar Klein Orbach <[email protected]>

* added TargetHealth_value UNKNOWN to rpc.pb.go

Signed-off-by: Tidhar Klein Orbach <[email protected]>

---------

Signed-off-by: Tidhar Klein Orbach <[email protected]>

* clarify documentation around selecor.relabel-config option

Signed-off-by: notafile <[email protected]>

* clarify writing around sharding

Signed-off-by: notafile <[email protected]>

* don't halt compaction due to overlapping sources when vertical compaction is enabled (#7225)

Signed-off-by: Ben Ye <[email protected]>

* Propagate the query plan

* Serialize the plan for remote executions

latest engine

Signed-off-by: Pedro Tanaka <[email protected]>

Propagate marshaled plan and introduce optimizer

Propagating the query plan in the remote engine requests and introduce new SetProjectionColumns optimizer

Signed-off-by: Pedro Tanaka <[email protected]>

* Fixing passing down of plan

Signed-off-by: Pedro Tanaka <[email protected]>

* go mod tidy

Signed-off-by: Pedro Tanaka <[email protected]>

* avoid panics

Signed-off-by: Pedro Tanaka <[email protected]>

* delete dev file

Signed-off-by: Pedro Tanaka <[email protected]>

* undo small refactor

Signed-off-by: Pedro Tanaka <[email protected]>

* improve test

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>

generating protos

Signed-off-by: Pedro Tanaka <[email protected]>

fixing v1

Signed-off-by: Pedro Tanaka <[email protected]>

delete unused method

Signed-off-by: Pedro Tanaka <[email protected]>

Set projection labels after distributing queries

* removing second precision engine, upstream already truncates

Signed-off-by: Pedro Tanaka <[email protected]>

* Passing the plan along as the query in remote executions

Signed-off-by: Pedro Tanaka <[email protected]>

* Using proper constructors passing the query plan

Signed-off-by: Pedro Tanaka <[email protected]>

* Refactor query creation from plan

Signed-off-by: Pedro Tanaka <[email protected]>

* fallback in case we cant use plan

Signed-off-by: Pedro Tanaka <[email protected]>

* refactor, add tests

Signed-off-by: Pedro Tanaka <[email protected]>

* Refactor to method

Signed-off-by: Pedro Tanaka <[email protected]>

* last nits

Signed-off-by: Pedro Tanaka <[email protected]>

* api/ui: show peak/total samples in analyze (#7269)

Show the new peak/total fields in analyze output next to each operator.
Add tooltips to explain what is the meaning of each field.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive/handler: implement tenant label splitting (#7256)

* receive/handler: implement tenant label splitting

Implement splitting incoming HTTP requests along some label inside of
the timeseries themselves. This functionality is useful when you have
one big application exposing lots of series and, for instance, you have
a label `team` that identifies different owners of metrics in that
application. Then using this you can use that `team` label to have
different tenants in Thanos.

Only negative thing that I could spot is that if after splitting one of
the requests fails then that code is used for all tenants and that skews
the Receiver metrics a little bit. I think that can be left as a TODO
task.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* test/e2e: add more receiver tests

Signed-off-by: Giedrius Statkevičius <[email protected]>

* thanos/receive: note that splitting takes precendence over HTTP

Signed-off-by: Giedrius Statkevičius <[email protected]>

* thanos/receive: fix typo

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>

* query: fixing dedup iterator when working on mixed sample types (#7271)

* query: fixing dedup iterator when working on mixed sample types

There was a panic in case the dedupiterator worked on two chunks with both Native Histograms and Float (XOR encoded).

Co-authored-by: Sebastian Rabenhorst <[email protected]>
Signed-off-by: Pedro Tanaka <[email protected]>

* Adding changelog

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing lint

Signed-off-by: Pedro Tanaka <[email protected]>

* removing comments

Signed-off-by: Pedro Tanaka <[email protected]>

* Fixing repro test case

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing initialization

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing changelog

Signed-off-by: Pedro Tanaka <[email protected]>

* adding header to new file

Signed-off-by: Pedro Tanaka <[email protected]>

* using t.Run

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing ordering of samples in tests

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
Co-authored-by: Sebastian Rabenhorst <[email protected]>

* change the reflect package to an unsafe package (#7143)

- as 'reflect.String.Header' is deprecated, it is replaced with an unsafe package.

Signed-off-by: Youngjun <[email protected]>

* Receive: fix issue-7248 with parallel receive_forward (#7267)

* Receive: fix issue-7248 by introducing a worker pool

Signed-off-by: Yi Jin <[email protected]>

* fix unit test bug

Signed-off-by: Yi Jin <[email protected]>

* fix CLI flags not pass into the receive handler

Signed-off-by: Yi Jin <[email protected]>

* address comments

Signed-off-by: Yi Jin <[email protected]>

* init context in constructor

Signed-off-by: Yi Jin <[email protected]>

---------

Signed-off-by: Yi Jin <[email protected]>

* Fix incorrect comments (#7268)

Signed-off-by: Magiceses <[email protected]>

* [CHORE] adding user agent (#7281)

Signed-off-by: Nicolas Takashi <[email protected]>

* receive/multitsdb: do not delete not uploaded blocks (#7166)

* receive/multitsdb: do not delete not uploaded blocks

If a block hasn't been uploaded yet then tell the TSDB layer not to
delete them. This prevents a nasty race where the TSDB layer can delete
a block before the shipper gets to it. I saw this happen with a very
small block.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive/multitsdb: change order

Signed-off-by: Giedrius Statkevičius <[email protected]>

* shipper/receive: just use a single lock

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Compact: Replace group with resolution in ownsample metrics (#7283)

Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: #5841

Signed-off-by: SuperQ <[email protected]>

* Propagate warnings from instant queries

Warnings from remote instant queries get turned into errors, which
is a bug. It should be up to the root client to decide whether warnings
should be show as such, or converted to errors.

Signed-off-by: Filip Petkovski <[email protected]>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <[email protected]>

* Receive: stop relying on grpc server config to set grpc client secure/skipVerify

Signed-off-by: Guillaume Lecerf <[email protected]>

* Show warnings in query frontend (#7289)

* Show warnings in query frontend

QFE currently does not parse warnings from downstream queriers.
This commit fixes that by adding the field to proto messages and
modifies the merge function to take warnings into account.

Signed-off-by: Filip Petkovski <[email protected]>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <[email protected]>

* Omit empty warnings

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* Optimize empty posting check in lazy posting (#7298)

* change lazy postings empty posting check to use cardinality

Signed-off-by: Ben Ye <[email protected]>

* update lazy posting test

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Sidecar: mark as unqueryable if prometheus is down (#7297)

If the prometheus that belongs to a sidecar is down we dont need to
query the sidecar. This PR makes it so that we take the sidecar out of
the endpoint set then. We do the same for all other store APIs by
retuning an error in the info/Info gRPC call if they are marked as not
ready.

Signed-off-by: Michael Hoffmann <[email protected]>

* [CHORE] considering X-Forwarded-For on HTTP Logging (#7303)

Signed-off-by: Nicolas Takashi <[email protected]>

* Query|Receiver: Do not log full request on ProxyStore by default (#7305)

* Query|Receiver|Store: Do not log full request on ProxyStore by default

We had a problem on our production where a sudden increase in requests with long matchers was putting our receivers under a lot of pressure.
Upon checking profiles we saw that the problem was calls to Log()

S…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: compact good first issue help wanted
Projects
None yet
Milestone
No milestone
4 participants
@SuperQ @yeya24 @matej-g @Akshit42-hue