thanos store sharding memory issue

[R4scal]

thanos, version 0.8.1 (branch: HEAD, revision: bd8278859b2321aaaa7514edde764816cc039d34)
  build user:       root@2227d9a2fdb1
  build date:       20191014-12:03:55
  go version:       go1.13.1

Object Storage Provider: minio

What happened:
I have multiple prometheus instances in single s3-bucket with different external labels service.
When I tried use --selector.relabel-config for sharding, but all my shards use some memory (as previously all-in-one store instance). So, because all shards have all-in-one memory, sharding not working now.

My configs:
3 prometheus witth external labels:
service: one
service: two
service: another

2 shards with configs:

 - source_labels: [service]
      separator: ;
      regex: one
      replacement: $1
      action: keep

 - source_labels: [service]
      separator: ;
      regex: not_exists
      replacement: $1
      action: keep

I specifically use a nonexistent label for the experiment. I this setup all my shards use around 10Gb Ram (not_exists too).

In thanos query ui i see:

• one: true labels and time range
• not_exists: no labels and time range beetween 292278994-08-17T07:12:55Z

In s3 logs i see that thanos get for blacklisted block all: index, index.cache.json, meta.json for all shards.

What you expected to happen:
Sharding with --selector.relabel-config requere memory as need for only shard blocks

[R4scal]

Additionally, since index.cache.json is not saved to disk for blacklisted blocks, it is downloaded from s3 each time a bucket is checked, and for some blocks it takes really big

[jojohappy]

@R4scal Is your first store memory usage in expectation?

[R4scal]

@jojohappy yes

[R4scal]

In addition, attach CPU and memory on a non-existent shard

[bwplotka]

This looks wrong, we are looking in this, thanks for reporting 👍 ~

[bwplotka]

Found the problem, fixing now

[bwplotka]

This should be fixed, can you verify? E.g by using master image: master-2019-10-22-64af185e from https://quay.io/repository/thanos/thanos?tag=latest&tab=tags

[R4scal]

Yep, I will try and reply today

[R4scal]

@bwplotka looks amazing

[bwplotka]

Awesome!

I think we internally agreed to not release v0.8.2 unless necessary as sharding is experimental. We want to release this in v0.9.0 around 20thNov.

Is that fine?

cc @jojohappy

[jojohappy]

I think we internally agreed to not release v0.8.2 unless necessary as sharding is experimental. We want to release this in v0.9.0 around 20thNov.

Agree.

[R4scal]

Ok, thanks