could not delete LabelSets (cluster not used) on store-gateway

[heisenbergye]

Thanos, Prometheus and Golang version used:

thanos version: v0.12.1
promethues version: v2.3.2

Object Storage Provider: minio

What happened:

I used a bucket for three clusters' sidecars, after several days, I assigned clusters “ci-prod-cluster-xxx” to another bucket and remove it on querier, but the LabelSets of “cluster=ci-prod-cluster-xxx” on store-gateway wasn't deleted.
And I also retry to restart & re-deploy store-gateway, but it didn‘t work.

What you expected to happen:
the LabelSets of “cluster=ci-prod-cluster-xxx” on store-gateway deleted automatically.

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

[GiedriusS]

Hmm, if I understood you correctly then you have removed the Sidecars that have produced those blocks but you still left those blocks in the remote object storage? In such a case, it is expected that they would still show up from Thanos Store since the data is still there. Could you please clarify this?

[heisenbergye]

@GiedriusS Yes, that's right, I didn't clean up the “ci-prod-cluster-xx” data in the bucket after remove it, because there are too many blocks in the bucket and I don't know which need to be deleted :-( That's very kind of you to tell me how to delete the residual block data.
from the docs block-deletion, it says that deletion-mark.json can help to delete the block that was chosen, but I don't know how to write this file and where to upload.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[GiedriusS]

Just delete the appropriate directories according to their external labels. deletion-mark.json is an internal implementation thing that Thanos uses to mark blocks before their deletion. You could create them yourself by looking at the code to gracefully delete blocks but I don't think this is the way to go forward. I believe this will be properly solved during GSoC. Take a look at this PR: #2780.