store: aborts query when trying to access just-compacted block

[kklimonda]

Thanos, Prometheus and Golang version used

thanos, version 0.4.0 (branch: HEAD, revision: a676095877e7a87576b0f7776fccd4b3f8b65188)
  build user:       root@3e7111579ad1
  build date:       20190504-09:31:11
  go version:       go1.11.9

Not sure if related, but I'm testing with Swift backend exposed by Ceph nautilus RadosGW.

What happened

thanos-store has aborted the query with error get range reader: Resource not found. The query in question was a simple sum(up) over a range of ~22 days. Prometheus is configured to retain only 14 days of metrics, the rest is pulled from object storage.

What you expected to happen

The query to execute and return result.

How to reproduce it (as minimally and precisely as possible):

Perhaps make a query just after compactor deletes a block?

Full logs to relevant components

Uncomment if you would like to post collapsible logs:

thanos-store logs:

May 22 19:49:25 monitor03 thanos[19478]: level=debug ts=2019-05-22T19:49:25.211923948Z caller=bucket.go:803 msg="stats query processed" stats="&{blocksQueried:115 postingsTouched:109 postingsTouchedSizeSum:18504 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:4517 seriesTouchedSizeSum:655888 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:79057 chunksTouchedSizeSum:6159822 chunksFetched:79057 chunksFetchedSizeSum:8746894 chunksFetchCount:109 chunksFetchDurationSum:393262394528 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01DBGBSAKK7VDWFN59P189J3KB: expanded matching posting: get postings: read postings range: get range reader: Resource not found"

thanos-compact logs

May 22 19:46:19 monitor03 thanos[25422]: level=debug ts=2019-05-22T19:46:19.304482368Z caller=compact.go:824 compactionGroup="0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}" msg="downloaded and verified blocks" blocks="[/var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBQHTHEE0736AW0B9G8KT2 /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBSAKK7VDWFN59P189J3KB /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBV5CYSYX0071F4FTPCTGD /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBX3V4FV3G8EEVAKNVH1MZ /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBZR2ZQ6C86FMBP2VCZGYN /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGC1SJSMXHTD9E1D4MMGZN5]" duration=56.552987761s
May 22 19:46:38 monitor03 thanos[25422]: level=info ts=2019-05-22T19:46:38.340457701Z caller=compact.go:441 msg="compact blocks" count=6 mint=1556928000000 maxt=1557100800000 ulid=01DBGH9147ARSNQ4QJWHHK6SD6 sources="[01DBGBQHTHEE0736AW0B9G8KT2 01DBGBSAKK7VDWFN59P189J3KB 01DBGBV5CYSYX0071F4FTPCTGD 01DBGBX3V4FV3G8EEVAKNVH1MZ 01DBGBZR2ZQ6C86FMBP2VCZGYN 01DBGC1SJSMXHTD9E1D4MMGZN5]" duration=19.034222752s
May 22 19:46:38 monitor03 thanos[25422]: level=debug ts=2019-05-22T19:46:38.417277842Z caller=compact.go:851 compactionGroup="0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}" msg="compacted blocks" blocks="[/var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBQHTHEE0736AW0B9G8KT2 /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBSAKK7VDWFN59P189J3KB /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBV5CYSYX0071F4FTPCTGD /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBX3V4FV3G8EEVAKNVH1MZ /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGBZR2ZQ6C86FMBP2VCZGYN /var/lib/thanos/compactor/compact/0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}/01DBGC1SJSMXHTD9E1D4MMGZN5]" duration=19.111046214s
May 22 19:47:40 monitor03 thanos[25422]: level=info ts=2019-05-22T19:47:40.597837216Z caller=compact.go:918 compactionGroup="0@{datacenter=\"dc\",environment=\"test-env\",region=\"region1\",replica=\"02\"}" msg="deleting compacted block" old_block=01DBGBSAKK7VDWFN59P189J3KB

Anything else we need to know

[GiedriusS]

Hello there. Seems like this is related to:
#564
#298
Can we keep the discussion in those issues?

[kklimonda]

Fine with me, thanks for the links - didn't find them. I'll let the maintainer mark it as duplicate.

[GiedriusS]

👍 let's move the discussion there, this is a known problem