Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store: Querying fails - "The specified key does not exist." #2224

Closed
der-eismann opened this issue Mar 6, 2020 · 4 comments
Closed

Store: Querying fails - "The specified key does not exist." #2224

der-eismann opened this issue Mar 6, 2020 · 4 comments
Labels
component: store stale

Comments

@der-eismann
Copy link

der-eismann commented Mar 6, 2020 •
edited
Loading

Thanos, Prometheus and Golang version used: Thanos 0.11.0, Prometheus 2.16.0 (official docker images)

Object Storage Provider: S3

What happened: We upgraded to Thanos 0.11.0 in hopes of reducing memory consumption, but now the store gateway has another problem that occured twice. After a while (1-2 days running) all our Grafana dashboards stop working and the querier and store are full of errors. Restarting the store (aka deleting the pod) fixes the problem temporarily. Important: We use the experimental binary headers (--experimental.enable-index-header).

What you expected to happen: I expected the store to run longer without issues.

How to reproduce it (as minimally and precisely as possible): Absolutely no idea.

Full logs to relevant components:

Store Logs 

level=debug ts=2020-03-06T16:04:04.663627136Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:24.866532532Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:04:24.866612694Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488740000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:24.933501993Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:3 postingsTouchedSizeSum:76 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:16 seriesTouchedSizeSum:1393 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:33 chunksTouchedSizeSum:3725 chunksFetched:33 chunksFetchedSizeSum:52483 chunksFetchCount:3 chunksFetchDurationSum:162657398 getAllDuration:66773564 mergedSeriesCount:8 mergedChunksCount:33 mergeDuration:56254}" err=null
level=debug ts=2020-03-06T16:04:24.933562828Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488980000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:24.943364281Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:2 postingsTouchedSizeSum:92 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:21 seriesTouchedSizeSum:1983 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:72 chunksTouchedSizeSum:5849 chunksFetched:72 chunksFetchedSizeSum:38126 chunksFetchCount:2 chunksFetchDurationSum:17565594 getAllDuration:9698921 mergedSeriesCount:12 mergedChunksCount:72 mergeDuration:61701}" err=null
level=debug ts=2020-03-06T16:04:24.943413785Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583487240000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:24.980845143Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:6 postingsTouchedSizeSum:144 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:10 seriesTouchedSizeSum:968 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:27 chunksTouchedSizeSum:6054 chunksFetched:27 chunksFetchedSizeSum:54865 chunksFetchCount:3 chunksFetchDurationSum:68042216 getAllDuration:37336942 mergedSeriesCount:4 mergedChunksCount:27 mergeDuration:44340}" err=null
level=debug ts=2020-03-06T16:04:24.980911625Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583487240000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.030550028Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:6 postingsTouchedSizeSum:144 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:10 seriesTouchedSizeSum:968 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:27 chunksTouchedSizeSum:2654 chunksFetched:27 chunksFetchedSizeSum:51049 chunksFetchCount:3 chunksFetchDurationSum:66257354 getAllDuration:49548942 mergedSeriesCount:4 mergedChunksCount:27 mergeDuration:39940}" err=null
level=debug ts=2020-03-06T16:04:25.030608559Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583487240000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.097568388Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:3 postingsTouchedSizeSum:92 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:20 seriesTouchedSizeSum:1936 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:54 chunksTouchedSizeSum:8708 chunksFetched:54 chunksFetchedSizeSum:58895 chunksFetchCount:3 chunksFetchDurationSum:84639746 getAllDuration:66851129 mergedSeriesCount:8 mergedChunksCount:54 mergeDuration:60914}" err=null
level=debug ts=2020-03-06T16:04:25.097627689Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.097784943Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:110468 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:9613}" err=null
level=debug ts=2020-03-06T16:04:25.097836849Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098011644Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:42 postingsTouchedSizeSum:12461044 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:148631 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:406}" err=null
level=debug ts=2020-03-06T16:04:25.098087151Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098234523Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:42 postingsTouchedSizeSum:12461044 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:116237 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:428}" err=null
level=debug ts=2020-03-06T16:04:25.09827799Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098321183Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:21663 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:232}" err=null
level=debug ts=2020-03-06T16:04:25.09835193Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098482086Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:42 postingsTouchedSizeSum:12461044 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:109587 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:214}" err=null
level=debug ts=2020-03-06T16:04:25.098502618Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.09853895Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:17787 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:162}" err=null
level=debug ts=2020-03-06T16:04:25.098566352Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098712771Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:46 postingsTouchedSizeSum:18551568 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:126715 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:214}" err=null
level=debug ts=2020-03-06T16:04:25.0987408Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.098870425Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:42 postingsTouchedSizeSum:12461044 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:103811 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:216}" err=null
level=debug ts=2020-03-06T16:04:25.098904673Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488860000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.099070901Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:46 postingsTouchedSizeSum:18551568 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:138524 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:267}" err=null
level=debug ts=2020-03-06T16:04:25.099252358Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583487240000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:25.132867082Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:6 postingsTouchedSizeSum:144 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:10 seriesTouchedSizeSum:968 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:27 chunksTouchedSizeSum:6054 chunksFetched:27 chunksFetchedSizeSum:54865 chunksFetchCount:3 chunksFetchDurationSum:53193957 getAllDuration:33517143 mergedSeriesCount:4 mergedChunksCount:27 mergeDuration:46865}" err=null
level=debug ts=2020-03-06T16:04:29.586213784Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.674654818Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:04:53.674732207Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488440000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.737254673Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:15 postingsTouchedSizeSum:1504 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:8 seriesTouchedSizeSum:985 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:20 chunksTouchedSizeSum:1326 chunksFetched:20 chunksFetchedSizeSum:65136 chunksFetchCount:3 chunksFetchDurationSum:146100558 getAllDuration:62426649 mergedSeriesCount:4 mergedChunksCount:20 mergeDuration:40815}" err=null
level=debug ts=2020-03-06T16:04:53.73732712Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488440000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.882086349Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:15 postingsTouchedSizeSum:1464 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:6 seriesTouchedSizeSum:769 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:18 chunksTouchedSizeSum:1197 chunksFetched:18 chunksFetchedSizeSum:64161 chunksFetchCount:3 chunksFetchDurationSum:199944163 getAllDuration:144610205 mergedSeriesCount:2 mergedChunksCount:18 mergeDuration:51425}" err=null
level=debug ts=2020-03-06T16:04:53.882147197Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488440000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.905806055Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:15 postingsTouchedSizeSum:2488 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:7 seriesTouchedSizeSum:861 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:19 chunksTouchedSizeSum:1237 chunksFetched:19 chunksFetchedSizeSum:64431 chunksFetchCount:3 chunksFetchDurationSum:58026787 getAllDuration:23581511 mergedSeriesCount:3 mergedChunksCount:19 mergeDuration:32346}" err=null
level=debug ts=2020-03-06T16:04:53.905861942Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488440000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.928250153Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:9 postingsTouchedSizeSum:1996 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:32 seriesTouchedSizeSum:3936 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:80 chunksTouchedSizeSum:5567 chunksFetched:80 chunksFetchedSizeSum:72641 chunksFetchCount:3 chunksFetchDurationSum:39166821 getAllDuration:22250095 mergedSeriesCount:16 mergedChunksCount:80 mergeDuration:90293}" err=null
level=debug ts=2020-03-06T16:04:53.928312742Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583485440000 maxt=1583508840000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.950882673Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:12 postingsTouchedSizeSum:5380 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:8 seriesTouchedSizeSum:920 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:22 chunksTouchedSizeSum:1629 chunksFetched:22 chunksFetchedSizeSum:138916 chunksFetchCount:3 chunksFetchDurationSum:40692698 getAllDuration:22480193 mergedSeriesCount:4 mergedChunksCount:22 mergeDuration:36263}" err=null
level=debug ts=2020-03-06T16:04:53.95094303Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583487240000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:53.997144125Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:12 postingsTouchedSizeSum:5368 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:8 seriesTouchedSizeSum:922 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:20 chunksTouchedSizeSum:1462 chunksFetched:20 chunksFetchedSizeSum:138213 chunksFetchCount:3 chunksFetchDurationSum:67249497 getAllDuration:46108549 mergedSeriesCount:4 mergedChunksCount:20 mergeDuration:41518}" err=null
level=debug ts=2020-03-06T16:04:53.997204486Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=3 MaximumResolution=0 mint=1583488740000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583481600000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:04:54.009644689Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:3 postingsTouched:12 postingsTouchedSizeSum:6048 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:8 seriesTouchedSizeSum:1008 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:20 chunksTouchedSizeSum:1330 chunksFetched:20 chunksFetchedSizeSum:150071 chunksFetchCount:3 chunksFetchDurationSum:31603307 getAllDuration:12354934 mergedSeriesCount:4 mergedChunksCount:20 mergeDuration:37945}" err=null
level=debug ts=2020-03-06T16:04:54.009702413Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=9223372036854775807 mint=1583424284000 maxt=1583510684000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:26.903634588Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:26.903745392Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488800000 maxt=1583510700000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:26.967032514Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:2 postingsTouchedSizeSum:56 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:12 seriesTouchedSizeSum:1001 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:29 chunksTouchedSizeSum:3147 chunksFetched:29 chunksFetchedSizeSum:35098 chunksFetchCount:2 chunksFetchDurationSum:102210962 getAllDuration:63172587 mergedSeriesCount:8 mergedChunksCount:29 mergeDuration:48476}" err=null
level=debug ts=2020-03-06T16:05:26.967113891Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1583424027000 maxt=1583424327000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583431200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:26.972622904Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:1 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:26.97270746Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:31.143826761Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:1 postingsTouchedSizeSum:136 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:33 seriesTouchedSizeSum:4120 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:110 chunksTouchedSizeSum:10649 chunksFetched:110 chunksFetchedSizeSum:27145 chunksFetchCount:1 chunksFetchDurationSum:40305294 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:31.143918058Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:35.631247614Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P7YY4XH4P9XZ3EJ30MMMX1: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:35.631371198Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:37.424233499Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: Get https://<bucket address>.s3.dualstack.eu-west-1.amazonaws.com/01E2Q3DV4WG47Z152VA6C57G7D/chunks/000001: context deadline exceeded"
level=debug ts=2020-03-06T16:05:37.424362321Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424000000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:38.410821706Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P7YY4XH4P9XZ3EJ30MMMX1: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:38.410935641Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424120000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:39.413721432Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P7YY4XH4P9XZ3EJ30MMMX1: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:39.41381846Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:39.840399858Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:39.840514647Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:41.947759393Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:41.947857111Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:43.359166994Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:43.359281689Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:43.666543638Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:2 postingsTouchedSizeSum:540 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:28 seriesTouchedSizeSum:3807 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:106 chunksTouchedSizeSum:9021 chunksFetched:106 chunksFetchedSizeSum:33485 chunksFetchCount:1 chunksFetchDurationSum:42087335 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:43.66665674Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:46.504462597Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:46.504565589Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:47.725631197Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QA9JCWM617J3C7JD1ZRDKP: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:47.725709066Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:48.109958626Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:48.110063777Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:49.936878545Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:49.936956614Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:52.046110311Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:52.046196047Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=9223372036854775807 mint=1583424344000 maxt=1583510744000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:05:57.972443671Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:1 postingsTouchedSizeSum:24 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:5 seriesTouchedSizeSum:445 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:5 chunksTouchedSizeSum:4576 chunksFetched:5 chunksFetchedSizeSum:19704 chunksFetchCount:1 chunksFetchDurationSum:8392729 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:05:57.972535709Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:06:00.512836658Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:06:00.512912186Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:06:16.09874759Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:13 postingsTouchedSizeSum:254228 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:57 seriesTouchedSizeSum:2811 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:91 chunksTouchedSizeSum:30580 chunksFetched:91 chunksFetchedSizeSum:47109 chunksFetchCount:1 chunksFetchDurationSum:10515733 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:06:16.098828272Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:06:40.354284509Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:06:40.354361147Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:01.795973388Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:01.796100881Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=9223372036854775807 mint=1583424404000 maxt=1583510804000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:21.060346666Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:4 postingsTouchedSizeSum:664 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:27 seriesTouchedSizeSum:2678 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:12 chunksTouchedSizeSum:816 chunksFetched:12 chunksFetchedSizeSum:36542 chunksFetchCount:2 chunksFetchDurationSum:18046943 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:21.060475995Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424060000 maxt=1583510760000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:21.905741366Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QA9JCWM617J3C7JD1ZRDKP: preload chunks: read range for 0: get range reader: Get https://<bucket address>.s3.dualstack.eu-west-1.amazonaws.com/01E2QA9JCWM617J3C7JD1ZRDKP/chunks/000001: context deadline exceeded"
level=debug ts=2020-03-06T16:07:21.905857285Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424240000 maxt=1583510760000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:25.72794908Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P7YY4XH4P9XZ3EJ30MMMX1: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:25.728027686Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1583488920000 maxt=1583510820000 lset="{replica=\"0\"}" spans="Range: 1583488800000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:25.771410249Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:2 postingsTouchedSizeSum:56 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:12 seriesTouchedSizeSum:1001 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:29 chunksTouchedSizeSum:3147 chunksFetched:29 chunksFetchedSizeSum:35098 chunksFetchCount:2 chunksFetchDurationSum:78974201 getAllDuration:43261951 mergedSeriesCount:8 mergedChunksCount:29 mergeDuration:45962}" err=null
level=debug ts=2020-03-06T16:07:29.629208197Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:30.293140439Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:30.293211815Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:31.456761576Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:31.456833825Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:33.680778378Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QA9JCWM617J3C7JD1ZRDKP: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:33.680866556Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:35.581624729Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:35.581698077Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:36.57484742Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:36.574931869Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:48.337571357Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:48.337651204Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423940000 maxt=1583510640000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:52.089505388Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:52.089637598Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:53.481297786Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P7YY4XH4P9XZ3EJ30MMMX1: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:53.481413428Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:53.535278129Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:12 postingsTouchedSizeSum:624 postingsToFetch:0 postingsFetched:1 postingsFetchedSizeSum:52 postingsFetchCount:1 postingsFetchDurationSum:53686820 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:53777373 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:2940}" err=null
level=debug ts=2020-03-06T16:07:53.535392859Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:53.538379967Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:53.538478852Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:54.522987373Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:54.523140204Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:54.949613282Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:54.949686904Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=9223372036854775807 mint=1583424464000 maxt=1583510864000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:07:55.382362079Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:07:55.382451555Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:08:01.700992893Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:08:01.962226739Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424180000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:08:07.704771765Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:08:07.704916776Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424240000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:08:17.268585493Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:08:17.268717093Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424360000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:08:41.789868866Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:08:41.789948885Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:04.713581619Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:04.713666031Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=9223372036854775807 mint=1583424524000 maxt=1583510924000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:06.416217863Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:06.651508256Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424180000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:07.899107823Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2P136WXH3F1NYYX5YK3VDRK: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:07.899230828Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424240000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:12.213991750Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PETNCWTHKXMCHNFX57HRM4: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:12.214130472Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=11 MaximumResolution=0 mint=1583424360000 maxt=1583510880000 lset="{replica=\"0\"}" spans="Range: 1583424000000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:17.226285400Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PNPCMWZGW14KJTJ82AJB6Y: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:17.226405870Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:21.695911224Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:21.696006689Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423400000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:32.851190976Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:11 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2PWJ3WW6PHARG4DED8W7WCZ: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:32.851276233Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:39.469662184Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2Q3DV4WG47Z152VA6C57G7D: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:39.469745309Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"
level=debug ts=2020-03-06T16:09:41.930755289Z caller=bucket.go:921 msg="stats query processed" stats="&{blocksQueried:12 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:0 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:0}" err="rpc error: code = Aborted desc = fetch series for block 01E2QH59MXGQQGFPE4EFG7N94T: preload chunks: read range for 0: get range reader: The specified key does not exist."
level=debug ts=2020-03-06T16:09:41.930838662Z caller=bucket.go:823 msg="Blocks source resolutions" blocks=12 MaximumResolution=0 mint=1583423700000 maxt=1583510400000 lset="{replica=\"0\"}" spans="Range: 1583395200000-1583503200000 Resolution: 0"

Querier Logs 

level=warn ts=2020-03-06T16:09:56.021265441Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.021609873Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:09:56.027352856Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.027801666Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:09:56.057632569Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.057863721Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:09:56.057882342Z caller=proxy.go:293 err="No StoreAPIs matched for this query" stores="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=debug ts=2020-03-06T16:09:56.05820273Z caller=proxy.go:289 msg="store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried;store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried"
level=warn ts=2020-03-06T16:09:56.108574392Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.108988853Z caller=proxy.go:289 msg="store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried;store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried"
level=warn ts=2020-03-06T16:09:56.132260219Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.132678611Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:09:56.138958805Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:09:56.13930672Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.380839438Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.380896596Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.38111858Z caller=proxy.go:289 msg="store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried;store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried"
level=warn ts=2020-03-06T16:10:00.381137371Z caller=proxy.go:293 err="No StoreAPIs matched for this query" stores="store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried;store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried"
level=debug ts=2020-03-06T16:10:00.381388181Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.392023552Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.392075368Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.392450703Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.392636592Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.392679167Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.392927802Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.394674897Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.39471971Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.39498843Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.398020237Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.398062503Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.398330342Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.414586069Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.414928408Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.415108504Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.415173997Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.415591298Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=debug ts=2020-03-06T16:10:00.415733191Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.417041342Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.417089037Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.417360307Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=debug ts=2020-03-06T16:10:00.417476174Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.419878139Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.420196187Z caller=proxy.go:289 msg="store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried;store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried"
level=warn ts=2020-03-06T16:10:00.422328609Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=debug ts=2020-03-06T16:10:00.422622589Z caller=proxy.go:289 msg="store Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807 queried;store Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000 queried"
level=warn ts=2020-03-06T16:10:00.426199657Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.426254591Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.426550389Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.428837032Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.432546852Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.437163955Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.437216545Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.437430806Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.437476056Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.444168382Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.444236761Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.444550317Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.444631535Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.452323943Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.452395315Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.452448482Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.77.83:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1551074400000 Maxt: 1583503200000: context deadline exceeded" msg="returning partial response"
level=warn ts=2020-03-06T16:10:00.45248831Z caller=proxy.go:450 err="failed to receive any data from Addr: 10.11.102.126:10901 LabelSets: [name:\"replica\" value:\"0\" ] Mint: 1583323200000 Maxt: 9223372036854775807: context deadline exceeded" msg="returning partial response"

@GiedriusS
Copy link
Member

Could you upload the goroutine dump (and others, if possible) when this happens of Thanos Store? They are available via /debug/pprof.

@der-eismann
Copy link
Author

It happened again, so there you go 🙂
store_allocs.txt
store_heap.txt
store_threadcreate.txt
store_goroutine.txt

@stale
Copy link

stale bot commented Apr 8, 2020

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

@stale stale bot added the stale label Apr 8, 2020
@bwplotka
Copy link
Member

bwplotka commented Apr 9, 2020

Hi, I am more than sure you are hitting this: #564

Which was fixed just in this upcoming release: https://github.com/thanos-io/thanos/releases

Feel free to update to v0.12.0-rc.1 and let us know how it goes! ❤️

Closing for now, let's reopen if not fixed.

@bwplotka bwplotka closed this as completed Apr 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: store stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@GiedriusS @bwplotka @der-eismann