Frequent sync of block metadata since v0.24.0

[gebn]

Thanos, Prometheus and Golang version used:

thanos, version 0.24.0 (branch: HEAD, revision: 7fa248981cd98d2cd1e86b06854957d5b9a213fd)
  build user:       root@e4d6a4c7a9e7
  build date:       20211222-15:55:00
  go version:       go1.16.12
  platform:         linux/arm64

prometheus, version 2.32.1 (branch: HEAD, revision: 41f1a8125e664985dd30674e5bdf6b683eff5d32)
  build user:       root@a2a62523c036
  build date:       20211217-21:54:28
  go version:       go1.17.5
  platform:         linux/arm64

Object Storage Provider: AWS S3

What happened:

Since upgrading from v0.23.1 to v0.24.0, without changing any params or config, Compact synchronises block metadata every 5 minutes.

What you expected to happen:

Given the config below, Compact to only synchronise every 8 hours, with no requests to S3 in between, as before.

How to reproduce it (as minimally and precisely as possible):

thanos compact \
  --block-viewer.global.sync-block-interval 8h \
  --compact.concurrency 2 \
  --compact.cleanup-interval 0 \
  --data-dir /var/opt/thanos-compact/data \
  --delete-delay 12h \
  --http-address :10912 \
  --objstore.config-file /etc/opt/thanos-compact/objstore.yaml \
  --wait \
  --wait-interval 8h

Full logs to relevant components:

Logs showing block sync running every 5 mins:

Dec 23 07:22:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:22:45.538606018Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.001738>
Dec 23 07:27:46 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:27:46.627406768Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.090862>
Dec 23 07:32:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:32:45.591566647Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.055186>
Dec 23 07:37:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:37:45.564000538Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.027163>
Dec 23 07:42:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:42:45.510549321Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=973.7479>
Dec 23 07:47:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:47:45.561325178Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.024910>
Dec 23 07:52:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:52:45.612337581Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.075470>
Dec 23 07:57:45 i-082ee735d95e76e4b thanos[15668]: level=info ts=2021-12-23T07:57:45.587737639Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.050668>

S3 operations by Compact in the last week (upgrade @ 6pm on 2021-12-22):

[yeya24]

@gebn This is one new feature in the latest release #4801.
The compactor now will synchronize blocks metadata every 5m to calculate the progress of compaction, downsampling and retention and expose the progress to metrics.

To change this behavior, you can modify the --compact.progress-interval flag to a larger time or use 0s to disable progress checking if you don't worry about your compaction progress.

[gebn]

@yeya24 Ah, thanks for the prompt reply! I did look over the changelog, but interpreted that entry as only exposing additional metrics rather than adding a new feature.