Reduce sync concurrency in store-gateway by default to reduce disk contention

[andyasp]

What this PR does

Follow-up to #5025. Internally we reduced the concurrency even when lazy loading was enabled. This upstreams those changes to the defaults.

Which issue(s) this PR fixes or relates to

Fixes N/A

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[pracucci]

I'm not super convinced we want to update the Mimir default, rather than changing the default config in Jsonnet and Helm. Reason is that this change makes sense if you run on HDD, but not on SSD. Changing this default we're assuming people run on HDD vs SSD, which I'm not sure it's a fair assumption.

[pracucci]

Discussed offline. LGTM, thanks!

[aallawala]

👋 @andyasp, is the guidance for this to keep this the same if we're running on SSDs? How can we validate whether this will introduce a regression or not?

[dimitarvdimitrov]

is the guidance for this to keep this the same if we're running on SSDs? How can we validate whether this will introduce a regression or not?

The performance of syncing blocks when running on SSDs should be minimal. In general I don't expect that reducing the concurrency there has a negative effect.

One way to verify whether this will have a negative effect is to check how long blocks syncing takes today. Currently measuring sync duration is only possible by looking at logs timestamps from the store-gateway: synchronizing TSDB blocks for all users when syncing starts and successfully synchronized TSDB blocks for all users when syncing finishes.

You want these syncs to not take longer than the sync interval (blocks-storage.bucket-store.sync-interval, default 15m). The reduction in concurrency was 50x in multi-tenant clusters and 20x in single-tenant ones. Depending on the SSD performance/throughput & IOPS limits it might be safe to assume a linear relationship between the concurrency and the time it takes to complete a sync, but you have to verify this in your environment. So roughly speaking if syncs today take less than 18 seconds, it's safe to take this change.