storegateway: improve contention around lazy-loading of blocks

[narqo]

What this PR does

These changes are meant to improve the contention around the internal set of blocks in storegateway.BucketStore.

Here, the idea is to rely on sync.Map as a data structure that holds blocks:

The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

Which issue(s) this PR fixes or relates to

Fixes #8258

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[narqo]

I moved the changes back to draft. I think we're now allowing a race between (*bucketBlock).Close() and (*bucketBlock).loadedIndexReader(), which was impossible before, due to a long-held lock around s.blocks. Will need to think more on how to resolve that.

[narqo]

I think we're now allowing a race between (*bucketBlock).Close() and (*bucketBlock).loadedIndexReader() ...

I believe it's now resolved, and so the PR is ready for review @56quarters @dimitarvdimitrov

[56quarters]

LGTM with a few small comments. I'd like to here @dimitarvdimitrov's opinion before merging though.

[dimitarvdimitrov]

sorry for the delay, i'll take a look tomorrow

[dimitarvdimitrov]

this makes lazy loading in LabelValues call sequential instead of concurrent. I think we should fix it before merging. Left some other less critical remarks.

[narqo]

@dimitarvdimitrov I've addressed all the comments. Could you give it another look?

[dimitarvdimitrov]

LGTM! 🚀