Reduce PostingsForMatchersCache.expire() pressure on mutex

[pracucci]

Currently, PostingsForMatchersCache.expire() is called for every single PostingsForMatchers() call. If the cache is full, it's expected that there's always something to expire from the cache (1 in, 1 out). However, in a high concurrency environment, there's no need to call expire() concurrently. We just need 1 goroutine to cleanup stale or over-the-capacity cached entries.

In this PR I propose a simple solution to only 1 expire() execution at a time.

I've added a new benchmark:

goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/tsdb
cpu: Apple M3 Pro
                                                          │  before.txt  │              after.txt              │
                                                          │    sec/op    │   sec/op     vs base                │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   1653.0n ± 4%   335.1n ± 8%  -79.73% (p=0.000 n=10)

                                                          │ before.txt  │              after.txt              │
                                                          │    B/op     │    B/op      vs base                │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   1446.5 ± 0%   1004.0 ± 0%  -30.59% (p=0.000 n=10)

                                                          │ before.txt │             after.txt              │
                                                          │ allocs/op  │ allocs/op   vs base                │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   29.00 ± 0%   20.00 ± 0%  -31.03% (p=0.000 n=10)

[56quarters]

Changes LGTM. It seems like this is a case where RWLock.TryRLock() could be used but I don't feel strongly about using it instead of the atomic here.

[pracucci]

Changes LGTM. It seems like this is a case where RWLock.TryRLock() could be used but I don't feel strongly about using it instead of the atomic here.

I tried it, but it looks slower. Here is the comparison between this PR vs Try...Lock:

goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/tsdb
cpu: Apple M3 Pro
                                                          │ before.txt  │              after.txt               │
                                                          │   sec/op    │   sec/op     vs base                 │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   303.8n ± 2%   795.8n ± 2%  +161.93% (p=0.000 n=10)

                                                          │ before.txt  │              after.txt              │
                                                          │    B/op     │    B/op      vs base                │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   1016.0 ± 0%   1371.5 ± 0%  +34.99% (p=0.000 n=10)

                                                          │ before.txt │             after.txt              │
                                                          │ allocs/op  │ allocs/op   vs base                │
PostingsForMatchersCache_ConcurrencyOnHighEvictionRate-11   21.00 ± 0%   27.00 ± 0%  +28.57% (p=0.000 n=10)

Note: it's still faster than main, but not as fast as this PR.

[56quarters]

Makes sense, thanks!