Cache expanded postings

[yeya24]

Is your proposal related to a problem?

#6416 mentioned some problems when fetching postings. One thing is that, if the postings contain a lot of series IDs but after intersection it only matches few series, we still need to fetch postings for all matchers

Describe the solution you'd like

We can try to cache the postings after expansion. This could save both CPU time for merging postings and mem/bandwidth for fetching postings from caches.

Let's say a query up{cluster="us", env="prod"}. Each matcher matches 100, 100K and 500K respectively and we need to fetch 600K postings.
If we have expanded postings cached, we only need to fetch < 100 postings.

Describe alternatives you've considered

Add new methods in index cache to store and fetch expanded postings. The data format can reuse what we have: store diff+varint+streamed snappy and the cache key will be blockID + label matchers.

[fpetkovski]

I wonder if postings would be better suited on disk with the index header, instead of in object storage 🤔

[yeya24]

@fpetkovski I think it is a good point. Maybe we could experiment with that. From some real data I can see, the posting part is relatively small compared to the series section in tsdb index so probably we can try that.

But I still think it is valuable to cache expanded postings as we don't have to process label matchers and do intersections.