Shard ingester queries.

[cyriltovena]

This is still experimental but already yield from 2x to 6x faster for short period queries.

I'm still playing with it but I want to share how I do it early.

Signed-off-by: Cyril Tovena [email protected]

[cyriltovena]

This is promising, I'm getting exceptional result in our biggest cluster.

[owen-d]

I've added some suggestions which I tested locally but didn't want to push to your branch. They are based off my fork of this PR.

This does two things:

1. Fixes the hashing calculations - notably maintaining that the internal hash space must be consistent. i.e. moving from 2 buckets to 4, the first bucket (0), covers the first 50% of the hash ring. Moving this same space into 4 buckets should actually map shards (0_of_2) -> (0_of_4, 1_of_4) instead of (0_of_2) -> (0_of_4, 2_of_4).
2. Adds logic for mapping any schema sharding factor into the relevant superset [shard] of any inverted index sharding factor. Notably, this would allow us to decouple the two. However, because the shard factors wouldn't need to be mutually divisible any more, we'd need to amend the Lookup, LabelNames, and LabelValues to also filter any returned values, ensuring we filter out any not belonging to the desired schema shard. Because this code is currently protected by the validateShard function, we can safely defer this decision for now.

One thing which I didn't include but expect we'll need is to align the hash functions used in the Cortex Index with the one used here. Notably, this is not exposed from cortex and is currently hardcoded into the Entries implementations. Since the chunk store is deprecated at this point, I expect we can copy that as it's not subject to change. The reason why we'll need to agree on a specific hash function here is because the querier queries both the store and the ingester for a particular shard. Our queries will quickly prove incorrect when series belong to different shards on each of these query paths.

[cyriltovena]

Our queries will quickly prove incorrect when series belong to different shards on each of these query paths.

Not sure about that.

[cyriltovena]

So I have an idea :) I'm going to change sharding in the store to align it with ingester see you soon !

[cyriltovena]

@owen-d I think this is ready I've aligned the shard computation in the ingester. This means they use the same type of hash. I would have prefer to keep it based of the fingerprint but it's hard to change in the storage.

Something to keep in mind when we redesign the storage.

[owen-d]

Couple nits, then LGTM! We'll need to eventually add a way to either enforce that schema configs are multiples of indexShards or find a way to refactor the inverted index to work with any arbitrary sharding factor.

[cyriltovena]

Couple nits, then LGTM! We'll need to eventually add a way to either enforce that schema configs are multiples of indexShards or find a way to refactor the inverted index to work with any arbitrary sharding factor.

I don't want to invest too much in this yet. Not sure what is the future of that code.