-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(approx_topk): Map approx_topk operation in all cases #16131
base: main
Are you sure you want to change the base?
Conversation
pkg/logql/syntax/ast.go
Outdated
// binops - comparison | ||
OpTypeGT: true, | ||
OpTypeGTE: true, | ||
OpTypeLT: true, | ||
OpTypeLTE: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure these are shardable? Is this an orthogonal change to the above?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I understand, we should treat them the same way as the arithmetic operations above. There's no special logic for those operations, they just get mapped to queriers and combined as usual.
I don't think sharding a approx_topk query containing a comparison will change the result, as far as I can see, though I don't have any proof beyond the existing tests that it is valid to do so.
If you prefer I can remove this change and those types of queries will fall into the "not shardable" logic instead. I've tested it locally and it does seem to give correct results - WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, so I'd push the operator don to each shard because it's filtering out the metrics. Does that makes sense? As in:
count_over_timer({...}[...]) > 0
should be
downstream<count_over_timer({..., __shard__=0}[...]) > 0>
++ ...
++ downstream<count_over_timer({..., __shard__=n}[...]) > 0>
and not
(downstream<count_over_timer({..., __shard__=0}[...])>
++ ...
++ downstream<count_over_timer({..., __shard__=n}[...])>
) > 0
I think sharding these is a little more evolved and should be a separarte PR.
What this PR does / why we need it:
Calling approx_topk on some types of queries was causing a panic in the querier.
This PR does 2 things:
1. Adds comparison operators to "shardableOps" map. This means they will be considered shardable and trigger the sharding logic. This is the same as the existing arithmetic operators - we don't have any special logic for them, though it doesn't really give us any benefit either.Removed because it not clear this is still correct.2. Adds a catch-all for the approx_topk operation. If the inner query is NOT shardable, we still rewrite the logic as if there are 0 shards and send the inner query to the querier. This won't panic but it shouldn't be happening a lot so I added an error log.
I added tests for these cases & did some local testing to check they behave as intended.
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
deprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR