Mixin: Lower alerting threshold for BucketIndexNotUpdated alert #7879
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
56quarters
force-pushed
the
56quarters/stale-bucket-index
branch
from
ba0c98e to
67aa6af
Compare
This change will cause us to page when a bucket index has not been updated by two compactor cleanup cycles (every 15 minutes) with a 5 minute buffer to avoid false-positives. The update time of the index is checked when queriers perform queries so as soon as it exceeds the 1 hour max age, queries begin to fail. Thus we need to alert on the age of the index before it begins to fail queries. Signed-off-by: Nick Pillitteri <[email protected]>
56quarters
force-pushed
the
56quarters/stale-bucket-index
branch
from
67aa6af to
8f388d2
Compare
dimitarvdimitrov
approved these changes
Apr 11, 2024
56quarters
added a commit
that referenced
this pull request
Apr 11, 2024
Add potential mitigations for stale bucket index alerts to the runbook. Related #7879 Signed-off-by: Nick Pillitteri <[email protected]>
4 tasks
56quarters
added a commit
that referenced
this pull request
Apr 11, 2024
…7883) Add potential mitigations for stale bucket index alerts to the runbook. Related #7879 Signed-off-by: Nick Pillitteri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
This change will cause us to page when a bucket index has not been updated by two compactor cleanup cycles (every 15 minutes) with a 5 minute buffer to avoid false-positives.
The update time of the index is checked when queriers perform queries so as soon as it exceeds the 1 hour max age, queries begin to fail. Thus we need to alert on the age of the index before it begins to fail queries.
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.