Fix MimirIngesterHasNotShippedBlocks alert #5396
Merged
+109
−162
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pracucci
commented
Jul 1, 2023
pracucci
force-pushed
the
fix-shipper-metrics
branch
from
00eca56 to
c95a23f
Compare
|
Thanks @krajorama for your review. However, after your review I changed the metric name to keep it consistent with the other shipper metrics (so having |
pstibrany
reviewed
Jul 4, 2023
...metamonitoring-values-generated/mimir-distributed/templates/metamonitoring/mixin-alerts.yaml
Outdated
Show resolved
Hide resolved
pstibrany
reviewed
Jul 4, 2023
pkg/ingester/shipper.go
Outdated
| Help: "Total number of TSDB block upload failures", | ||
| }), | ||
| lastSuccessfulUploadTime: promauto.With(reg).NewGauge(prometheus.GaugeOpts{ | ||
| Name: "cortex_ingester_shipper_last_successful_upload_time", |
There was a problem hiding this comment.
_timestamp_seconds is recommended suffix for metric like this.
There was a problem hiding this comment.
Addressed in e266481
Signed-off-by: Marco Pracucci <[email protected]>
…_ingester_shipper_last_successful_upload_timestamp_seconds Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
fix-shipper-metrics
branch
from
c95a23f to
e266481
Compare
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-5396-to-r245 origin/r245
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 7d9b3d42468c6f39aad236dbb7093895580400e6
# Push it to GitHub
git push --set-upstream origin backport-5396-to-r245
git switch main
# Remove the local backport branch
git branch -D backport-5396-to-r245Then, create a pull request where the |
aknuds1
pushed a commit
that referenced
this pull request
Jul 5, 2023
* Fix MimirIngesterHasNotShippedBlocks alert Signed-off-by: Marco Pracucci <[email protected]> * Renamed cortex_ingester_shipper_last_successful_upload_time to cortex_ingester_shipper_last_successful_upload_timestamp_seconds Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> (cherry picked from commit 7d9b3d4)
3 tasks
aknuds1
added a commit
that referenced
this pull request
Jul 5, 2023
* Fix MimirIngesterHasNotShippedBlocks alert * Renamed cortex_ingester_shipper_last_successful_upload_time to cortex_ingester_shipper_last_successful_upload_timestamp_seconds --------- Signed-off-by: Marco Pracucci <[email protected]> (cherry picked from commit 7d9b3d4) Co-authored-by: Marco Pracucci <[email protected]>
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Earlier today we've got an issue for which I was expecting
MimirIngesterHasNotShippedBlocksto fire, but it didn't. Turns out the reason is that the metricthanos_shipper_last_successful_upload_timeis not exported anymore (it's tracked in the code, but not mapped intsdbMetrics). This PR fixes it.User-facing changes:
cortex_ingester_shipper_last_successful_upload_timestamp_secondscortex_ingester_shipper_dir_syncs_totalbecause useless (never used in our mixing)cortex_ingester_shipper_dir_sync_failures_totalbecause it was never incremented (never used in our mixin)Note to reviewers:
metricsstruct has been renamed toshipperMetricsand is now shared across all shipper instances. The metrics tracked are safe to be shared because they're either counters (safe) orcortex_ingester_shipper_last_successful_upload_timestamp_seconds(gauge) which when updated is always set to current time (so we effectively track the "last successful upload time").Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]