Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename and clarify TempoFlushesFailing -> TempoIngesterFlushesFailing #887

Merged
merged 1 commit into from
Aug 18, 2021
Merged

Rename and clarify TempoFlushesFailing -> TempoIngesterFlushesFailing #887

merged 1 commit into from
Aug 18, 2021

Conversation

yvrhdn
Copy link
Member

@yvrhdn yvrhdn commented Aug 18, 2021

What this PR does:
Clarifies the TempoFlushesFailing alert and renames it to TempoIngesterFlushesFailing.

  • The original description mentions failed compactions, but this alert can only be triggered by ingesters. It uses the tempo_ingester_failed_flushes_total metric that is created in modules/ingester/flush.go
  • The section about the compactor creating orphaned blocks seems useful, maybe we should move it to another alert?

Which issue(s) this PR fixes:
Fixes #

annanay25
annanay25 approved these changes Aug 18, 2021
View reviewed changes
Copy link
Contributor

@annanay25 annanay25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Wonder why we had the compactor stuff in the alert runbook before 🤔

This is the only place the metric is used -

tempo/modules/ingester/flush.go

Line 213 in addfa70

metricFailedFlushes.Inc()

@yvrhdn yvrhdn merged commit ca82cb8 into grafana:main Aug 18, 2021
@yvrhdn yvrhdn deleted the flushes-failing-alert branch August 18, 2021 13:09
@yvrhdn yvrhdn mentioned this pull request Sep 14, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@annanay25 annanay25 annanay25 approved these changes

@dgzlopes dgzlopes Awaiting requested review from dgzlopes

@joe-elliott joe-elliott Awaiting requested review from joe-elliott joe-elliott is a code owner

@mapno mapno Awaiting requested review from mapno mapno is a code owner

@mdisibio mdisibio Awaiting requested review from mdisibio mdisibio is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yvrhdn @annanay25