Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.1: Report ConsumeWorkerMetrics on 20ms interval (backport of #4708) #4722

Merged
merged 1 commit into from
Feb 3, 2025

Conversation

mergify[bot]
Copy link

@mergify mergify bot commented Jan 31, 2025

Problem

  • 1s interval is too large to report worker metrics
  • reporting on a 20s interval, many times per slot, gives us a picture of when and how things are processed as our leader slots progress

Summary of Changes

  • Report ConsumerWorkerMetrics on a 20ms interval
  • Gives us really great charts like this, showing how quickly we're able to fill blocks:
    image

Fixes #


This is an automatic backport of pull request #4708 done by [Mergify](https://mergify.com).

@mergify mergify bot requested a review from a team as a code owner January 31, 2025 13:59
@t-nelson
Copy link

the idea here is to dos influx?

@alessandrod
Copy link

the idea here is to dos influx?

If you won't shut it down, we will.

Metrics are batched so it's not like this does a POST every 20ms. The number of requests stays the same, the payload grows slightly larger. The metric is only reported when the workers have done work, so only by leaders during their leader slots.

Copy link

@t-nelson t-nelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the payload grows slightly larger

50x? welcome to the devops team!

@alessandrod alessandrod merged commit 5b47e36 into v2.1 Feb 3, 2025
28 checks passed
@alessandrod alessandrod deleted the mergify/bp/v2.1/pr-4708 branch February 3, 2025 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants