Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include distributor queue length in tempo-mixin #2623

Merged
merged 5 commits into from
Oct 16, 2023

Conversation

zalegrala
Copy link
Contributor

What this PR does:

Include the distributor queue length in the write path row of the tempo operational dashboard.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@zalegrala
Copy link
Contributor Author

image

"uid": "cortex-ops-01"
},
"editorMode": "code",
"expr": "sum(tempo_distributor_queue_length{cluster=~\"$cluster\", namespace=~\"$namespace\", job=~\".*/distributor\"}) by (pod)\n",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should aggregate by name as well. There can be multiple queues in distributors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking again, I think you're right, but what about aggregate by name only? Then we get a sum of the length across all instances, which might clarify a little.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Copy link
Member

@mapno mapno Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's about preference. Either sound good to me. I'll leave to you to decide.

@zalegrala zalegrala force-pushed the distributorQueueLengthPanel branch from 5cc3463 to 3303831 Compare August 30, 2023 15:08
@carlosjgp
Copy link

There are some errors in the mixin files, I think

No newline at end of file

This is a great feature to get, thank you!

@zalegrala zalegrala merged commit 83d3835 into grafana:main Oct 16, 2023
@zalegrala zalegrala deleted the distributorQueueLengthPanel branch October 16, 2023 15:07
yvrhdn pushed a commit that referenced this pull request Oct 16, 2023
* Update `make docs` procedure (#3026)

* Update `make docs` procedure

* Trigger CI

---------

Co-authored-by: grafanabot <[email protected]>
Co-authored-by: Jack Baldry <[email protected]>

* Fix: pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions (#3012)

* fix: handle case when userconfigurableoverrides are enabled but empty

* Add CHANGELOG.md

* fix panic in test

* fix GetTargetInfoExcludedDimens & GetFilterPolicies

* update changelog

* Include distributor queue length in tempo-mixin (#2623)

* Include distributor queue length in tempo-mixin

* Rebuild

* Compile jsonnet

* Compile jsonnet

* Use C jsonnet to format mixin

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: grafanabot <[email protected]>
Co-authored-by: Jack Baldry <[email protected]>
Co-authored-by: Suraj Nath <[email protected]>
Co-authored-by: Zach Leslie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants