Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge prometheus.write.queue into main. #1564

Merged
merged 13 commits into from
Oct 16, 2024
Merged

Merge prometheus.write.queue into main. #1564

merged 13 commits into from
Oct 16, 2024

Conversation

mattdurham
Copy link
Collaborator

@mattdurham mattdurham commented Aug 27, 2024

Ready for merge into main, there have only been minor changes to the name, changelog, and two tests that were flaky since the component pr.

Fixes #370

@mattdurham mattdurham marked this pull request as draft August 27, 2024 18:22
mattdurham and others added 11 commits August 27, 2024 14:29
* Checkin for file queue

* add comment

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* naming and error handling feedback from PR

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* drop benchmark

* rename get to pop

---------

Co-authored-by: Piotr <[email protected]>
* Adding the serialization features.

* Dont test this with race condition since we access vars directly.

* Fix test.

* Fix typo in file name and return early in DeserializeToSeriesGroup.

* Update internal/component/prometheus/remote/queue/serialization/appender.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/serialization/serializer.go

Co-authored-by: Piotr <[email protected]>

* Rename to indicate that TimeSeries are Put/Get from a pool.

* Remove func that was about the same number of lines as inlining.

* Update internal/component/prometheus/remote/queue/types/serialization.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/serialization/serializer.go

Co-authored-by: Piotr <[email protected]>

* Change benchmark to be more specific.

---------

Co-authored-by: Piotr <[email protected]>
* Checkin the networking items.

* Fix for config updating and tests.

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: William Dumont <[email protected]>

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: Piotr <[email protected]>

* pr feedback

* pr feedback

* simplify stats

* PR feedback

---------

Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Piotr <[email protected]>
* Checkin the networking items.

* Fix for config updating and tests.

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: William Dumont <[email protected]>

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: Piotr <[email protected]>

* pr feedback

* pr feedback

* simplify stats

* simplify stats

* Initial push.

* docs and some renaming

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Changes and testing.

* Update docs.

* Update docs.

* Fix race conditions in unit tests.

* Tweaking unit tests.

* lower threshold more.

* lower threshold more.

* Fix deadlock in manager tests.

* rollback to previous

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Paulin Todev <[email protected]>

* Docs PR feedback

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Piotr <[email protected]>

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* Fix typo

* Fix typo

* Fix bug.

* Fix docs

---------

Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Piotr <[email protected]>
Co-authored-by: Clayton Cornell <[email protected]>
Co-authored-by: Paulin Todev <[email protected]>
@mattdurham
Copy link
Collaborator Author

Only significant change is changelog and renaming from prometheus.remote.queue to prometheus.write.queue.

@mattdurham mattdurham changed the title WIP: Dev branch for queue based wal Merge prometheus.write.queue into main. Oct 15, 2024
@mattdurham mattdurham marked this pull request as ready for review October 15, 2024 15:15
@mattdurham mattdurham requested review from clayton-cornell and a team as code owners October 15, 2024 15:15
Copy link
Contributor

@clayton-cornell clayton-cornell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No changes from the docs standpoint. LGTM

go.mod Outdated Show resolved Hide resolved
Copy link
Contributor

@thampiotr thampiotr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to go with it, dogfood and fix any issues we find.
Would be great to link a tracking issue with follow-up work so we can prioritise.

Copy link
Contributor

@wildum wildum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as Piotr, let's roll it and polish it along the way. Congratz again!

@mattdurham mattdurham merged commit eb1c840 into main Oct 16, 2024
17 checks passed
@mattdurham mattdurham deleted the dev.new-wal branch October 16, 2024 19:32
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Durable Queue as a WAL in Flow
5 participants