feat(spool): Introduce in-memory EnvelopeBuffer as alternative to the old spooler
#3863
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iambriccardo
changed the title
EnvelopeBuffer as alternative to the old spooler
iambriccardo
approved these changes
Jul 31, 2024
loewenheim
reviewed
Jul 31, 2024
Comment on lines
42
to
45
| /// Used to notify callers of `peek()` of any changes in the buffer. | ||
| notify: tokio::sync::Notify, | ||
| /// Used to notify callers of `peek()` of any changes in the buffer. | ||
| changed: AtomicBool, |
There was a problem hiding this comment.
These fields have the same documentation, is that intentional?
There was a problem hiding this comment.
Yes, both are used for notification. I didn't find a good way to make it work with only a Notify.
Co-authored-by: Sebastian Zivota <[email protected]>
Add a gauge metric for the total number of stacks in the priority queue.
jjbayer
added a commit
to getsentry/sentry
that referenced
this pull request
Aug 1, 2024
`test_histogram_outliers` became flaky after getsentry/relay#3863. Add info to the assert to get more insights.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new envelope buffer as designed in https://github.com/getsentry/team-ingest/issues/345, though which all envelopes flow before being processed. Buffered envelopes are stored in per-project stacks. Stacks are sorted by whether the project configs needed for processing are ready, and by latest receive time, such that the latest envelope with a valid project config is always the first in line to be processed.
Currently, only memory-based envelope stacks are supported, but this PR already contains the implementation for disk-based stacks as well.
Class Diagram (partial)
classDiagram RequestHandler --> "writes to" GuardedEnvelopeBuffer RequestHandler --> "requests update" ProjectCache ProjectCache --> "notifies" GuardedEnvelopeBuffer GuardedEnvelopeBuffer "1" *-- "1" PolymorphicEnvelopeBuffer class PolymorphicEnvelopeBuffer { << enum >> } PolymorphicEnvelopeBuffer <|-- EnvelopeBuffer~StackProvider~ class EnvelopeBuffer { + push(envelope) + pop() Option~Envelope~ + peek() Option~&Envelope~ + mark_ready(ProjectKey) + mark_pending(ProjectKey) } EnvelopeBuffer "1" *-- "1" StackProvider EnvelopeBuffer "1" *-- "1" PriorityQueue PriorityQueue "1" *-- "n" EnvelopeStack class StackProvider { << interface >> + create_stack() } StackProvider <|-- MemoryStackProvider StackProvider <|-- SqliteStackProvider StackProvider --> "creates" EnvelopeStack class EnvelopeStack { <<interface>> + push(envelope) + pop() optional Envelope } EnvelopeStack <|-- MemoryEnvelopeStack EnvelopeStack <|-- SqliteEnvelopeStackref: https://github.com/getsentry/team-ingest/issues/345