Experimental support for more scalable coalesce #333
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| service/host combination that passes through it (limited by :ttl). Every time | ||
| it receives an event, it passes on *all* events it remembers. When events | ||
| service/host combination that passes through it (limited by :ttl). Every second, | ||
| it passes on *all* events it remembers. When events | ||
| expire, they are included in the emitted sequence of events *once*, and | ||
| removed from the state table thereafter. | ||
|
|
||
| Use coalesce to combine states that arrive at different times--for instance, | ||
| to average the CPU use over several hosts." | ||
| [& children] |
There was a problem hiding this comment.
Let's have an optional number dt for the first argument; specifying how often to send coalesced events downstream.
|
I think the only thing left to to lock around the "flushing" function--I'm not sure which function you're referring to, though. |
| period-manager (periodically-until-expired dt callback)] | ||
| (fn [e] | ||
| (.put chm [(:service e) (:host e)] e) | ||
| (call-rescue e [period-manager])))) |
There was a problem hiding this comment.
I think you can just (period-manager e) here.
This commit comes from work we did to make it possible to coalesce 50k events/second, so that riemann can be used to compute real-time aggregations of stats on large clusters.
|
I have pushed a candidate for merging--there is just one commit with a message explaining the motivation. |
|
Let me know what else I can provide, or what other changes I should make. Thanks! |
|
Looks good, thanks @dgrnbrg! :) |
aphyr
added a commit
that referenced
this pull request
Mar 11, 2014
Experimental support for more scalable coalesce
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm opening this pull request to get your feedback on this feature--it's not ready for merging, and i'm trying to get it there. There are 2 problems I'm having now:
advance!, the time doesn't seem to be advancing for the callback that's scheduled by theevery!. Could this be due to thewith-redefsintime.controlled, or should I be reviewing my implementation? The only thing that I haven't tested is usingcall-rescueon the children rather than invoking them directly (as I have been using previously).every!task should be canceled when the callback is empty. Do you mind (stylewise) if I use theConcurrentHashMapas the monitor for guarding the creation/deletion of the task withlock?