Aggregate offsets and commit before poll #862
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ennru
commented
Aug 14, 2019
| val replyTo = sender() | ||
| commit(offsets, replyTo ! _) | ||
| commitAggregate ++= offsets | ||
| commitSenders = commitSenders :+ sender() |
There was a problem hiding this comment.
A thing to consider would be to request an extra poll by setting up a "delayed poll in flight" as the RequestMessages do.
|
Pushed an update so that the order in which the offsets are received doesn't matter anymore. With this we can increase the parallelism in the Committer and switch to |
2m
approved these changes
Aug 16, 2019
|
Failure was #832 |
This was referenced Aug 19, 2019
Closed
This was referenced Aug 30, 2019
Merged
ennru
added a commit
to ennru/alpakka-kafka
that referenced
this pull request
Nov 21, 2019
seglo
pushed a commit
that referenced
this pull request
Nov 21, 2019
* Commit refreshing: fix bug introduced #862 * Run RetentionPeriodSpec against Kafka 2.0.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Reduce calls to
consumer.commitAsyncby aggregating commits in the actor and sending them just before polling.References
Idea explained in #849
Implementation partly stolen from #851
Changes
The almost unchanged tests indicate that this doesn't change the behaviour a lot.
This change may make a commit wait until the next poll (default 50 ms). To fully make use of this, the committer defaults should be changed.