Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow user to commit offsets without handling response when using committable sources #845

Closed
seglo opened this issue Jul 3, 2019 · 2 comments
Milestone

Comments

@seglo
Copy link
Contributor

seglo commented Jul 3, 2019

Short description

Allow the user to commit offsets back to Kafka asynchronously without waiting for a response.

Details

The Committer.batchFlow will use mayAsync to backpressure offset commit callbacks. When we call commitScalaDsl we will use the Ask pattern to ask the KafkaConsumerActor to commit the offset map immediately (using KafkaConsumer.commitAsync) and reply with a response when the commit response is handled during a subsequent poll. This implementation has the benefit of making sure we lose as few messages as possible in the event of a failure while using at least once message delivery, but it can also considerably slow down the whole consuming stream. A compromise would be to commit and not wait for a reply which would allow the stream to process records faster, with the tradeoff of potentially reprocessing more records after a failure or rebalance.

It would be nice if there was a way to "fire and forget" the commit so we don't back-pressure in the Committer.batchFlow waiting for replies. We also wouldn't create the overhead of additional callback threads.

@seglo
Copy link
Contributor Author

seglo commented Jul 4, 2019

FYI I'm going to pick this up!

@ennru
Copy link
Member

ennru commented Sep 5, 2019

Implemented with #883

@ennru ennru closed this as completed Sep 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants