You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Allow the user to commit offsets back to Kafka asynchronously without waiting for a response.
Details
The Committer.batchFlow will use mayAsync to backpressure offset commit callbacks. When we call commitScalaDsl we will use the Ask pattern to ask the KafkaConsumerActor to commit the offset map immediately (using KafkaConsumer.commitAsync) and reply with a response when the commit response is handled during a subsequent poll. This implementation has the benefit of making sure we lose as few messages as possible in the event of a failure while using at least once message delivery, but it can also considerably slow down the whole consuming stream. A compromise would be to commit and not wait for a reply which would allow the stream to process records faster, with the tradeoff of potentially reprocessing more records after a failure or rebalance.
It would be nice if there was a way to "fire and forget" the commit so we don't back-pressure in the Committer.batchFlow waiting for replies. We also wouldn't create the overhead of additional callback threads.
The text was updated successfully, but these errors were encountered:
Short description
Allow the user to commit offsets back to Kafka asynchronously without waiting for a response.
Details
The
Committer.batchFlow
will usemayAsync
to backpressure offset commit callbacks. When we callcommitScalaDsl
we will use the Ask pattern to ask theKafkaConsumerActor
to commit the offset map immediately (usingKafkaConsumer.commitAsync
) and reply with a response when the commit response is handled during a subsequent poll. This implementation has the benefit of making sure we lose as few messages as possible in the event of a failure while using at least once message delivery, but it can also considerably slow down the whole consuming stream. A compromise would be to commit and not wait for a reply which would allow the stream to process records faster, with the tradeoff of potentially reprocessing more records after a failure or rebalance.It would be nice if there was a way to "fire and forget" the commit so we don't back-pressure in the
Committer.batchFlow
waiting for replies. We also wouldn't create the overhead of additional callback threads.The text was updated successfully, but these errors were encountered: