Kafka: Add support to configure heartbeat interval and session timeout to kafka's consumer

[denisbchrsk]

Description

Added the option to configure heartbeat interval and session timeout in Kafka Pubsub and Kafka Binding - relevant for certain use cases where the sidecar might not be able to consistently send heartbeats to the Kafka brokers, or real time cases where you need the sidecar to send heartbeats more frequently than sarama's default values.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: Add Kafka heartbeat interval and session timeout to component reference docs#4074

[ItalyPaleAle]

Thanks for this PR :)

It looks similar to #3371, can you confirm? CC: @sicoyle

[denisbchrsk]

Thanks for this PR :)

It looks similar to #3371, can you confirm? CC: @sicoyle

It definitely looks very similar, but if I'm not mistaken, #3371 reinforces the network-level connection to the broker, vs a new-initialized consumer in the consumer group defining a SLA to the broker; if no heartbeat occurred after sessionTimeout, then the consumer instance is removed from the consumer group, and the broker rebalances the consumer group and re-assigns the partitions (Kafka-specific configuration, there's a better explanation in confluent's docs https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#session-timeout-ms).

[sicoyle]

Yes, I think we will want the network-level connection with the broker in my PR you've linked. My PR gets at keeping the sidecar with a constant network connection to the broker which is what you want in the event a sidecar becomes idle in the eyes of the broker.

[denisbchrsk]

Thanks @sicoyle for the confirmation, to summarize the differences in (hopefully) simpler terms from my previous comment:

• The PR in feat(kafka): add producer config capabilities for connections #3371 helps with the network-level connection from the sidecar to the broker, and keeping it alive (dealing with sidecar inactivity).
• This PR helps with defining the SLA from the sidecar to the broker specifically when consuming from the topic; if the broker is not accessible consistently within the default values that sarama has for the consumer / the sidecar dies, then this PR allows the user to configure the relevant values (heartbeatInterval / sessionTimeout) for the aforementioned consumer instance SLA (this is a known Kafka-specific configuration). With these changes, the user may be able to control this SLA, and may reduce unwarranted rebalances or increase warranted rebalances in the consumer group.

Both of these PRs are relevant in their own domain and are different.

[denisbchrsk]

Hey @ItalyPaleAle, is it possible to take a look at this again? I've explained in the previous comment why it's completely different from #3371

[ItalyPaleAle]

Hey @ItalyPaleAle, is it possible to take a look at this again? I've explained in the previous comment why it's completely different from #3371

Yes, I think this is helpful. I've just merged #3371 and now there are some merge conflicts. Could you please take a look? I'll review after this is ready

[denisbchrsk]

Fixed the merge conflicts, it's ready for review @ItalyPaleAle

[ItalyPaleAle]

LGTM thanks!

Please don't forget to open a PR (or at least an issue) in dapr/docs to have the new options documented!