Add support for Kafka's ConsumerRebalanceListener

[pcasaes]

Sometimes earliest and latest don't quite cut it, not to mention some interesting deliver exactly once cases as described in the Kafka: The Definitive Guide. There would be a fait bit of work cut out though. From what I can understand from the code subscription is started during configuration. The listener would have to be registered before subscription somehow. Fortunately Vert.x' Kafka client implementation already has support for this.

[cescoffier]

Can do give me a bit more details? Do you mean adding the ability to use a custom "poller" ?

[cescoffier]

I finally understand...
https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/rebalance-listener/ provides a good example.

It's definitely something we should add.

[pcasaes]

That's a good example. The Listener is always called in the first poll within the poll loop thread. It is also called whenever there's a rebalance which can be caused by an unhealthy consumer, but it can also be caused by adding/removing consumers as well as increase the number of partitions.

One example I have is a kafka backed in memory cache. The topic has a retention period but I don't want to wait for kafka to clean up the partitions. I also don't want my warm up to take too long loading up from earliest since it definitely will have data that is too old. I can use a rebalance to do an offset by timestamp look up (there's an index in kafka for that) and then set the offsets accordingly.

[cescoffier]

It should be rather simple to integrate.

[cescoffier]

So, I was thinking to integrate it as follow:

• the ConsumerRebalanceListener would be exposed as a bean, maybe @Named if needed
• if @Named is used, add an attribute to configure the name
• The Kafka Connector would retrieve the bean using the bean manager
• In Kafka Source, use the subscribe method that supports the listener, like in:

this.consumer.getDelegate().unwrap().subscribe(topic, listener);

@pcasaes fancy a PR? I can guide you.

[pcasaes]

Sounds good. I'll give it a shot.

[cescoffier]

Thanks @pcasaes, don't hesitate to ping me if you have any questions.