Kafka consumer consumes all CPU when shutting down the broker

[gunnarmorling]

With SmallRye Reactive Messaging Kafka, when shutting down the broker, an application consuming topics will spike up to 100% CPU usage.

// CC @cescoffier

[cescoffier]

Do you have a reproducer? That's weird...

[gsmet]

@gunnarmorling if you don't have a reproducer, maybe a thread dump would help?

[gunnarmorling]

One reproducer, albeit not exactly minimal is here: https://github.com/debezium/debezium-examples/tree/master/distributed-caching#running-the-quarkus-services-locally-in-dev-mode. Make sure to adjust the exposed host name in the Compose file as described; then:

export DEBEZIUM_VERSION=1.4
docker-compose up --build --scale order-service=0 --scale cache-update-service=0 --scale pgadmin=0

# wait until everything is up
mvn compile quarkus:dev -f cache-update-service/pom.xml

Then stop the broker (docker-compose down), and CPU gets usurpated.

[cescoffier]

I believe I know what's happening. The last time I checked it was not possible to configure the Kafka Network Client, and it was retrying to connect without any delay. I will have another look this afternoon.

[cescoffier]

Fixed upstream. (keeping it opened as I will close it when we update Quarkus with SmallRye Reactive Messaging 2.8.0)

[cescoffier]

@tsegismont investigated, and it seems to be a Kafka Client 2.5 issue. It does behave correctly with 2.6.

[cescoffier]

For the record: https://issues.apache.org/jira/browse/KAFKA-10134.

[gunnarmorling]

Wow, ok. That's quite something. Happy it got fixed.

…