Update Kafka to 2.7.0

[cescoffier]

PR based on #14706 but extend the Kafka Streams Processor to register the Task Assignors implementation.

[gunnarmorling]

LGTM, but let's go for 2.6.1. 2.7.0 is out too, but I think waiting for a bugfix micro isn't a bad idea.

[gsmet]

Hmmmm. There's something not working very on Windows in Kafka Streams:

Caused by: java.lang.UnsupportedOperationException
61789
	at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2079)
61790
	at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:115)
61791
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:745)
61792
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:657)
61793
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:567)
61794
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer.initializeKafkaStreams(KafkaStreamsProducer.java:144)
61795
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer.<init>(KafkaStreamsProducer.java:95)
61796
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.create(KafkaStreamsProducer_Bean.zig:742)
61797
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.create(KafkaStreamsProducer_Bean.zig:762)
61798
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:96)
61799
	at io.quarkus.arc.impl.AbstractSharedContext.access$000(AbstractSharedContext.java:14)
61800
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:29)
61801
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:26)
61802
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
61803
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
61804
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:26)
61805
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.get(KafkaStreamsProducer_Bean.zig:794)
61806
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.get(KafkaStreamsProducer_Bean.zig:810)

[cescoffier]

trying with 2.7.0. If it does not work, I will revert to 2.6.0 which was passing all the tests.

Looking at the release notes, I can't see why there is new Windows file system issues.
We need 2.6+ to avoid the high load on rebalance.

[gunnarmorling]

The Windows issue is this regression https://issues.apache.org/jira/browse/KAFKA-12190 (in both 2.7.0 and 2.6.1). Seems we're between a rock and a hard place here.

[cescoffier]

Damned, I didn't see it... So, revert to 2.6.0, until we can bump?

[cescoffier]

Back to 2.6.0

[gunnarmorling]

Hum, hum. I think the CPU issue actually has more far-fetching consequences than the state store things for Kafka Streams on Windows. But it s$cks either way.

[gsmet]

Another option would be to upgrade and skip the tests on Windows until the issue is fixed.

[gunnarmorling]

Yes, I think that would be my preferred option.

…

[gsmet]

OK, let me work on that, I need an easy one before going to PTO.

[gsmet]

I pushed an update to Kafka 2.7.0, let's see how it goes.

[cescoffier]

Don't you fear that users won't be able to develop on their windows box because of this?

[gunnarmorling]

Don't you fear that users won't be able to develop on their windows box because of this?

Yes, that is a concern for sure, and perhaps it's not the right thing. Here's my rationale:

• This only affects Kafka Streams users (i.e. a subset of all Kafka users)
• The issue is very apparent, you should be able to find out about it during testing and then can mitigate it by downgrading to 2.5
• The CPU issue on the other hand potentially affects every Kafka users, and it has bigger odds that you only notice it in prod, when it's too late

Hence my inclination to accept that Kafka Streams issue. A third alternative would be sitting it out and waiting for 2.6.2 perhaps?

[cescoffier]

Yes, updating to 2.6.2 as soon as it's available will be the right thing to do.
I'm ok with explaining the issue to windows users for now, as it's temporary.

[cescoffier]

@gsmet Look like 2.7.0 requires a Scala bump/adjustement...

2021-02-02T14:10:49.4875189Z 2021-02-02 14:09:51,710 ERROR [io.qua.ver.htt.dep.dev.HttpRemoteDevClient] (Remote dev client thread) Remote dev request failed: java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$4

[gsmet]

Hmmm, that's weird. I bumped Scala and it was working locally. I'll have a closer look.

[gsmet]

OK, so AFAICS, Debezium is still incompatible with Scala 2.13:

Caused by: java.lang.InstantiationError: scala.collection.mutable.ArraySeq
	at io.debezium.kafka.KafkaServer.startup(KafkaServer.java:218)
	at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)
	at io.debezium.kafka.KafkaCluster.startup(KafkaCluster.java:250)
	at io.quarkus.it.kafka.KafkaTestResource.start(KafkaTestResource.java:28)
	at io.quarkus.test.common.TestResourceManager.lambda$start$1(TestResourceManager.java:95)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

so we will have to keep 2.12 for now.

I have a branch ready for the upgrade to 2.13.

[gsmet]

OK, the Scala Maven plugin situation is not very good and fixing the properties ends up breaking the codestarts. I reintroduced the duplicated property for now as I have no idea if we can break the compatibility on this.

[cescoffier]

@gsmet looks ok now (restarted the CI because of an unrelated transient network issue)

[cescoffier]

ping @gsmet ?

[gwenneg]

Any news about this update? We (the Insights team) would like to use the PEM file support for private keys that comes with Kafka 2.7.0.

[gsmet]

I wasn't sure people wanted an upgrade to a .0 of Kafka, that's why I didn't push for it. I'm rebasing the PR and will push soon.

[gwenneg]

Well I dont know much about which Kafka version should be used or not, but the previous version was also a .0 so I suppose this update is ok. Thanks @gsmet!