Support official Apache Kafka images in Kafka Dev Services

[nscuro]

Description

Dev Services for Kafka currently supports three providers:

• Redpanda (default)
• Strimzi
• Kafka Native

All of which require a slightly different setup:

quarkus/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java

Lines 228 to 276 in 888266f

	switch (config.provider) {
	case REDPANDA:
	RedpandaKafkaContainer redpanda = new RedpandaKafkaContainer(
	DockerImageName.parse(config.imageName).asCompatibleSubstituteFor("vectorized/redpanda"),
	config.fixedExposedPort,
	launchMode.getLaunchMode() == LaunchMode.DEVELOPMENT ? config.serviceName : null,
	useSharedNetwork, config.redpanda);
	timeout.ifPresent(redpanda::withStartupTimeout);
	redpanda.withEnv(config.containerEnv);
	redpanda.start();
	return new RunningDevService(Feature.KAFKA_CLIENT.getName(),
	redpanda.getContainerId(),
	redpanda::close,
	KAFKA_BOOTSTRAP_SERVERS, redpanda.getBootstrapServers());
	case STRIMZI:
	StrimziKafkaContainer strimzi = new StrimziKafkaContainer(config.imageName)
	.withBrokerId(1)
	.withKraft()
	.waitForRunning();
	ConfigureUtil.configureSharedNetwork(strimzi, "kafka");
	if (config.serviceName != null) {
	strimzi.withLabel(DevServicesKafkaProcessor.DEV_SERVICE_LABEL, config.serviceName);
	}
	if (config.fixedExposedPort != 0) {
	strimzi.withPort(config.fixedExposedPort);
	}
	timeout.ifPresent(strimzi::withStartupTimeout);
	strimzi.withEnv(config.containerEnv);
	strimzi.start();
	return new RunningDevService(Feature.KAFKA_CLIENT.getName(),
	strimzi.getContainerId(),
	strimzi::close,
	KAFKA_BOOTSTRAP_SERVERS, strimzi.getBootstrapServers());
	case KAFKA_NATIVE:
	KafkaNativeContainer kafkaNative = new KafkaNativeContainer(DockerImageName.parse(config.imageName),
	config.fixedExposedPort,
	launchMode.getLaunchMode() == LaunchMode.DEVELOPMENT ? config.serviceName : null,
	useSharedNetwork);
	timeout.ifPresent(kafkaNative::withStartupTimeout);
	kafkaNative.withEnv(config.containerEnv);
	kafkaNative.start();
	return new RunningDevService(Feature.KAFKA_CLIENT.getName(),
	kafkaNative.getContainerId(),
	kafkaNative::close,
	KAFKA_BOOTSTRAP_SERVERS, kafkaNative.getBootstrapServers());
	}

Since Kafka 3.7.0, the Apache foundation finally publishes official container images, in both JVM and native flavor:

• https://hub.docker.com/r/apache/kafka
• https://hub.docker.com/r/apache/kafka-native

The org.testcontainers:kafka module already supports them: https://java.testcontainers.org/modules/kafka/#using-orgtestcontainerskafkakafkacontainer

Implementation ideas

Add an additional APACHE_KAFKA provider that defaults to apache/kafka-native. Users can override the image to use the JVM-based apache/kafka image if desired.

Consider deprecating the STRIMZI and KAFKA_NATIVE providers.

[quarkus-bot]

/cc @alesj (kafka), @cescoffier (kafka), @geoand (devservices), @ozangunalp (kafka), @stuartwdouglas (devservices)

[ozangunalp]

I think we can add an additional dev service provider "Official" for upstream Kafka images.

FYI In the original kafka-native project, we've planned to support releases until Kafka 4.0 and then deprecate the project.

We'd keep the kafka-native provider for a while but not update the image.

[cescoffier]

Agreed with @ozangunalp. My tests of the official Kafka native are not very convincing. The amount of memory it uses upfront is concerning (in comparison to Ozan's variant).

We can add it as a possibility, but it won't be the default.

[nscuro]

That's fair. But for what it's worth, you can limit the heap size via KAFKA_OPTS env var:

docker run -it --rm -e "KAFKA_OPTS=-Xmx256m" apache/kafka-native:3.8.0

[cescoffier]

Sure, we can limit the memory, but something looks weird - it probably initialize too much or load too many classes. Our kafka-native does not need that.

[cescoffier]

@ozangunalp, we may need to move on with this. I'm very worried about the memory issue - in a sense, it may introduce a requirement on the host machine (that needs a lot more memory). So, I would say let's add it as another option, not the default one.