Native compilation failed with KafkaStreams & Avro Serdes since version of kafka-streams-avro-serde > 7.3.3

[davevano]

Describe the bug

Context:
Quarkus application uses the Quarkus Kafka Streams extension, the Quarkus Conluent Schema Registry Avro extension and the Confluent dependency kafka-streams-avro-serde.
The application contains an avro schema (Test.avsc in the resource/avro folder)
The maven avro plugin generate a Java class from the avro schema
The dependency kafka-srteams-avro-serde allows to serialize/deserialize the Avro with this code:

` @ConfigProperty(name = "schema-registry")
String schemaRegistry;

@Produces
public Topology buildTopology() {

    Map<String, String> cfg = new HashMap<>();
    cfg.put("schema.registry.url", schemaRegistry);

    SpecificAvroSerde<AvroTest> avroTestSerde = new SpecificAvroSerde<>();
    avroTestSerde.configure(cfg, false);

    StreamsBuilder sb = new StreamsBuilder();

    sb
            .stream("test-in", Consumed.with(Serdes.String(), avroTestSerde))
            .to("test-out", Produced.with(Serdes.String(), avroTestSerde));

    return sb.build();
}`

The application interacts with Confluent Kafka & Confluent Schema Registry

It works fine in java mode, was working fine with the version 7.3.3 of the kafka-streams-avro-serde dependency.
Since the upgrade to the version 7.4.0 of kafka-streams-avro-serde dependency, the native compilation failed:

Caused by: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: com.fasterxml.jackson.dataformat.csv.CsvMapper. This error is reported at image build time because class io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe$ListPropertyParser is registered for linking at image build time

Expected behavior

Was working with the previous (7.3.3) version of kafka-srteams-avro-serde.
Native compilation is no more available with Kafka Streams libraries and Avro Serdes.

Actual behavior

Full stack trace during natice compilation:

Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing com.oracle.svm.core.code.FactoryMethodHolder.AbstractKafkaSchemaSerDe$ListPropertyParser_constructor_04f3ff3892110ddc0d5dea3bafe5ce2df32b872d(generated:0)
Parsing context:
at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe$MapPropertyParser.(AbstractKafkaSchemaSerDe.java:830)
at com.oracle.svm.core.code.FactoryMethodHolder.AbstractKafkaSchemaSerDe$MapPropertyParser_constructor_82bff9e3814f5819ba8f5fef92601f65f4b649c7(generated:0)
at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.configureClientProperties(AbstractKafkaSchemaSerDe.java:175)
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.configure(AbstractKafkaAvroDeserializer.java:111)
at io.confluent.kafka.serializers.KafkaAvroDeserializer.(KafkaAvroDeserializer.java:97)
at com.oracle.svm.core.code.FactoryMethodHolder.KafkaAvroDeserializer_constructor_6f4b51cce94ab7ce47f1c6aa7839267465e14e52(generated:0)
at static root method.(Unknown Source)

            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:149)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:178)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:152)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:110)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultStaticInvokeTypeFlow.lambda$update$0(DefaultStaticInvokeTypeFlow.java:67)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.LightImmutableCollection.forEach(LightImmutableCollection.java:90)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultStaticInvokeTypeFlow.update(DefaultStaticInvokeTypeFlow.java:66)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$1.run(PointsToAnalysis.java:474)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
            at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
            at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
            at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
            at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
            at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: com.fasterxml.jackson.dataformat.csv.CsvMapper. This error is reported at image build time because class io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe$ListPropertyParser is registered for linking at image build time by command line
at parsing io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe$ListPropertyParser.(AbstractKafkaSchemaSerDe.java:794)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.throwParserError(BytecodeParser.java:2536)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.throwParserError(SharedGraphBuilderPhase.java:169)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3414)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.iterateBytecodesForBlock(SharedGraphBuilderPhase.java:712)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.handleBytecodeBlock(BytecodeParser.java:3366)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3208)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1134)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.build(SharedGraphBuilderPhase.java:152)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1026)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:97)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:114)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:434)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:146)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.parseGraph(AnalysisMethod.java:819)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:784)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:767)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.lookupEncodedGraph(InlineBeforeAnalysisGraphDecoder.java:120)
at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.doInline(PEGraphDecoder.java:1190)
at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.tryInline(PEGraphDecoder.java:1173)
at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.trySimplifyInvoke(PEGraphDecoder.java:1028)
at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.handleInvoke(PEGraphDecoder.java:982)
at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.processNextNode(GraphDecoder.java:871)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.processNextNode(InlineBeforeAnalysisGraphDecoder.java:186)
at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.decode(GraphDecoder.java:600)
at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.decode(PEGraphDecoder.java:854)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysis.decodeGraph(InlineBeforeAnalysis.java:77)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:193)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:583)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:165)
... 14 more
Caused by: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: com.fasterxml.jackson.dataformat.csv.CsvMapper. This error is reported at image build time because class io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe$ListPropertyParser is registered for linking at image build time by command line
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.reportUnresolvedElement(SharedGraphBuilderPhase.java:521)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.reportUnresolvedElement(SharedGraphBuilderPhase.java:515)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.handleUnresolvedType(SharedGraphBuilderPhase.java:407)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.handleUnresolvedNewInstance(SharedGraphBuilderPhase.java:263)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genNewInstance(BytecodeParser.java:4531)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genNewInstance(BytecodeParser.java:4524)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5334)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3406)
... 43 more

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

source code is available here:
https://github.com/davevano/quarkus-kafka-streams.git

Simply launch native compilation
mvn install package -Pnative -DskipTests

[quarkus-bot]

/cc @alesj (kafka,kafka-streams), @cescoffier (kafka), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @rquinio (kafka-streams)

[ozangunalp]

Please add an explicit dependency to com.fasterxml.jackson.dataformat:jackson-dataformat-csv in your project.

In version 7.4.0 kafka-schema-serializer added a code dependency to Jackson CsvSchema. jackson-dataformat-csv is transitively provided by the kafka server, but I think confluent ignored that jackson csv dependency doesn't need to be present in Kafka client applications.

You don't have the issue in jvm mode because csv schema is only used with the config use.latest.with.metadata. But the native-image compiler explores all code paths so tries to load the csv schema classes.

[davevano]

Please add an explicit dependency to com.fasterxml.jackson.dataformat:jackson-dataformat-csv in your project.

In version 7.4.0 kafka-schema-serializer added a code dependency to Jackson CsvSchema. jackson-dataformat-csv is transitively provided by the kafka server, but I think confluent ignored that jackson csv dependency doesn't need to be present in Kafka client applications.

You don't have the issue in jvm mode because csv schema is only used with the config use.latest.with.metadata. But the native-image compiler explores all code paths so tries to load the csv schema classes.

Working fine, thanks!