Quarkus native build fails with Quarkus CXF WS-Security

[adamov8]

While having the dependency quarkus-cxf-rt-ws-security on classpath, quarkus native build fails with
UnsupportedFeatureException: No instances of io.netty.buffer.PooledByteBufAllocator are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=io.netty.buffer.PooledByteBufAllocator.

This happens on any sample project created using:

• Quarkus version: 2.15.3.Final (latest to date, also applies to any version above 2.12.0.Final AFAIK) with Java 11
• quarkus-cxf-rt-ws-security version: 1.7.3

Steps to reproduce:

1. create project with CXF dependencies including Quarkus CXF WS-Security: https://code.quarkus.io/?a=native-build-test&j=11&e=io.quarkiverse.cxf%3Aquarkus-cxf&e=io.quarkiverse.cxf%3Aquarkus-cxf-rt-ws-security&extension-search=cxf
2. run ./mvnw package -Pnative -Dquarkus.native.container-build=true (Docker required as this is a container-build)

Things that I tried:

• log suggested --trace-object-instantiation to find root cause
• --initialize-at-run-time native build arg with some possible culprits
• --initialize-at-build-time native build arg with some possible culprits

Any help/suggestions appreciated.

[mecsafonso]

Facing the same problem here.

Quarkus version: 2.16.0.Final with Java 11
quarkus-cxf-rt-ws-security version: 1.7.3

I updated the quarkus version because the cxf extension was tested against Quarkus 2.16.0.Final (https://quarkiverse.github.io/quarkiverse-docs/quarkus-cxf/dev/user-guide/create-project.html#_which_quarkus_platform_versions_are_compatible_with_a_given_quarkus_cxf_bom_version)

[froilson]

Facing the same problem here.

I test it with:
Quarkus version: 2.15.3.Final with Java 17
quarkus-cxf-rt-ws-security version: 1.7.3

Quarkus version: 2.16.0.Final with Java 17
quarkus-cxf-rt-ws-security version: 1.7.3

Quarkus version: 2.15.3.Final with Java 17
quarkus-cxf-rt-ws-security version: 1.5.0

All That versions don't passed the native compilation with the same error message that adamov9 said.

I test it with:
Quarkus version: 2.15.3.Final with Java 17
quarkus-cxf-rt-ws-security version: 1.4.0

This compile natively successfuly with no errors but with the workaround explain in this issue #456.

[shumonsharif]

I believe these errors may be due to the org.ehcache.shadow.org.terracotta.utilities.io.Files being repackaged to org.terracotta.utilities.io.Files in the version of ehcache we depend on.

Would any of you be able to run a small test for us - essentially, you would need to build the quarkus-cxf project locally, replacing the lines here with the code below. You can download the relevant source code from the releases page.

    @BuildStep
    void runtimeInitializedClass(BuildProducer<RuntimeInitializedClassBuildItem> runtimeInitializedClass) {
        Stream.of(
                "org.ehcache.sizeof.impl.JvmInformation",
                //"org.ehcache.shadow.org.terracotta.utilities.io.Files",
                "org.terracotta.utilities.io.Files",
                "org.ehcache.shadow.org.terracotta.offheapstore.util.PhysicalMemory",
                "org.ehcache.shadow.org.terracotta.offheapstore.paging.UpfrontAllocatingPageSource",
                "org.ehcache.shadow.org.terracotta.offheapstore.storage.portability.BooleanPortability",
                "org.ehcache.xml.XmlConfiguration",
                "org.ehcache.xml.ResourceConfigurationParser")
                .map(RuntimeInitializedClassBuildItem::new)
                .forEach(runtimeInitializedClass::produce);
    }

Let me know if any questions on the above.

[adamov8]

I believe these errors may be due to the org.ehcache.shadow.org.terracotta.utilities.io.Files being repackaged to org.terracotta.utilities.io.Files in the version of ehcache we depend on.

Would any of you be able to run a small test for us - essentially, you would need to build the quarkus-cxf project locally, replacing the lines here with the code below. You can download the relevant source code from the releases page.

    @BuildStep
    void runtimeInitializedClass(BuildProducer<RuntimeInitializedClassBuildItem> runtimeInitializedClass) {
        Stream.of(
                "org.ehcache.sizeof.impl.JvmInformation",
                //"org.ehcache.shadow.org.terracotta.utilities.io.Files",
                "org.terracotta.utilities.io.Files",
                "org.ehcache.shadow.org.terracotta.offheapstore.util.PhysicalMemory",
                "org.ehcache.shadow.org.terracotta.offheapstore.paging.UpfrontAllocatingPageSource",
                "org.ehcache.shadow.org.terracotta.offheapstore.storage.portability.BooleanPortability",
                "org.ehcache.xml.XmlConfiguration",
                "org.ehcache.xml.ResourceConfigurationParser")
                .map(RuntimeInitializedClassBuildItem::new)
                .forEach(runtimeInitializedClass::produce);
    }

Let me know if any questions on the above.

Thank you @shumonsharif for looking into this.

I tried it with a dummy project as described in my original post and it seems like the issue with the image build is solved. 🎉
Right now I can not carry out any functional tests but as quarkus-cxf build's tests ran successfully I assume all is good.

Any chance of including the fix in an upcoming release?

[shumonsharif]

Thanks @adamov8 for your help; I couldn't recreate the issue exactly as you all described (similar errors, but different classes ending up on the image heap for me) - so I wanted you to confirm the fix.

I'm not yet sure how the existing quarkus-cxf code passed the build / integration tests, I think it should have failed. I noticed that 1.7.2 was the last successful execution of a build relating to a release: https://github.com/quarkiverse/quarkus-cxf/actions/runs/3731967005. The build job seems to be getting skipped entirely since 1.7.3: https://github.com/quarkiverse/quarkus-cxf/actions/runs/3930187089.

@ppalaga @famod Could you please advise; is the build skipping during release by design, or an issue that needs to be resolved? I recall we had some build issues at some point that we were trying to fix, but don't remember if it's the same. Also, pls share any thoughts on possibly releasing a fix for this issue, which branches should we target?

EDIT 1: I just came across this: 4e379f9 So it looks like the builds should indeed be getting skipped for release PRs.

EDIT 2: I'm honestly a little stumped as to how all the quarkus-cxf CI builds are passing without issue. For example here is one of the more recent successful builds: https://github.com/quarkiverse/quarkus-cxf/actions/runs/4193564013/jobs/7270688235#step:5:438. Main difference I have observed is that ./mvnw package -Pnative -Dquarkus.native.container-build=true uses a different native builder container image on my Mac (GraalVM CE) versus on the Linux based CI builds (Mandrel) - though I don't believe that is in any way contributing to this issue.

[shumonsharif]

Some good news ... I think I finally identified the root cause for this issue. The quarkus-cxf BOM is not used in https://code.quarkus.io generated projects. This results in these generated projects using the incorrect version of ehcache.

@adamov8 Can you please try the workaround below and let us know if it resolves your issue?

Add the quarkus-cxf-bom dependency:

  <dependencyManagement>
    <dependencies>
      ....
      <dependency>
        <groupId>io.quarkiverse.cxf</groupId>
        <artifactId>quarkus-cxf-bom</artifactId>
        <version>1.7.3</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Remove all the version elements for the quarkus-cxf dependencies:

    <dependency>
      <groupId>io.quarkiverse.cxf</groupId>
      <artifactId>quarkus-cxf-rt-ws-security</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkiverse.cxf</groupId>
      <artifactId>quarkus-cxf</artifactId>
    </dependency>

Depending on the version of quarkus-cxf you're using, you may also need to add the following in your application.properties file:

quarkus.native.additional-build-args=--initialize-at-run-time=org.ehcache.shadow.org.terracotta.offheapstore.util.PhysicalMemory

[adamov8]

Thanks @adamov8 for your help; I couldn't recreate the issue exactly as you all described (similar errors, but different classes ending up on the image heap for me) - so I wanted you to confirm the fix.
...

I can't reproduce the exact error either, my hunch is that this is because of the Mandrel image being used - the floating tag might point to a newer version of the underlying GraalVM.

@adamov8 Can you please try the workaround below and let us know if it resolves your issue?
...

Just for the record, as of the time of testing I have version GraalVM 22.3.1.0-Final Java 17 Mandrel Distribution logged when using container build with Quarkus version 2.15.3.Final.

All works fine using the BOM, did not have to use quarkus.native.additional-build-args.

[shumonsharif]

All works fine using the BOM, did not have to use quarkus.native.additional-build-args.

Thank you so much @adamov8 for your help once again!

I will defer to @ppalaga and @famod on how to best handle this issue. I think we may need to explicitly state somewhere that the BOM should be used instead of version tags for the quarkus-cxf dependencies. We also have an open issue around the BOM inclusion for the generated projects: #651.

[famod]

I will defer to @ppalaga and @famod on how to best handle this issue. I think we may need to explicitly state somewhere that the BOM should be used instead of version tags for the quarkus-cxf dependencies.

It's already there, no? https://quarkiverse.github.io/quarkiverse-docs/quarkus-cxf/dev/user-guide/create-project.html#dependency-management

[shumonsharif]

Thanks @famod! I think the doc you referenced is very good, but may possibly not be as obvious to get to, for folks generating their projects from https://code.quarkus.io. Would it be worth including the same content here: https://github.com/quarkiverse/quarkus-cxf#configuration? Or maybe even a codestart which simply includes a README file that gets bundled with the generated projects? Not sure if there really is a good solution for this, just throwing out some ideas.