Netty requires more classes to be runtime initialized #37633

zakkak · 2023-12-08T23:36:58Z

In #37347 we moved
PlatformDependent and PlatformDependent0 classes to
run-time-initialization as they are platform-dependent and thus need to
be initialized on the target platform. It looks like there are more
classes like these that need to be runtime initialized as they
transitively rely on platform dependent values.

Closes Karm/mandrel-integration-tests#236

Supersedes #37628

CI run with latest Mandrel 24.0-dev https://github.com/graalvm/mandrel/actions/runs/7147836492

franz1981 · 2023-12-09T11:05:11Z

extensions/netty/deployment/src/main/java/io/quarkus/netty/deployment/NettyProcessor.java

+                .addRuntimeReinitializedClass("io.netty.buffer.UnpooledByteBufAllocator")
+                .addRuntimeReinitializedClass("io.netty.buffer.Unpooled")
+                .addRuntimeReinitializedClass("io.vertx.core.http.impl.Http1xServerResponse")
+                .addRuntimeReinitializedClass("io.netty.handler.codec.http.HttpObjectAggregator");


I am surprised, we use it? In theory Vertx should handle the whole http 1.1 stack with https://github.com/eclipse-vertx/vert.x/blob/4.x/src/main/java/io/vertx/core/http/impl/HttpServerWorker.java#L284

+1 - who's loading this?

I will have a look on Monday (along with the rest of the failing tests).

@franz1981 @cescoffier The stack trace (according to GraalVM) leading to the use of this class is:

The following detailed trace displays from which field in the code the object was reached. Detailed message: Trace: Object was reached by reading field io.netty.buffer.EmptyByteBuf.alloc of constant io.netty.buffer.EmptyByteBuf@546adc44: EmptyByteBufBE scanning root io.netty.buffer.EmptyByteBuf@546adc44: EmptyByteBufBE embedded in io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java) parsing method io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:245) reachable via the parsing context at io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:86) at io.netty.handler.codec.MessageAggregator.invokeHandleOversizedMessage(MessageAggregator.java:403) at io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:291) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.util.OptionalInt.ifPresentOrElse(OptionalInt.java:185) at root method.(Unknown Source)

hum it may have found a path that is actually not used. However, adding a substitution might be a bit overkill. So, +1 to re-initialize it.

@cescoffier FYI to our so far understanding this happens because when using --enable-monitoring=jfr (i.e. -Dquarkus.native.monitoring=jfr) GraalVM opens all packages of java.base to the unnamed module, which appears to have side-effects on the code reachability of netty.

Karm/mandrel-integration-tests#236 (comment)

In quarkusio#37347 we moved `PlatformDependent` and `PlatformDependent0` classes to run-time-initialization as they are platform-dependent and thus need to be initialized on the target platform. It looks like there are more classes like these that need to be runtime initialized as they transitively rely on platform dependent values. Closes Karm/mandrel-integration-tests#236 Supersedes quarkusio#37628

quarkus-bot · 2023-12-11T19:30:39Z

✔️ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

zakkak · 2023-12-11T19:45:57Z

I ended up adding even more classes to the set of runtime initialized ones. All native tests look good now.

jerboaa · 2023-12-12T09:33:51Z

CI run with latest Mandrel 24.0-dev https://github.com/graalvm/mandrel/actions/runs/7147836492

@zakkak Could you please trigger another run. Many of the shown failures should be gone with a later run. Thanks!

zakkak · 2023-12-12T09:42:22Z

@zakkak Could you please trigger another run. Many of the shown failures should be gone with a later run. Thanks!

I scheduled a re-run https://github.com/graalvm/mandrel/actions/runs/7147836492 (same url).

jerboaa · 2023-12-12T16:15:14Z

Mandrel IT tests are OK with this now, so +1.

`io.netty.buffer.PooledByteBufAllocator` is already registered for runtime initialization earlier in NettyProcessor.java Follow up to quarkusio#37633 Also fixes the following warning when using Mandrel and GraalVM 23.0: ``` com.oracle.svm.core.util.UserError$UserException: Incompatible change of initialization policy for io.netty.buffer.PooledByteBufAllocator: trying to change RUN_TIME from 'META-INF/native-image/io.netty/netty-buffer/native-image.properties' in 'file:///home/zakkak/code/mandrel-integration-tests/apps/jfr-native-image-performance/target/jfr-plaintext-native-image-source-jar/lib/io.netty.netty-buffer-4.1.100.Final.jar' with 'io.netty.buffer.PooledByteBufAllocator' and from feature io.quarkus.runner.Feature.beforeAnalysis with 'PooledByteBufAllocator.class' to RERUN Quarkus at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:103) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:117) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:117) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:117) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:117) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insert(ClassInitializationConfiguration.java:64) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ProvenSafeClassInitializationSupport.rerunInitialization(ProvenSafeClassInitializationSupport.java:162) at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source) at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:757) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:89) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:757) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:582) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134) at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94) ```

zakkak · 2024-01-11T09:36:11Z

@gsmet it looks like we should have backported this to 3.6 but missed it.

ThoSap · 2024-01-11T10:37:50Z

@zakkak @gsmet do we have to wait until Quarkus 3.7.0 releases on January 31st, or will there be a 3.6.6 release?

gsmet · 2024-01-12T14:48:13Z

This is important enough that I'll try to release a 3.6.6 next Tuesday. Still unsure about my availability to do it but I'll try.

gsmet · 2024-01-12T14:48:37Z

That was for @ThoSap ^

quarkus-bot bot added the area/native-image label Dec 8, 2023

zakkak mentioned this pull request Dec 8, 2023

Netty PlatformDependent0.<init> uses DirectByteBuffer reflectively #37628

Closed

zakkak requested review from cescoffier, gsmet and jamesnetherton December 8, 2023 23:38

This comment has been minimized.

Sign in to view

franz1981 reviewed Dec 9, 2023

View reviewed changes

zakkak force-pushed the 2023-12-09-fix-netty-platform-dependent-run-time-init branch from 3d01c78 to 72d7b79 Compare December 11, 2023 16:07

cescoffier approved these changes Dec 11, 2023

View reviewed changes

jerboaa mentioned this pull request Dec 12, 2023

[CI] Mandrel integration tests fail with Java 22 mandrel build of graal/master on Linux Karm/mandrel-integration-tests#208

Closed

jerboaa mentioned this pull request Dec 12, 2023

[23.0] Sync with upstream graalvm/mandrel#633

Merged

gsmet merged commit 7d49614 into quarkusio:main Dec 12, 2023
49 checks passed

quarkus-bot bot added this to the 3.7 - main milestone Dec 12, 2023

zakkak deleted the 2023-12-09-fix-netty-platform-dependent-run-time-init branch December 12, 2023 21:01

zakkak mentioned this pull request Dec 12, 2023

[23.1] Sync with upstream graalvm/mandrel#634

Merged

This was referenced Dec 13, 2023

Remove duplicate runtime init io.netty.buffer.PooledByteBufAllocator #37707

Merged

JFRTest#jfrPerfTest fails with Quarkus main Karm/mandrel-integration-tests#236

Closed

ThoSap mentioned this pull request Jan 4, 2024

Netty's PlatformDependent and PlatformDependent0 class initializers require reflective access to some classes ? #37626

Closed

zakkak mentioned this pull request Jan 11, 2024

[NATIVE] NullPointerException on PlatformDependent0.newDirectBuffer #38139

Closed

zakkak added the triage/backport? label Jan 11, 2024

gsmet modified the milestones: 3.7 - main, 3.6.6 Jan 15, 2024

gsmet removed the triage/backport? label Jan 15, 2024

zakkak mentioned this pull request Apr 24, 2024

unexpected stacktraces about missing Http1xServerResponse and RecordParserImpl vertx classes in native build #40243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Netty requires more classes to be runtime initialized #37633

Netty requires more classes to be runtime initialized #37633

zakkak commented Dec 8, 2023 •

edited

Loading

This comment has been minimized.

franz1981 Dec 9, 2023

cescoffier Dec 9, 2023

zakkak Dec 9, 2023 •

edited

Loading

zakkak Dec 11, 2023 •

edited

Loading

cescoffier Dec 11, 2023

zakkak Dec 11, 2023

quarkus-bot bot commented Dec 11, 2023

zakkak commented Dec 11, 2023

jerboaa commented Dec 12, 2023

zakkak commented Dec 12, 2023

jerboaa commented Dec 12, 2023

zakkak commented Jan 11, 2024

ThoSap commented Jan 11, 2024

gsmet commented Jan 12, 2024

gsmet commented Jan 12, 2024

Netty requires more classes to be runtime initialized #37633

Netty requires more classes to be runtime initialized #37633

Conversation

zakkak commented Dec 8, 2023 • edited Loading

This comment has been minimized.

franz1981 Dec 9, 2023

Choose a reason for hiding this comment

cescoffier Dec 9, 2023

Choose a reason for hiding this comment

zakkak Dec 9, 2023 • edited Loading

Choose a reason for hiding this comment

zakkak Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

cescoffier Dec 11, 2023

Choose a reason for hiding this comment

zakkak Dec 11, 2023

Choose a reason for hiding this comment

quarkus-bot bot commented Dec 11, 2023

zakkak commented Dec 11, 2023

jerboaa commented Dec 12, 2023

zakkak commented Dec 12, 2023

jerboaa commented Dec 12, 2023

zakkak commented Jan 11, 2024

ThoSap commented Jan 11, 2024

gsmet commented Jan 12, 2024

gsmet commented Jan 12, 2024

zakkak commented Dec 8, 2023 •

edited

Loading

zakkak Dec 9, 2023 •

edited

Loading

zakkak Dec 11, 2023 •

edited

Loading