Officially support Java 11.

[gianm]

There aren't any changes in this patch that improve Java 11
compatibility; these changes have already been done separately. This
patch merely updates documentation and explicit Java version checks.

The log message adjustments in DruidProcessingConfig are there to make
things a little nicer when running in Java 11, where we can't measure
direct memory directly, and so we may auto-size processing buffers
incorrectly.

[clintropolis]

I wonder if worth mentioning in the docs that there might be scary but harmless logs about things that can go away with the relevant jvm module flags

[xvrl]

@gianm should we call out the hadoop stuff and just caveat that?

[gianm]

@gianm should we call out the hadoop stuff and just caveat that?

Probably, although I'm not sure what the call-out should say. It could just say that you may have issues with Hadoop indexing, since Hadoop 2.x on Java 11 isn't officially supported. I'll add that to the Hadoop indexing docs unless someone has a better idea.

[gianm]

I wonder if worth mentioning in the docs that there might be scary but harmless logs about things that can go away with the relevant jvm module flags

Good point. What are the flags? IMO, we should add them ourselves to the default jvm.config file.

The flags will cause Java 8 to bomb out, so we'll need to either have two jvm.configs and have the startup script pick the right one, or add -XX:+IgnoreUnrecognizedVMOptions too so Java 8 ignores them.

[xvrl]

@gianm some are required to get JVM stats, e.g. https://github.com/apache/druid/blob/master/core/src/main/java/org/apache/druid/java/util/metrics/JvmMonitor.java#L169-L175
Other than those, I'm not aware of any that are required to run with Java 11

[gianm]

I think there's also some flag that stops these messages from happening:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:dist/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I see them on all server startups when running on Java 11. They seem harmless but they look scary, like the cops are going to come and toss you in jail.

[xvrl]

@gianm are you planning to add those flags or do we want to do this as a follow-up?

[gianm]

I think I should at least update the Hadoop docs. I can come back and do that next week. In the meantime, if anyone chimes in with knowing what flags are needed to quiet the warnings in #12232 (comment) then I can add those too. Otherwise we can do the flags in a follow up.

[FrankChen021]

I think there's also some flag that stops these messages from happening:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:dist/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I see them on all server startups when running on Java 11. They seem harmless but they look scary, like the cops are going to come and toss you in jail.

@gianm To eliminate this warning, I think we can upgrade the guice from current 4.1.0 to the 4.2.2 which is the lowest version that supports Java 11.

[gianm]

OK, it turns out I didn't have time to pursue those warnings after all. I'll merge this and then we can improve it in future patches. Still very exciting 😄

[FrankChen021]

I think there's also some flag that stops these messages from happening:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:dist/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I see them on all server startups when running on Java 11. They seem harmless but they look scary, like the cops are going to come and toss you in jail.

I made an further investigation on this.

Guice to 5.0.1 (see the highlights) eliminates this warning.
But unfortunately, guice 5.0.1 uses some APIs in newer guava, and the lowest guava version that is compatible with guice 5.0.1 is 20.0. Currently Druid is using guava 16.0.

I noticed that there're some PRs(#11668, #10290, #10683) working on upgrading guava to new version, but they were blocked due to compatibility problem with hadoop 2.

So, before we upgrade guava to newer version, we can't eliminate this warning by upgrading guice.
cc @gianm

[gianm]

Hmm. Thanks for the investigation @FrankChen021. It would be great to solve the guava/guice/hadoop2 problem.