Support building with Java 21 #10474

findepi · 2024-06-10T08:53:27Z

This is currently the latest Java LTS version, so we should support
building with it.

gradle/wrapper/gradle-wrapper.properties

.github/workflows/java-ci.yml

.java-version

build.gradle

nastra · 2024-06-28T06:40:48Z

spark/v3.3/build.gradle

@@ -59,6 +59,10 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
    implementation project(':iceberg-parquet')
    implementation project(':iceberg-arrow')
    implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}:${libs.versions.scala.collection.compat.get()}")
+    if (scalaVersion == '2.12') {
+      // scala-collection-compat_2.12 pulls scala 2.12.17 and we need 2.12.18 for JDK 21 support


does this also apply for Scala 2.13?

apparently not, i think the Ci would detect that

do you remember what the exact error msg was here?

i don't, i should have kept a note

findepi · 2024-06-28T10:20:51Z

Finally not draft anymore!

nastra · 2024-06-28T10:58:32Z

baseline.gradle

-  apply plugin: 'com.diffplug.spotless'
+  // We need to update Google Java Format to 1.17.0+ to run spotless on JDK 8, but that requires dropping support for JDK 8.
+  if (JavaVersion.current() != JavaVersion.VERSION_21) {
+    apply plugin: 'com.diffplug.spotless'


I don't think we can merge the PR without having spotless enabled unfortunately

I wish this was avoidable. I do not know of a way to run same formatting rules under Java 8 and Java 21 and that's we support today.
I do believe that being able to test and actually testing with 21 is incremental improvement over current state. I see value in adding even partial support for 21. In particular, this PR identifies a problem with scala 2.12.

Can you bring this topic up on the Dev mailing list to see what others think about adding JDK21 support without being able to run spotless?

I think the comment is not accurate: we need to update Google Java Format to 1.17.0+ to run spotless on JDK21, but it breaks JDK8.
I'm more in favor of jumping to JDK21 with spotless/GJF 1.17.0+, as soon as we don't have any module requiring JDK8.

@jbonofre we can't upgrade the GJF as long as we support JDK8 because upgrading the GJF version will produce different results depending on the underlying JDK it is executed with and so CI actions (and locally) will fail.

That's my point: I would postpone the JDK21 update right now, and jump directly to JDK21 without supporting JDK8 anymore.
I think it will be very difficult to support both JDK21 and JDK8 from spotless/GJF standpoint. So, I would directly jump to new version when possible.

Dropping JDK8 support means dropping Hive support (which I'm totally fine with) but we need to properly communicate which Iceberg version will drop Hive support. My understanding was that we were planning to drop Hive support with Iceberg 2.0

Yes exactly. It's what I proposed while ago in the Iceberg 2.0 discussion thread. So, I propose:

on 1.x (including 1.6.0 release currently in preparation), we keep JDK8

for 2.x, we merge this PR and jump to JDK21

I can reply on the Iceberg 2.0 thread if you want or Piotr can start a new thread about JDK21 on Iceberg 2.0.
I would love to have feedback on the Iceberg 2.0 thread 😄

Dropping JDK8 support means dropping Hive support (which I'm totally fine with) but we need to properly communicate which Iceberg version will drop Hive support. My understanding was that we were planning to drop Hive support with Iceberg 2.0

Sir. Maybe the iceberg-hive module should be maintained by the hive community in the hive project. Because hive 3.X will EOL sooner or later.

Or just keep the catalog support.

@BsoBird newer versions of Hive are maintained by the Hive community but the Iceberg community still maintains an older version of Hive

pan3793 · 2024-07-09T12:36:10Z

.github/workflows/spark-ci.yml

        spark: ['3.3', '3.4', '3.5']
        scala: ['2.12', '2.13']
+        exclude:
+          # Spark 3.5 is the first version supporting Java 21 (https://issues.apache.org/jira/browse/SPARK-42369)


Actually Spark 3.5 does not fully support Java 21, the right ticket is https://issues.apache.org/jira/browse/SPARK-43831

Thanks @pan3793 for your comment and the link. This is good reference.
If our tests with Spark 3.5 pass with Java 21, we don't need to disable them, but it's good to know we should be expecting challenges. I will rework the comment.

This is currently the latest Java LTS version, so we should support building with it.

findepi · 2024-07-10T13:26:51Z

Since spotless plugin is not installed on JDK 21, ./gradlew spotlessApply would fail with

$ JAVA_HOME=~/.jenv/versions/21 ./gradlew spotlessApply

FAILURE: Build failed with an exception.

* What went wrong:
Task 'spotlessApply' not found in root project 'iceberg' and its subprojects.

* Try:
> Run gradlew tasks to get a list of available tasks.
> For more on name expansion, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:name_abbreviation in the Gradle documentation.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 14s

Added spotlessApply stub, for more actionable message:

$ JAVA_HOME=~/.jenv/versions/21 ./gradlew spotlessApply

FAILURE: Build failed with an exception.

* Where:
Script '/Users/findepi/repos/iceberg/baseline.gradle' line: 51

* What went wrong:
A problem occurred evaluating script.
> Spotless plugin is currently disabled when running on JDK 21 (until we drop JDK 8). To run spotlessApply please use a different JDK version.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 3s

nastra · 2024-07-11T11:42:44Z

baseline.gradle

+  if (JavaVersion.current() == JavaVersion.VERSION_21) {
+    task spotlessApply {
+      doLast {
+        throw new GradleException("Spotless plugin is currently disabled when running on JDK 21 (until we drop JDK 8). To run spotlessApply please use a different JDK version.")


thanks for adding that msg, that's going to be helpful

nastra · 2024-07-25T12:45:27Z

thanks for working on this @findepi

findepi · 2024-07-25T20:02:19Z

thanks for the merge!

github-actions bot added INFRA docs build labels Jun 10, 2024

findepi marked this pull request as draft June 10, 2024 09:51

github-actions bot added API spark parquet core flink ORC AWS NESSIE GCP DELL SNOWFLAKE labels Jun 10, 2024

findepi force-pushed the findepi/21 branch 2 times, most recently from 9d01de8 to 04a5c77 Compare June 10, 2024 15:59

findepi commented Jun 10, 2024

View reviewed changes

gradle/wrapper/gradle-wrapper.properties Outdated Show resolved Hide resolved

.github/workflows/java-ci.yml Outdated Show resolved Hide resolved

.java-version Outdated Show resolved Hide resolved

findepi force-pushed the findepi/21 branch 3 times, most recently from 5b03499 to f564c36 Compare June 11, 2024 08:50

findepi mentioned this pull request Jun 11, 2024

Run Flink tests on Java 17 too #10477

Merged

findepi force-pushed the findepi/21 branch 5 times, most recently from 7ca0b11 to d9bae17 Compare June 11, 2024 13:03

github-actions bot added the hive label Jun 11, 2024

findepi mentioned this pull request Jun 11, 2024

Run revapi workflow on workflow/build system changes #10485

Merged

findepi marked this pull request as ready for review June 11, 2024 19:56

findepi force-pushed the findepi/21 branch 2 times, most recently from 344c830 to 92c17e6 Compare June 20, 2024 14:22

findepi force-pushed the findepi/21 branch from 92c17e6 to 4f9f29a Compare June 27, 2024 20:16

findepi requested a review from nastra June 27, 2024 20:17

findepi force-pushed the findepi/21 branch from 4f9f29a to 5b48c98 Compare June 27, 2024 20:19

nastra reviewed Jun 28, 2024

View reviewed changes

build.gradle Show resolved Hide resolved

nastra reviewed Jun 28, 2024

View reviewed changes

findepi force-pushed the findepi/21 branch from 5b48c98 to fb4a6fb Compare June 28, 2024 10:20

findepi marked this pull request as ready for review June 28, 2024 10:20

findepi requested review from nastra and Fokko June 28, 2024 10:20

nastra reviewed Jun 28, 2024

View reviewed changes

findepi force-pushed the findepi/21 branch from fb4a6fb to 77d06cf Compare June 28, 2024 13:11

findepi requested a review from nastra June 29, 2024 20:48

nastra added this to the Iceberg 2.0.0 milestone Jul 2, 2024

pan3793 reviewed Jul 9, 2024

View reviewed changes

Support building with Java 21

0e66bd0

This is currently the latest Java LTS version, so we should support building with it.

findepi force-pushed the findepi/21 branch from 77d06cf to 0e66bd0 Compare July 9, 2024 13:34

Add spotlessApply stub for JDK 21

27e9f55

findepi force-pushed the findepi/21 branch from 61b8665 to 27e9f55 Compare July 11, 2024 10:26

nastra reviewed Jul 11, 2024

View reviewed changes

nastra approved these changes Jul 23, 2024

View reviewed changes

Fokko approved these changes Jul 23, 2024

View reviewed changes

nastra merged commit 7bced33 into apache:main Jul 25, 2024
61 checks passed

findepi deleted the findepi/21 branch July 25, 2024 20:02

zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request Dec 23, 2024

Build: Support building with Java 21 (apache#10474)

9f15840

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support building with Java 21 #10474

Support building with Java 21 #10474

findepi commented Jun 10, 2024

nastra Jun 28, 2024

findepi Jun 28, 2024

nastra Jul 23, 2024

findepi Jul 24, 2024

findepi commented Jun 28, 2024

nastra Jun 28, 2024

findepi Jun 28, 2024

nastra Jul 1, 2024

jbonofre Jul 1, 2024

nastra Jul 1, 2024

jbonofre Jul 1, 2024

nastra Jul 1, 2024

jbonofre Jul 1, 2024

BsoBird Jul 2, 2024 •

edited

Loading

nastra Jul 2, 2024

pan3793 Jul 9, 2024

findepi Jul 9, 2024

findepi commented Jul 10, 2024

nastra Jul 11, 2024

nastra commented Jul 25, 2024

findepi commented Jul 25, 2024

Support building with Java 21 #10474

Support building with Java 21 #10474

Conversation

findepi commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Jun 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BsoBird Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Jul 10, 2024

Choose a reason for hiding this comment

nastra commented Jul 25, 2024

findepi commented Jul 25, 2024

BsoBird Jul 2, 2024 •

edited

Loading