From 9785353684bdc2a2c7445b7e6b9ab85154f6933f Mon Sep 17 00:00:00 2001 From: yangjie01 Date: Wed, 17 May 2023 11:18:14 -0500 Subject: [PATCH] [SPARK-43537][INFA][BUILD] Upgrading the ASM dependencies used in the `tools` module to 9.4 ### What changes were proposed in this pull request? This pr aims upgrade ASM related dependencies in the `tools` module from version 7.1 to version 9.4 to make `GenerateMIMAIgnore` can process Java 17+ compiled code. Additionally, this pr defines `asm.version` to manage versions of ASM. ### Why are the changes needed? The classpath processed by `GenerateMIMAIgnore` cannot contain Java 17+ compiled code now due to the ASM version use by `tools` module is too low, but https://github.com/bmc/classutil has not been updated for a long time, we can't solve the problem by upgrading `classutil`, so this pr make the `tools` module explicitly rely on ASM 9.4 for workaround. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Action - Manual checked `dev/mima` due to this pr upgrade the dependency of tools module ``` dev/mima ``` and ``` dev/change-scala-version.sh 2.13 dev/mima -Pscala-2.13 ``` - A case that can reproduce the problem: run following script with master branch: ``` set -o pipefail set -e FWDIR="$(cd "`dirname "$0"`"/..; pwd)" cd "$FWDIR" export SPARK_HOME=$FWDIR echo $SPARK_HOME if [[ -x "$JAVA_HOME/bin/java" ]]; then JAVA_CMD="$JAVA_HOME/bin/java" else JAVA_CMD=java fi TOOLS_CLASSPATH="$(build/sbt -DcopyDependencies=false "export tools/fullClasspath" | grep jar | tail -n1)" ASSEMBLY_CLASSPATH="$(build/sbt -DcopyDependencies=false "export assembly/fullClasspath" | grep jar | tail -n1)" rm -f .generated-mima* $JAVA_CMD \ -Xmx2g \ -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.util.jar=ALL-UNNAMED \ -cp "$TOOLS_CLASSPATH:$ASSEMBLY_CLASSPATH" \ org.apache.spark.tools.GenerateMIMAIgnore rm -f .generated-mima* ``` **Before** ``` Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 61 at org.objectweb.asm.ClassReader.(ClassReader.java:195) at org.objectweb.asm.ClassReader.(ClassReader.java:176) at org.objectweb.asm.ClassReader.(ClassReader.java:162) at org.objectweb.asm.ClassReader.(ClassReader.java:283) at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:222) at org.clapper.classutil.ClassFinder.classData(ClassFinder.scala:404) at org.clapper.classutil.ClassFinder.$anonfun$processOpenZip$2(ClassFinder.scala:359) at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.toStream(Iterator.scala:1417) at scala.collection.Iterator.toStream$(Iterator.scala:1416) at scala.collection.AbstractIterator.toStream(Iterator.scala:1431) at scala.collection.Iterator.$anonfun$toStream$1(Iterator.scala:1417) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.immutable.Stream.$anonfun$map$1(Stream.scala:418) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.immutable.Stream.filterImpl(Stream.scala:506) at scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173) at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163) at scala.collection.generic.Growable.loop$1(Growable.scala:57) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:61) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:381) at scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:329) at scala.collection.TraversableLike.to(TraversableLike.scala:786) at scala.collection.TraversableLike.to$(TraversableLike.scala:783) at scala.collection.AbstractTraversable.to(Traversable.scala:108) at scala.collection.TraversableOnce.toSet(TraversableOnce.scala:360) at scala.collection.TraversableOnce.toSet$(TraversableOnce.scala:360) at scala.collection.AbstractTraversable.toSet(Traversable.scala:108) at org.apache.spark.tools.GenerateMIMAIgnore$.getClasses(GenerateMIMAIgnore.scala:156) at org.apache.spark.tools.GenerateMIMAIgnore$.privateWithin(GenerateMIMAIgnore.scala:57) at org.apache.spark.tools.GenerateMIMAIgnore$.main(GenerateMIMAIgnore.scala:122) at org.apache.spark.tools.GenerateMIMAIgnore.main(GenerateMIMAIgnore.scala) ``` The script run failed without this pr due to `ASSEMBLY_CLASSPATH` contains jackson-core 2.15.0 and it contains code compiled from Java 17+ image **After** No more errors like `Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 61` with this pr. Closes #41198 from LuciferYang/tools-asm. Authored-by: yangjie01 Signed-off-by: Sean Owen --- pom.xml | 5 +++-- tools/pom.xml | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index c0506d7e31de5..a81e6df057dd5 100644 --- a/pom.xml +++ b/pom.xml @@ -117,6 +117,7 @@ 3.8.8 1.6.0 spark + 9.4 2.0.7 2.20.0 @@ -3130,12 +3131,12 @@ org.ow2.asm asm - 9.4 + ${asm.version} org.ow2.asm asm-commons - 9.4 + ${asm.version} diff --git a/tools/pom.xml b/tools/pom.xml index ce2aebe9e598a..9dc136d426871 100644 --- a/tools/pom.xml +++ b/tools/pom.xml @@ -44,8 +44,29 @@ org.clapper classutil_${scala.binary.version} + + + org.ow2.asm + * + + 1.5.1 + + org.ow2.asm + asm + ${asm.version} + + + org.ow2.asm + asm-commons + ${asm.version} + + + org.ow2.asm + asm-util + ${asm.version} +