[SPARK-43537][INFA][BUILD] Upgrading the ASM dependencies used in the tools module to 9.4

[LuciferYang]

What changes were proposed in this pull request?

This pr aims upgrade ASM related dependencies in the tools module from version 7.1 to version 9.4 to make GenerateMIMAIgnore can process Java 17+ compiled code.

Additionally, this pr defines asm.version to manage versions of ASM.

Why are the changes needed?

The classpath processed by GenerateMIMAIgnore cannot contain Java 17+ compiled code now due to the ASM version use by tools module is too low, but https://github.com/bmc/classutil has not been updated for a long time, we can't solve the problem by upgrading classutil, so this pr make the tools module explicitly rely on ASM 9.4 for workaround.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass GitHub Action
• Manual checked dev/mima due to this pr upgrade the dependency of tools module

dev/mima

and

dev/change-scala-version.sh 2.13
dev/mima -Pscala-2.13

• A case that can reproduce the problem: run following script with master branch:

set -o pipefail
set -e

FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
cd "$FWDIR"
export SPARK_HOME=$FWDIR
echo $SPARK_HOME

if [[ -x "$JAVA_HOME/bin/java" ]]; then
  JAVA_CMD="$JAVA_HOME/bin/java"
else
  JAVA_CMD=java
fi

TOOLS_CLASSPATH="$(build/sbt -DcopyDependencies=false "export tools/fullClasspath" | grep jar | tail -n1)"
ASSEMBLY_CLASSPATH="$(build/sbt -DcopyDependencies=false "export assembly/fullClasspath" | grep jar | tail -n1)"

rm -f .generated-mima*

$JAVA_CMD \
  -Xmx2g \
  -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.util.jar=ALL-UNNAMED \
  -cp "$TOOLS_CLASSPATH:$ASSEMBLY_CLASSPATH" \
  org.apache.spark.tools.GenerateMIMAIgnore

rm -f .generated-mima*

Before

Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 61
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:195)
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:176)
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:162)
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:283)
  at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:222)
  at org.clapper.classutil.ClassFinder.classData(ClassFinder.scala:404)
  at org.clapper.classutil.ClassFinder.$anonfun$processOpenZip$2(ClassFinder.scala:359)
  at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
  at scala.collection.Iterator.toStream(Iterator.scala:1417)
  at scala.collection.Iterator.toStream$(Iterator.scala:1416)
  at scala.collection.AbstractIterator.toStream(Iterator.scala:1431)
  at scala.collection.Iterator.$anonfun$toStream$1(Iterator.scala:1417)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.$anonfun$map$1(Stream.scala:418)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.filterImpl(Stream.scala:506)
  at scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.generic.Growable.loop$1(Growable.scala:57)
  at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:61)
  at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  at scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:381)
  at scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:329)
  at scala.collection.TraversableLike.to(TraversableLike.scala:786)
  at scala.collection.TraversableLike.to$(TraversableLike.scala:783)
  at scala.collection.AbstractTraversable.to(Traversable.scala:108)
  at scala.collection.TraversableOnce.toSet(TraversableOnce.scala:360)
  at scala.collection.TraversableOnce.toSet$(TraversableOnce.scala:360)
  at scala.collection.AbstractTraversable.toSet(Traversable.scala:108)
  at org.apache.spark.tools.GenerateMIMAIgnore$.getClasses(GenerateMIMAIgnore.scala:156)
  at org.apache.spark.tools.GenerateMIMAIgnore$.privateWithin(GenerateMIMAIgnore.scala:57)
  at org.apache.spark.tools.GenerateMIMAIgnore$.main(GenerateMIMAIgnore.scala:122)
  at org.apache.spark.tools.GenerateMIMAIgnore.main(GenerateMIMAIgnore.scala)

The script run failed without this pr due to ASSEMBLY_CLASSPATH contains jackson-core 2.15.0 and it contains code compiled from Java 17+

After

No more errors like Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 61 with this pr.

[LuciferYang]

cc @dongjoon-hyun @HyukjinKwon @srowen @hvanhovell FYI

I want to use GenerateMIMAIgnore to generate default exclude filters file base on master branch to solve SPARK-43246 for connect module and encountered this problem.

In the long run, we may also encounter this issue when we need run dev/mima with previousSparkVersion = 3.5.0

[srowen]

To be clear this particular issue is not related to MiMa right? this ASM change fixes Jackson + Java bytecode version issues? they are related?

[LuciferYang]

Yes, this is not related to MiMa check itself.

1. I hope to reuse GenerateMIMAIgnore base on master to generate .generated-mima-class-excludes and .generated-mima-member-excludes during dev/connect-jvm-client-mima-check check process([SPARK-43246][BUILD] Ignore privateClasses and privateMembers from connect mima check as default #40925), then I encountered this problem.

2. In the long run, when we need to performing mima check between Spark 3.6.0 and Spark 3.5.0, we will also need to solve this problem if jackson-core always with Java 11/19 code(due to the OLD_DEPS_CLASSPATH will also contain jackson-core 2.15.0+), but this can indeed be fixed when things happen

I have give an independent pr to solve this issue. If this is not appropriate, I can directly place this change in #40925 for workaround

[LuciferYang]

And I think the key issue solved by this PR is that the classpath processed by GenerateMIMAIgnore cannot contain Java 17 compiled code now due to the ASM version is too low

[LuciferYang]

Updated the pr description.

[dongjoon-hyun]

+1, LGTM.

[srowen]

Oh shoot, I was looking at the wrong PR - I'm not sure that tests passed before I merged. Let me watch the result.

[dongjoon-hyun]

No worry, @srowen ~ I'll monitor together.

[LuciferYang]

If there are any issues, please revert and I will resubmit one :)

[srowen]

OK yeah it was fine, false alarm. Oops.

[LuciferYang]

Thanks @srowen @dongjoon-hyun ~