[SPARK-37302][BUILD] Explicitly downloads guava and jetty-io in test-dependencies.sh.

[sarutak]

What changes were proposed in this pull request?

This PR change dev/test-dependencies.sh to download guava and jetty-io explicitly.

dev/run-tests.py fails if Scala 2.13 is used and guava or jetty-io is not in the both of Maven and Coursier local repository.

$ rm -rf ~/.m2/repository/*
$ # For Linux
$ rm -rf ~/.cache/coursier/v1/*
$ # For macOS
$ rm -rf ~/Library/Caches/Coursier/v1/*
$ dev/change-scala-version.sh 2.13
$ dev/test-dependencies.sh
$ build/sbt -Pscala-2.13 clean compile
...
[error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java:24:1:  error: package com.google.common.primitives does not exist
[error] import com.google.common.primitives.Ints;
[error]                                    ^
[error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:30:1:  error: package com.google.common.annotations does not exist
[error] import com.google.common.annotations.VisibleForTesting;
[error]                                     ^
[error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:31:1:  error: package com.google.common.base does not exist
[error] import com.google.common.base.Preconditions;
...

[error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:25: Class org.eclipse.jetty.io.ByteBufferPool not found - continuing with a stub.
[error]     val connector = new ServerConnector(
[error]                         ^
[error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:21: multiple constructors for ServerConnector with alternatives:
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector
[error]  cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.HttpConnectionFactory)
[error]     val connector = new ServerConnector(
[error]                     ^
[error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:207:13: Class org.eclipse.jetty.io.ClientConnectionFactory not found - continuing with a stub.
[error]         new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null)
[error]             ^
[error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:287:25: multiple constructors for ServerConnector with alternatives:
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and>
[error]   (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector
[error]  cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.ConnectionFactory)
[error]         val connector = new ServerConnector(

The reason is that exec-maven-plugin used in test-dependencies.sh downloads pom of guava and jetty-io but doesn't downloads the corresponding jars, and skip dependency testing if Scala 2.13 is used (if dependency testing runs, Maven downloads those jars).

if [[ "$SCALA_BINARY_VERSION" != "2.12" ]]; then
  # TODO(SPARK-36168) Support Scala 2.13 in dev/test-dependencies.sh
  echo "Skip dependency testing on $SCALA_BINARY_VERSION"
  exit 0
fi

$ find ~/.m2 -name "guava*"
...
/home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom
/home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom.sha1
...
/home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
/home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom.sha1
...

$ find ~/.m2 -name "jetty*"
...
/home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom
/home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom.sha1
...

Under the circumstances, building Spark using SBT fails.
run-tests.py builds Spark using SBT after the dependency testing so run-tests.py fails with Scala 2.13.

Further, I noticed that this issue can even happen with Sala 2.12 if the script exit at the following part.

OLD_VERSION=$($MVN -q \
    -Dexec.executable="echo" \
    -Dexec.args='${project.version}' \
    --non-recursive \
    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+')
if [ $? != 0 ]; then
    echo -e "Error while getting version string from Maven:\n$OLD_VERSION"
    exit 1
fi

This phenomenon is similar to SPARK-34762 (#31862).
So the fix is to get guava and jetty-io explicitly by mvn dependency:get.

Why are the changes needed?

To keep the Maven local repository sanity. Then, we can avoid such confusable build error.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that we can successfully compile code with the following commands.

rm -rf ~/.m2/repository/*
# For Linux
rm -rf ~/.cache/coursier/v1/*
# For macOS
rm -rf ~/Library/Caches/Coursier/v1/*
dev/change-scala-version.sh 2.13
dev/test-dependencies.sh
./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 compile test:compile

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49628/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49628/

[SparkQA]

Test build #145157 has finished for PR 34570 at commit 9e24167.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you revise the verification section like Section What changes were proposed in this pull request?, @sarutak ? I guess you need to add the followings.

• dev/change-scala-version.sh 2.13
• dev/test-dependencies.sh

[sarutak]

@dongjoon-hyun Oh, I forgot to add them. Thank you.

[dongjoon-hyun]

In the master branch, when I delete Guava and compile Scala 2.13, it seems to re-download Guava correctly. Is there something I can do more to verify this PR?

$ find ~/Library/Caches/Coursier/v1 -name "guava*"
$ find ~/.m2 -name "guava*"
/Users/dongjoon/.m2/repository/org/sonatype/sisu/inject/guava-parent
/Users/dongjoon/.m2/repository/org/sonatype/sisu/inject/guava-parent/0.9.9/guava-parent-0.9.9.pom.sha1
/Users/dongjoon/.m2/repository/org/sonatype/sisu/inject/guava-parent/0.9.9/guava-parent-0.9.9.pom
$ build/sbt -Pscala-2.13 clean compile
Using /Users/dongjoon/.jenv/versions/zulu11 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Attempting to fetch sbt
Launching sbt from build/sbt-launch-1.5.5.jar
[info] welcome to sbt 1.5.5 (Azul Systems, Inc. Java 11.0.12)
[info] loading settings for project spark-merge-build from plugins.sbt ...
[info] loading project definition from /Users/dongjoon/APACHE/spark-merge/project
[info] Updating
https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.pom
  100.0% [##########] 5.5 KiB (28.0 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava-parent/23.0/guava-parent-23.0.pom
  100.0% [##########] 9.1 KiB (227.9 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava/30.1.1-jre/guava-30.1.1-jre.pom
  100.0% [##########] 11.9 KiB (701.4 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava-parent/30.1.1-jre/guava-parent-30.1.1-jre.pom
  100.0% [##########] 13.9 KiB (463.3 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.pom
  100.0% [##########] 2.2 KiB (130.9 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.pom
  100.0% [##########] 2.4 KiB (39.3 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava-parent/26.0-android/guava-parent-26.0-android.pom
  100.0% [##########] 9.9 KiB (292.4 KiB / s)
...
[warn] 1 warning.
[success] Total time: 107 s (01:47), completed Nov 12, 2021, 10:23:28 AM

[sarutak]

Hmm, it's funny. How about ~/.ivy2 ?
I didn't notice at that time but if guava is present on ~/.ivy2, this issue might not happen.

[sarutak]

@dongjoon-hyun
BTW, the following commands are executed after dev/test-dependencies?

$ find ~/Library/Caches/Coursier/v1 -name "guava*"
$ find ~/.m2 -name "guava*"

If so, guava-14.0.1.pom should be present on ~/.m2 but the result of find it doesn't appear in the result you showed.
In case guava-14.0.1.pom is present but guava-14.0.1.jar is not, this issue happens.

[dongjoon-hyun]

Thanks. Let me retry step-by-step~

[dongjoon-hyun]

+1, LGTM. I verified successfully. Thank you, @sarutak .
Merged to master/3.2.

[sarutak]

@dongjoon-hyun
Thank you for verifying and merging !