[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli #31862

sarutak · 2021-03-17T07:05:50Z

What changes were proposed in this pull request?

This PR fixes the build failure with Scala 2.13 which is related to commons-cli.
The last few days, build with Scala 2.13 on GA continues to fail and the error message says like as follows.

[error] /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java:26:1:  error: package org.apache.commons.cli does not exist
1278[error] import org.apache.commons.cli.GnuParser;

The reason is that mvn help in change-scala-version.sh downloads the POM file of commons-cli but doesn't download the JAR file, leading the build failure.

This PR also adds commons-cli to the dependencies explicitly because HiveThriftServer depends on it.

Why are the changes needed?

Expect to fix the build failure with Scala 2.13.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I confirmed that build successfully finishes with Scala 2.13 on my laptop.

find ~/.m2 -name commons-cli -exec rm -rf {} \;
find ~/.ivy2 -name commons-cli -exec rm -rf {} \;
find ~/.cache/ -name commons-cli -exec rm -rf {} \; // For Linux
find ~/Library/Caches -name commons-cli -exec rm -rf {} \; // For macOS

dev/change-scala-version 2.13
./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 clean compile test:compile

HyukjinKwon · 2021-03-17T07:26:10Z

Awesome @sarutak!

pom.xml

SparkQA · 2021-03-17T08:29:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40730/

SparkQA · 2021-03-17T08:33:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40730/

cloud-fan

great job!

sarutak · 2021-03-17T13:25:52Z

Actually, adding commons-cli explicitly didn't resolve the issue (6812c0b fails) but I noticed #31863 succeed to build with Scala 2.13 with sbt. So I found, if commons-cli is already in .m2, the build will fail with sbt.
In change-scala-version.sh, mvn command runs and this downloads commons-cli to .m2.

So I added a workaround to change-scala-version.sh that removes ~/.m2/repository/commons-cli.
Let's see whether the workaround works well.

After I found the workaround works, I'll change the title and description.

sarutak · 2021-03-17T13:54:19Z

I might find the real cause.
The help plugin used in change-scala-version.sh downloads only POM file and it's hash file for commons-cli, and the jar file is not downloaded.

ls /home/kou/.m2/repository/commons-cli/commons-cli/1.2/
_remote.repositories  commons-cli-1.2.pom  commons-cli-1.2.pom.sha1

SparkQA · 2021-03-17T15:32:47Z

Test build #136160 has finished for PR 31862 at commit 4c6e8dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-03-18T00:49:09Z

Finally, build with Scala 2.13 seems to pass.

HyukjinKwon · 2021-03-18T03:30:42Z

I'm gonna merge this for other PRs.

HyukjinKwon · 2021-03-18T03:31:33Z

Merged to master and branch-3.1.

…elated to commons-cli ### What changes were proposed in this pull request? This PR fixes the build failure with Scala 2.13 which is related to `commons-cli`. The last few days, build with Scala 2.13 on GA continues to fail and the error message says like as follows. ``` [error] /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java:26:1: error: package org.apache.commons.cli does not exist 1278[error] import org.apache.commons.cli.GnuParser; ``` The reason is that `mvn help` in `change-scala-version.sh` downloads the POM file of `commons-cli` but doesn't download the JAR file, leading the build failure. This PR also adds `commons-cli` to the dependencies explicitly because HiveThriftServer depends on it. ### Why are the changes needed? Expect to fix the build failure with Scala 2.13. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed that build successfully finishes with Scala 2.13 on my laptop. ``` find ~/.m2 -name commons-cli -exec rm -rf {} \; find ~/.ivy2 -name commons-cli -exec rm -rf {} \; find ~/.cache/ -name commons-cli -exec rm -rf {} \; // For Linux find ~/Library/Caches -name commons-cli -exec rm -rf {} \; // For macOS dev/change-scala-version 2.13 ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 clean compile test:compile ``` Closes #31862 from sarutak/commons-cli. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit c5cadfe) Signed-off-by: HyukjinKwon <[email protected]>

sarutak · 2021-03-18T04:26:22Z

@HyukjinKwon
Thanks for merging!
I was going to merge by myself after I get agreement on the change but it's ok as I have approval from you now.

dongjoon-hyun · 2021-03-18T19:18:25Z

Thank you, @sarutak and all!

…elated to commons-cli ### What changes were proposed in this pull request? This PR fixes the build failure with Scala 2.13 which is related to `commons-cli`. The last few days, build with Scala 2.13 on GA continues to fail and the error message says like as follows. ``` [error] /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java:26:1: error: package org.apache.commons.cli does not exist 1278[error] import org.apache.commons.cli.GnuParser; ``` The reason is that `mvn help` in `change-scala-version.sh` downloads the POM file of `commons-cli` but doesn't download the JAR file, leading the build failure. This PR also adds `commons-cli` to the dependencies explicitly because HiveThriftServer depends on it. ### Why are the changes needed? Expect to fix the build failure with Scala 2.13. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed that build successfully finishes with Scala 2.13 on my laptop. ``` find ~/.m2 -name commons-cli -exec rm -rf {} \; find ~/.ivy2 -name commons-cli -exec rm -rf {} \; find ~/.cache/ -name commons-cli -exec rm -rf {} \; // For Linux find ~/Library/Caches -name commons-cli -exec rm -rf {} \; // For macOS dev/change-scala-version 2.13 ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 clean compile test:compile ``` Closes apache#31862 from sarutak/commons-cli. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit c5cadfe) Signed-off-by: HyukjinKwon <[email protected]>

…dependencies.sh ### What changes were proposed in this pull request? This PR change `dev/test-dependencies.sh` to download `guava` and `jetty-io` explicitly. `dev/run-tests.py` fails if Scala 2.13 is used and `guava` or `jetty-io` is not in the both of Maven and Coursier local repository. ``` $ rm -rf ~/.m2/repository/* $ # For Linux $ rm -rf ~/.cache/coursier/v1/* $ # For macOS $ rm -rf ~/Library/Caches/Coursier/v1/* $ dev/change-scala-version.sh 2.13 $ dev/test-dependencies.sh $ build/sbt -Pscala-2.13 clean compile ... [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java:24:1: error: package com.google.common.primitives does not exist [error] import com.google.common.primitives.Ints; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:30:1: error: package com.google.common.annotations does not exist [error] import com.google.common.annotations.VisibleForTesting; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:31:1: error: package com.google.common.base does not exist [error] import com.google.common.base.Preconditions; ... ``` ``` [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:25: Class org.eclipse.jetty.io.ByteBufferPool not found - continuing with a stub. [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:21: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.HttpConnectionFactory) [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:207:13: Class org.eclipse.jetty.io.ClientConnectionFactory not found - continuing with a stub. [error] new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null) [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:287:25: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.ConnectionFactory) [error] val connector = new ServerConnector( ``` The reason is that `exec-maven-plugin` used in `test-dependencies.sh` downloads pom of guava and jetty-io but doesn't downloads the corresponding jars, and skip dependency testing if Scala 2.13 is used (if dependency testing runs, Maven downloads those jars). ``` if [[ "$SCALA_BINARY_VERSION" != "2.12" ]]; then # TODO(SPARK-36168) Support Scala 2.13 in dev/test-dependencies.sh echo "Skip dependency testing on $SCALA_BINARY_VERSION" exit 0 fi ``` ``` $ find ~/.m2 -name "guava*" ... /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom.sha1 ... /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom.sha1 ... $ find ~/.m2 -name "jetty*" ... /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom.sha1 ... ``` Under the circumstances, building Spark using SBT fails. `run-tests.py` builds Spark using SBT after the dependency testing so `run-tests.py` fails with Scala 2.13. Further, I noticed that this issue can even happen with Sala 2.12 if the script exit at the following part. ``` OLD_VERSION=$($MVN -q \ -Dexec.executable="echo" \ -Dexec.args='${project.version}' \ --non-recursive \ org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+') if [ $? != 0 ]; then echo -e "Error while getting version string from Maven:\n$OLD_VERSION" exit 1 fi ``` This phenomenon is similar to SPARK-34762 (#31862). So the fix is to get guava and jetty-io explicitly by `mvn dependency:get`. ### Why are the changes needed? To keep the Maven local repository sanity. Then, we can avoid such confusable build error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that we can successfully compile code with the following commands. ``` rm -rf ~/.m2/repository/* # For Linux rm -rf ~/.cache/coursier/v1/* # For macOS rm -rf ~/Library/Caches/Coursier/v1/* dev/change-scala-version.sh 2.13 dev/test-dependencies.sh ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 compile test:compile ``` Closes #34570 from sarutak/fix-test-dependencies-issue. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…dependencies.sh ### What changes were proposed in this pull request? This PR change `dev/test-dependencies.sh` to download `guava` and `jetty-io` explicitly. `dev/run-tests.py` fails if Scala 2.13 is used and `guava` or `jetty-io` is not in the both of Maven and Coursier local repository. ``` $ rm -rf ~/.m2/repository/* $ # For Linux $ rm -rf ~/.cache/coursier/v1/* $ # For macOS $ rm -rf ~/Library/Caches/Coursier/v1/* $ dev/change-scala-version.sh 2.13 $ dev/test-dependencies.sh $ build/sbt -Pscala-2.13 clean compile ... [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java:24:1: error: package com.google.common.primitives does not exist [error] import com.google.common.primitives.Ints; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:30:1: error: package com.google.common.annotations does not exist [error] import com.google.common.annotations.VisibleForTesting; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:31:1: error: package com.google.common.base does not exist [error] import com.google.common.base.Preconditions; ... ``` ``` [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:25: Class org.eclipse.jetty.io.ByteBufferPool not found - continuing with a stub. [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:21: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.HttpConnectionFactory) [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:207:13: Class org.eclipse.jetty.io.ClientConnectionFactory not found - continuing with a stub. [error] new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null) [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:287:25: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.ConnectionFactory) [error] val connector = new ServerConnector( ``` The reason is that `exec-maven-plugin` used in `test-dependencies.sh` downloads pom of guava and jetty-io but doesn't downloads the corresponding jars, and skip dependency testing if Scala 2.13 is used (if dependency testing runs, Maven downloads those jars). ``` if [[ "$SCALA_BINARY_VERSION" != "2.12" ]]; then # TODO(SPARK-36168) Support Scala 2.13 in dev/test-dependencies.sh echo "Skip dependency testing on $SCALA_BINARY_VERSION" exit 0 fi ``` ``` $ find ~/.m2 -name "guava*" ... /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom.sha1 ... /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom.sha1 ... $ find ~/.m2 -name "jetty*" ... /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom.sha1 ... ``` Under the circumstances, building Spark using SBT fails. `run-tests.py` builds Spark using SBT after the dependency testing so `run-tests.py` fails with Scala 2.13. Further, I noticed that this issue can even happen with Sala 2.12 if the script exit at the following part. ``` OLD_VERSION=$($MVN -q \ -Dexec.executable="echo" \ -Dexec.args='${project.version}' \ --non-recursive \ org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+') if [ $? != 0 ]; then echo -e "Error while getting version string from Maven:\n$OLD_VERSION" exit 1 fi ``` This phenomenon is similar to SPARK-34762 (#31862). So the fix is to get guava and jetty-io explicitly by `mvn dependency:get`. ### Why are the changes needed? To keep the Maven local repository sanity. Then, we can avoid such confusable build error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that we can successfully compile code with the following commands. ``` rm -rf ~/.m2/repository/* # For Linux rm -rf ~/.cache/coursier/v1/* # For macOS rm -rf ~/Library/Caches/Coursier/v1/* dev/change-scala-version.sh 2.13 dev/test-dependencies.sh ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 compile test:compile ``` Closes #34570 from sarutak/fix-test-dependencies-issue. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 01ab0be) Signed-off-by: Dongjoon Hyun <[email protected]>

…dependencies.sh ### What changes were proposed in this pull request? This PR change `dev/test-dependencies.sh` to download `guava` and `jetty-io` explicitly. `dev/run-tests.py` fails if Scala 2.13 is used and `guava` or `jetty-io` is not in the both of Maven and Coursier local repository. ``` $ rm -rf ~/.m2/repository/* $ # For Linux $ rm -rf ~/.cache/coursier/v1/* $ # For macOS $ rm -rf ~/Library/Caches/Coursier/v1/* $ dev/change-scala-version.sh 2.13 $ dev/test-dependencies.sh $ build/sbt -Pscala-2.13 clean compile ... [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java:24:1: error: package com.google.common.primitives does not exist [error] import com.google.common.primitives.Ints; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:30:1: error: package com.google.common.annotations does not exist [error] import com.google.common.annotations.VisibleForTesting; [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:31:1: error: package com.google.common.base does not exist [error] import com.google.common.base.Preconditions; ... ``` ``` [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:25: Class org.eclipse.jetty.io.ByteBufferPool not found - continuing with a stub. [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala:87:21: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.HttpConnectionFactory) [error] val connector = new ServerConnector( [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:207:13: Class org.eclipse.jetty.io.ClientConnectionFactory not found - continuing with a stub. [error] new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null) [error] ^ [error] /home/kou/work/oss/spark-scala-2.13/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:287:25: multiple constructors for ServerConnector with alternatives: [error] (x$1: org.eclipse.jetty.server.Server,x$2: java.util.concurrent.Executor,x$3: org.eclipse.jetty.util.thread.Scheduler,x$4: org.eclipse.jetty.io.ByteBufferPool,x$5: Int,x$6: Int,x$7: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.util.ssl.SslContextFactory,x$3: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector <and> [error] (x$1: org.eclipse.jetty.server.Server,x$2: Int,x$3: Int,x$4: org.eclipse.jetty.server.ConnectionFactory*)org.eclipse.jetty.server.ServerConnector [error] cannot be invoked with (org.eclipse.jetty.server.Server, Null, org.eclipse.jetty.util.thread.ScheduledExecutorScheduler, Null, Int, Int, org.eclipse.jetty.server.ConnectionFactory) [error] val connector = new ServerConnector( ``` The reason is that `exec-maven-plugin` used in `test-dependencies.sh` downloads pom of guava and jetty-io but doesn't downloads the corresponding jars, and skip dependency testing if Scala 2.13 is used (if dependency testing runs, Maven downloads those jars). ``` if [[ "$SCALA_BINARY_VERSION" != "2.12" ]]; then # TODO(SPARK-36168) Support Scala 2.13 in dev/test-dependencies.sh echo "Skip dependency testing on $SCALA_BINARY_VERSION" exit 0 fi ``` ``` $ find ~/.m2 -name "guava*" ... /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.pom.sha1 ... /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom /home/kou/.m2/repository/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom.sha1 ... $ find ~/.m2 -name "jetty*" ... /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom /home/kou/.m2/repository/org/eclipse/jetty/jetty-io/9.4.43.v20210629/jetty-io-9.4.43.v20210629.pom.sha1 ... ``` Under the circumstances, building Spark using SBT fails. `run-tests.py` builds Spark using SBT after the dependency testing so `run-tests.py` fails with Scala 2.13. Further, I noticed that this issue can even happen with Sala 2.12 if the script exit at the following part. ``` OLD_VERSION=$($MVN -q \ -Dexec.executable="echo" \ -Dexec.args='${project.version}' \ --non-recursive \ org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+') if [ $? != 0 ]; then echo -e "Error while getting version string from Maven:\n$OLD_VERSION" exit 1 fi ``` This phenomenon is similar to SPARK-34762 (apache#31862). So the fix is to get guava and jetty-io explicitly by `mvn dependency:get`. ### Why are the changes needed? To keep the Maven local repository sanity. Then, we can avoid such confusable build error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that we can successfully compile code with the following commands. ``` rm -rf ~/.m2/repository/* # For Linux rm -rf ~/.cache/coursier/v1/* # For macOS rm -rf ~/Library/Caches/Coursier/v1/* dev/change-scala-version.sh 2.13 dev/test-dependencies.sh ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 compile test:compile ``` Closes apache#34570 from sarutak/fix-test-dependencies-issue. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 01ab0be) Signed-off-by: Dongjoon Hyun <[email protected]>

Add commons-cli to the dependencies.

9b80445

sarutak changed the title ~~[SPARK-34762][BUILD] Add commons-cli to the dependencies.~~ [SPARK-34762][BUILD] Add commons-cli to the dependencies explicitly. Mar 17, 2021

github-actions bot added BUILD SQL labels Mar 17, 2021

yaooqinn reviewed Mar 17, 2021

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

change 1.4 -> ${commons-cli.version}

464edc0

HyukjinKwon reviewed Mar 17, 2021

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

HyukjinKwon reviewed Mar 17, 2021

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

Choose version 1.2 for commons-cli.

6812c0b

sarutak force-pushed the commons-cli branch from 18a5f59 to 6812c0b Compare March 17, 2021 07:46

HyukjinKwon approved these changes Mar 17, 2021

View reviewed changes

LuciferYang mentioned this pull request Mar 17, 2021

[SPARK-34749][SQL] Simplify ResolveCreateNamedStruct #31843

Closed

Add workaround.

18c93a0

sarutak force-pushed the commons-cli branch from 4c6e8dc to 18c93a0 Compare March 17, 2021 12:57

cloud-fan approved these changes Mar 17, 2021

View reviewed changes

Download commons-cli in change-scala-version.sh.

8b1dd5f

sarutak changed the title ~~[SPARK-34762][BUILD] Add commons-cli to the dependencies explicitly.~~ [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli Mar 17, 2021

HyukjinKwon approved these changes Mar 18, 2021

View reviewed changes

HyukjinKwon closed this in c5cadfe Mar 18, 2021

sarutak mentioned this pull request Mar 18, 2021

[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution #31880

Closed

sarutak mentioned this pull request Nov 12, 2021

[SPARK-37302][BUILD] Explicitly downloads guava and jetty-io in test-dependencies.sh. #34570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli #31862

[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli #31862

sarutak commented Mar 17, 2021 •

edited

Loading

HyukjinKwon commented Mar 17, 2021

SparkQA commented Mar 17, 2021

SparkQA commented Mar 17, 2021

cloud-fan left a comment

sarutak commented Mar 17, 2021 •

edited

Loading

sarutak commented Mar 17, 2021 •

edited

Loading

SparkQA commented Mar 17, 2021

sarutak commented Mar 18, 2021

HyukjinKwon commented Mar 18, 2021

HyukjinKwon commented Mar 18, 2021

sarutak commented Mar 18, 2021 •

edited

Loading

dongjoon-hyun commented Mar 18, 2021

[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli #31862

[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli #31862

Conversation

sarutak commented Mar 17, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon commented Mar 17, 2021

SparkQA commented Mar 17, 2021

SparkQA commented Mar 17, 2021

cloud-fan left a comment

Choose a reason for hiding this comment

sarutak commented Mar 17, 2021 • edited Loading

sarutak commented Mar 17, 2021 • edited Loading

SparkQA commented Mar 17, 2021

sarutak commented Mar 18, 2021

HyukjinKwon commented Mar 18, 2021

HyukjinKwon commented Mar 18, 2021

sarutak commented Mar 18, 2021 • edited Loading

dongjoon-hyun commented Mar 18, 2021

sarutak commented Mar 17, 2021 •

edited

Loading

sarutak commented Mar 17, 2021 •

edited

Loading

sarutak commented Mar 17, 2021 •

edited

Loading

sarutak commented Mar 18, 2021 •

edited

Loading