[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

HyukjinKwon · 2020-01-23T07:10:21Z

What changes were proposed in this pull request?

This PR backports #27307

This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list.

Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under .m2.

Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing)
Maven Central (default) seems less stable but synced quickly.

We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo.

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines 2111 to 2118 in abf759a

    
           val ADDITIONAL_REMOTE_REPOSITORIES = 
        
             buildConf("spark.sql.additionalRemoteRepositories") 
        
               .doc("A comma-delimited string config of the optional additional remote Maven mirror " + 
        
                 "repositories. This is only used for downloading Hive jars in IsolatedClientLoader " + 
        
                 "if the default Maven Central repo is unreachable.") 
        
               .stringConf 
        
               .createWithDefault( 
        
                 "https://maven-central.storage-download.googleapis.com/repos/central/data/")

Currently, we have the hard-corded repository in sbt-pom-reader and this seems overwriting Maven's existing resolver by the same ID central with http:// when initially the pom file is ported into SBT instance. This uses http:// which latently Maven Central disallowed (see [SPARK-30534][INFRA] Use mvn in dev/scalastyle #27242)

My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it seems using central with https properly. See also [SPARK-30601][BUILD] Add a Google Maven Central as a primary repository #27307 (comment).

I double checked that we use https properly from the SBT build as well:
```
[debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ...
[debug] 	public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom
[debug] 	public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1
```
This was fixed by adding the same repo ([SPARK-30572][BUILD] Add a fallback Maven repository #27281), central_without_mirror, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback.

While I am here, I fix another issue. Github Action at [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS.

org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve     (DefaultPluginDependenciesResolver.java:131)

mvn-scalafmt exists in Maven central:

$ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    ...

whereas not in GCS mirror:

$ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom

<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>%

In this PR, simply make both repos accessible by adding to pluginRepositories.

Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second)

Why are the changes needed?

To make the build and Github Action more stable.

Does this PR introduce any user-facing change?

No, dev only change.

How was this patch tested?

I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3).

This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see apache#27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also apache#27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (apache#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at apache#27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) To make the build and Github Action more stable. No, dev only change. I roughly checked local and PR against my fork (#2 and #3). Closes apache#27307 from HyukjinKwon/SPARK-30572. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon · 2020-01-23T07:11:01Z

.github/workflows/branch-2.4.yml

-        # In other words, in GitHub Action environment, `central` is mirrored by `Google Maven Central` first.
-        # If `Google Maven Central` doesn't provide the artifact due to its slowness, `central_without_mirror` will be used.
-        # Note that we aim to achieve the above while keeping the existing behavior of non-`GitHub Action` environment unchanged.
-        echo "<settings><mirrors><mirror><id>google-maven-central</id><name>GCS Maven Central mirror</name><url>https://maven-central.storage-download.googleapis.com/repos/central/data/</url><mirrorOf>central</mirrorOf></mirror></mirrors></settings>" > ~/.m2/settings.xml
        ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Pscala-${{ matrix.scala }} -P${{ matrix.hadoop }} -Phadoop-cloud install


This line was the conflict so there isn't an actual conflict. I just opened it to make sure builds pass.

Thank you for making a separate PR. Yes. It will be safer.

HyukjinKwon · 2020-01-23T07:11:10Z

cc @dongjoon-hyun

dongjoon-hyun

+1, LGTM.

SparkQA · 2020-01-23T08:05:02Z

Test build #117284 has finished for PR 27335 at commit 711852d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-01-23T09:22:02Z

retest this please

SparkQA · 2020-01-23T12:04:19Z

Test build #117294 has finished for PR 27335 at commit 711852d.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-01-23T16:56:05Z

All tests passed and the R failure is a known incoming feasibility test flakiness. cc @viirya

* checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : 
  dims [product 26] do not match the length of object [0]

dongjoon-hyun · 2020-01-23T16:56:13Z

Merged to branch-2.4.

…ository ### What changes were proposed in this pull request? This PR backports #27307 This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see #27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also #27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) ### Why are the changes needed? To make the build and Github Action more stable. ### Does this PR introduce any user-facing change? No, dev only change. ### How was this patch tested? I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3). Closes #27335 from HyukjinKwon/tmp. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2020-01-23T16:57:32Z

Thank you, @HyukjinKwon !

viirya · 2020-01-23T18:22:37Z

Requested a help from CRAN. They replied it is fixed now.

…

On Thu, Jan 23, 2020, 08:57 Dongjoon Hyun ***@***.***> wrote: All tests passed and the R failure is a known incoming feasibility test flakiness. cc @viirya <https://github.com/viirya> * checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 26] do not match the length of object [0] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27335?email_source=notifications&email_token=AAAQZ52OJ7O5MVNDJLCKINTQ7HEARA5CNFSM4KKRYFFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYBQKY#issuecomment-577771563>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQZ52HA52S555LWPHJ2ETQ7HEARANCNFSM4KKRYFFA> .

HyukjinKwon · 2020-01-24T04:08:15Z

Thanks @dongjoon-hyun and @viirya!

HyukjinKwon commented Jan 23, 2020

View reviewed changes

dongjoon-hyun approved these changes Jan 23, 2020

View reviewed changes

dongjoon-hyun added the BUILD label Jan 23, 2020

dongjoon-hyun closed this Jan 23, 2020

HyukjinKwon deleted the tmp branch March 3, 2020 01:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

HyukjinKwon commented Jan 23, 2020

HyukjinKwon Jan 23, 2020

dongjoon-hyun Jan 23, 2020

HyukjinKwon commented Jan 23, 2020

dongjoon-hyun left a comment

SparkQA commented Jan 23, 2020

HyukjinKwon commented Jan 23, 2020

SparkQA commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020 •

edited

Loading

dongjoon-hyun commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020

viirya commented Jan 23, 2020 via email

HyukjinKwon commented Jan 24, 2020

	val ADDITIONAL_REMOTE_REPOSITORIES =
	buildConf("spark.sql.additionalRemoteRepositories")
	.doc("A comma-delimited string config of the optional additional remote Maven mirror " +
	"repositories. This is only used for downloading Hive jars in IsolatedClientLoader " +
	"if the default Maven Central repo is unreachable.")
	.stringConf
	.createWithDefault(
	"https://maven-central.storage-download.googleapis.com/repos/central/data/")

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

Conversation

HyukjinKwon commented Jan 23, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon Jan 23, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jan 23, 2020

Choose a reason for hiding this comment

HyukjinKwon commented Jan 23, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

SparkQA commented Jan 23, 2020

HyukjinKwon commented Jan 23, 2020

SparkQA commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020 • edited Loading

dongjoon-hyun commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020

viirya commented Jan 23, 2020 via email

HyukjinKwon commented Jan 24, 2020

dongjoon-hyun commented Jan 23, 2020 •

edited

Loading