[SPARK-24372][build] Add scripts to help with preparing releases. #21515

vanzin · 2018-06-08T23:44:03Z

The "do-release.sh" script asks questions about the RC being prepared,
trying to find out as much as possible automatically, and then executes
the existing scripts with proper arguments to prepare the release. This
script was used to prepare the 2.3.1 release candidates, so was tested
in that context.

The docker version runs that same script inside a docker image especially
crafted for building Spark releases. That image is based on the work
by Felix C. linked in the bug. At this point is has been only midly
tested.

I also added a template for the vote e-mail, with placeholders for
things that need to be replaced, although there is no automation around
that for the moment. It shouldn't be hard to hook up certain things like
version and tags to this, or to figure out certain things like the
repo URL from the output of the release scripts.

The "do-release.sh" script asks questions about the RC being prepared, trying to find out as much as possible automatically, and then executes the existing scripts with proper arguments to prepare the release. This script was used to prepare the 2.3.1 release candidates, so was tested in that context. The docker version runs that same script inside a docker image especially crafted for building Spark releases. That image is based on the work by Felix C. linked in the bug. At this point is has been only midly tested. I also added a template for the vote e-mail, with placeholders for things that need to be replaced, although there is no automation around that for the moment. It shouldn't be hard to hook up certain things like version and tags to this, or to figure out certain things like the repo URL from the output of the release scripts.

SparkQA · 2018-06-08T23:48:58Z

Test build #91590 has finished for PR 21515 at commit 44454b6.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-06-11T23:53:18Z

Test build #91673 has finished for PR 21515 at commit 1366f23.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2018-06-12T06:24:18Z

dev/create-release/vote.tmpl

@@ -0,0 +1,64 @@
+Please vote on releasing the following candidate as Apache Spark version {version}.
+
+The vote is open until {deadline} and passes if a majority of at least 3 +1 PMC votes are cast.


nit: personally I find 3 PMC +1 votes more clear

The rules are a little bit more complicated than that. Updated.

felixcheung · 2018-06-12T06:25:47Z

dev/.rat-excludes

@@ -106,3 +106,4 @@ spark-warehouse
 structured-streaming/*
 kafka-source-initial-offset-version-2.1.0.bin
 kafka-source-initial-offset-future-version.bin
+vote.tmpl


even if rat doesn't check, isn't vote.tmpl packaged into the source release this way?

for example, .git is removed from release here https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L157

Are you saying this file should not be packaged in the source release? Not sure I see why that would be the case. There's a lot of stuff in .rat-excludes that is still packaged.

felixcheung · 2018-06-12T06:29:41Z

dev/create-release/spark-rm/Dockerfile

+WORKDIR /opt/spark-rm/output
+
+ARG UID
+RUN useradd -m -s /bin/bash -p spark-rm -u $UID spark-rm


does that mean the do-release script is run as the user "spark-rm"?
I thought it's generally best practice to gpg sign as yourself?

The gpg signature is completely unrelated to the user running the process.

Also, Docker being the weird thing it is, the user name in the container doesn't really matter much. It's running with the same UID as the host user who built the image, which in this case is expected the be the person preparing the release, and this will only really matter for files that are written back to the host's file system.

tgravescs · 2018-06-13T00:38:03Z

I guess we will need to update the release process website once these go in?

Its not clear to me how the docker stuff applies here, are we recommending to create the release inside docker or leaving it as optional? did you use it for 2.3.1 release?

vanzin · 2018-06-13T00:54:05Z

We could update the docs once these get in.

I did not use docker for 2.3.1 but I plan to give it a shot for 2.1.3. The goal of the docker path is to always build Spark in a known environment, so things like R packages and everything else are the same. It could also help folks who use other OSes (although Sameer used MacOS for 2.3.0 and it seemed to work fine, aside from some warnings from gnu tar when decompressing the tarballs on Linux).

SparkQA · 2018-06-13T02:10:23Z

Test build #91734 has finished for PR 21515 at commit 04f6371.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-06-14T17:29:13Z

FYI I'm playing with building 2.1.3 using the docker-based approach, and there are a few things that I need to adjust before that works. I hope to update this PR later today.

- Allow mounting a local JDK (2.1 wants 1.7). - Tweak profiles and builds to 2.1 - Restore support for Scala 2.10 - set up JAVA_HOME in the docker container I also expanded "dry run" mode a bit to build one of the binary packages, and also the documentation. The build documentation is now also kept in the output directory along with the other artifacts.

SparkQA · 2018-06-15T02:10:50Z

Test build #91869 has finished for PR 21515 at commit 48eda71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-06-18T15:00:27Z

dev/create-release/do-release.sh

+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/release-util.sh"
+
+while getopts "bn" opt; do


it would be nice to have a high level description in the script just saying this does a release which does things like tag, build, etc and pushes things to the asf spark repo.

tgravescs · 2018-06-18T16:12:56Z

dev/create-release/release-util.sh

+  # Check if the RC already exists, and if re-creating the RC, skip tag creation.
+  RELEASE_TAG="v${RELEASE_VERSION}-rc${RC_COUNT}"
+  SKIP_TAG=0
+  if check_for_tag "RELEASE_TAG"; then


this needs to be $RELEASE_TAG

tgravescs · 2018-06-18T16:36:52Z

not sure if this was something in my env but somehow it pushed next version as 2.2-3-snapshot.
https://github.com/apache/spark/commits/branch-2.2
I manually fixed it, and am building now but skipped the push of tag since it was already there. The build had failed due to JAVA_HOME not set.
If I reproduce I'll debug.

vanzin · 2018-06-18T16:51:04Z

pushed next version as 2.2-3-snapshot

Hmm must be something I introduced in the last changes (2.3.2 seems correct). Sorry about that.

failed due to JAVA_HOME not set

I think that might happen if you don't use the -j option I added, but let me double check.

vanzin · 2018-06-18T17:14:26Z

I was able to run without -j, no problems with JAVA_HOME...

SparkQA · 2018-06-18T21:00:23Z

Test build #92036 has finished for PR 21515 at commit 7fe133a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Seems like Maven Central updated something over the weekend, and Java 7 was having trouble connecting; needed some extra SSL-related options. I also made the build sequential, which makes it slower, but avoids some issues with parallel maven processes that I was hitting when building 2.1.

SparkQA · 2018-06-19T00:02:40Z

Test build #92044 has finished for PR 21515 at commit faab8ae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-06-19T01:06:39Z

Test build #92045 has finished for PR 21515 at commit 82ae00e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

- Override SBT_OPTS for Java 7 - Allow executins individual steps in docker - Fix docker image so that changing what to install also forces an "apt update" to happen.

- Some bugs I introduced in last commit. - Fix release-tag.sh with Java 1.7. - Expand dry run to tag and publish steps.

SparkQA · 2018-06-19T03:12:38Z

Test build #92051 has finished for PR 21515 at commit 5771354.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-06-19T06:27:12Z

Test build #92058 has finished for PR 21515 at commit a39933e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-06-19T07:05:02Z

Test build #92061 has finished for PR 21515 at commit b6d4c70.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-06-19T18:25:22Z

dev/create-release/release-util.sh

+  # Gather some user information.
+  export ASF_USERNAME=$(read_config "ASF user" "$LOGNAME")
+
+  GIT_NAME=$(git config user.name)


script silently fails is user.name not set

SparkQA · 2018-06-19T23:25:26Z

Test build #92097 has finished for PR 21515 at commit 745342d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-06-20T16:01:17Z

just a note I successfully used these scripts with docker with dry-run option with spark 2.2.2 rc1.

tgravescs · 2018-06-22T14:15:05Z

at this point should we just merge these and make add anything else that comes up in other prs ?

vanzin · 2018-06-22T16:22:19Z

SGTM.

tgravescs · 2018-06-22T17:35:20Z

ok, +1, will merge shortly

tgravescs · 2018-06-22T17:39:40Z

merged, did you have a jira to update the documents to match?

tgravescs · 2018-06-22T17:40:28Z

Also thanks again for doing these scripts, the docker was much easier then me trying to get all dependencies on rhel!

vanzin · 2018-06-22T17:44:36Z

I did not have a task filed for the doc updates.

cloud-fan · 2018-09-16T04:12:20Z

I hit an issue when using do-release-docker.sh to prepare 2.4.0 rc1

useradd: UID 0 is not unique
The command '/bin/sh -c useradd -m -s /bin/bash -p spark-rm -u $UID spark-rm' returned a non-zero code: 4

Any ideas about how to resolve it?

felixcheung · 2018-09-16T04:27:47Z

UID already exists?

cloud-fan · 2018-09-16T04:45:10Z

dev/create-release/do-release-docker.sh

+$GPG --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE"
+
+run_silent "Building spark-rm image with tag $IMGTAG..." "docker-build.log" \
+  docker build -t "spark-rm:$IMGTAG" --build-arg UID=$UID "$SELF/spark-rm"


So we need to do export UID=xxx before running this script?

got it. This is a system variable. So we can't run this script with root user...

cloud-fan · 2018-09-16T04:54:12Z

Problem solved. This script assumes the current user can run docker command without sudo, while it may not be true. But we can't run this script with root user as we can't add a root user in the docker container.

Following the steps in https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user solves it.

Add RAT exclude.

1366f23

felixcheung reviewed Jun 12, 2018

View reviewed changes

Update vote count wording.

04f6371

tgravescs reviewed Jun 18, 2018

View reviewed changes

Small fixes.

7fe133a

Marcelo Vanzin added 2 commits June 18, 2018 13:22

Another version number fix.

faab8ae

Un-hardcode dry run mode with docker.

82ae00e

Marcelo Vanzin added 2 commits June 18, 2018 18:44

More fixes.

a39933e

- Override SBT_OPTS for Java 7 - Allow executins individual steps in docker - Fix docker image so that changing what to install also forces an "apt update" to happen.

Even more fixes.

b6d4c70

- Some bugs I introduced in last commit. - Fix release-tag.sh with Java 1.7. - Expand dry run to tag and publish steps.

tgravescs reviewed Jun 19, 2018

View reviewed changes

Avoid silent failure if user.name is not set.

745342d

asfgit closed this in 4e7d867 Jun 22, 2018

vanzin deleted the SPARK-24372 branch June 22, 2018 21:29

cloud-fan reviewed Sep 16, 2018

View reviewed changes

		@@ -0,0 +1,64 @@
		Please vote on releasing the following candidate as Apache Spark version {version}.

		The vote is open until {deadline} and passes if a majority of at least 3 +1 PMC votes are cast.

[SPARK-24372][build] Add scripts to help with preparing releases. #21515

[SPARK-24372][build] Add scripts to help with preparing releases. #21515

Conversation

vanzin commented Jun 8, 2018

SparkQA commented Jun 8, 2018

SparkQA commented Jun 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented Jun 13, 2018

vanzin commented Jun 13, 2018

SparkQA commented Jun 13, 2018

vanzin commented Jun 14, 2018

SparkQA commented Jun 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented Jun 18, 2018

vanzin commented Jun 18, 2018

vanzin commented Jun 18, 2018

SparkQA commented Jun 18, 2018

SparkQA commented Jun 19, 2018

SparkQA commented Jun 19, 2018

SparkQA commented Jun 19, 2018

SparkQA commented Jun 19, 2018

SparkQA commented Jun 19, 2018

Choose a reason for hiding this comment

SparkQA commented Jun 19, 2018

tgravescs commented Jun 20, 2018

tgravescs commented Jun 22, 2018

vanzin commented Jun 22, 2018

tgravescs commented Jun 22, 2018

tgravescs commented Jun 22, 2018

tgravescs commented Jun 22, 2018

vanzin commented Jun 22, 2018

cloud-fan commented Sep 16, 2018 • edited Loading

felixcheung commented Sep 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Sep 16, 2018

cloud-fan commented Sep 16, 2018 •

edited

Loading