[SPARK-24547][K8S] Allow for building spark on k8s docker images without cache and don't forget to push spark-py container.

[rayburgemeestre]

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-24547

TL;DR from JIRA issue:

• First time I generated images for 2.4.0 Docker was using it's cache, so actually when running jobs, old jars where still in the Docker image. This produces errors like this in the executors:

java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 6155820641931972169, local class serialVersionUID = -3720498261147521051

• The second problem was that the spark container is pushed, but the spark-py container wasn't yet. This was just forgotten in the initial PR.

• A third problem I also ran into because I had an older docker was [K8S] Fix issue in 'docker-image-tool.sh' #21551 so I have not included a fix for that in this ticket.

How was this patch tested?

I've tested it on my own Spark on k8s deployment.

[felixcheung]

can you edit the title to add [k8s] and fix the text which is truncated?

[felixcheung]

ok to test

[SparkQA]

Test build #91966 has finished for PR 21555 at commit c17179b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ifilonenko]

LGTM. Thank you for this. @mccheah @foxish @erikerlandson for review before merge

[ifilonenko]

retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/299/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/299/

[ifilonenko]

retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/307/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/307/

[SparkQA]

Test build #92054 has finished for PR 21555 at commit c17179b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #92063 has finished for PR 21555 at commit c17179b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ifilonenko]

retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/330/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/330/

[SparkQA]

Test build #92103 has finished for PR 21555 at commit c17179b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ifilonenko]

Good to merge with tests success. @mccheah and @foxish please merge

[foxish]

Lgtm. Will merge. Thanks

[mccheah]

This change makes it such that using the tool forces building and pushing both Python and non-Python, but, what if the user wants to only build one to save time? I can imagine that being the case in something like a dev-CI workflow. Ideally the tool would allow one to be selective in managing either image or both.

[ifilonenko]

@mccheah Good note. As this is a blocker for other PRs. It is probably best to refactor the docker-image-tool.sh in a separate PR for that is the not the focus of this PR.

[foxish]

@mccheah, that is a good point but I agree with @ifilonenko that we can do it in a subsequent PR. I'm thinking we merge this as-is and I can try to get a follow-up PR here for dockerfile refactor.

[erikerlandson]

LGMT, I am OK to merge.

Most of the automated image build tooling I've seen is custom, but I agree w/ Matt that being able to selectively build is worth supporting, via a followup PR. It might make it easier to write custom logic using this as an abstraction.

[mccheah]

Yup feel free to merge and follow up separately.

[foxish]

Will follow-up. Thanks all for the comments! Merging to master.