Skip to content
This repository has been archived by the owner on Nov 22, 2024. It is now read-only.

Updated Spark base image to use Alpine; Fixed Spark build script #578

Merged
merged 2 commits into from
Jul 14, 2020

Conversation

yuchaoran2011
Copy link
Contributor

@yuchaoran2011 yuchaoran2011 commented Jul 13, 2020

What changes were proposed in this pull request?

Addressed:
https://github.com/lightbend/cloudflow-internal-ops/issues/63
#468

The Spark build script no longer builds with R profile. It doesn't create a Github release either. We'll move away from maintaining a Lightbend fork eventually.

Why are the changes needed?

Alpine is a lighter-weight image that has no known security vulnerabilities. The build script was also broken.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I ported Stavros' changes at SPARK-24748 to a separate branch named "lightbend-2.4.5". See diffs here.

Then I tested with call-record-aggregator app because it uses Spark runtime. By setting in its build.sbt:

cloudflowSparkBaseImage := Some("lightbend/spark:2.1.0-SNAPSHOT-cloudflow-spark-2.4.5-scala-2.12")

which is an image built using the script contained in this PR, along with the Spark fork mentioned above, the executor pods start up successfully.

With the old Spark image lightbend/spark:2.0.5-cloudflow-spark-2.4.5-scala-2.12, the executor pods couldn't pull the image from GCR because they didn't have the same service account as the driver pods:

$ k -n call-record-aggregator get po
NAME                                                         READY   STATUS         RESTARTS   AGE
call-record-aggregator-cdr-aggregator-1594750835890-exec-1   0/1     ErrImagePull   0          111s
call-record-aggregator-cdr-aggregator-driver                 1/1     Running        0          2m23s
call-record-aggregator-cdr-generator1-1594750836384-exec-1   0/1     ErrImagePull   0          111s
call-record-aggregator-cdr-generator1-driver                 1/1     Running        0          2m23s
call-record-aggregator-cdr-generator2-1594750835093-exec-1   0/1     ErrImagePull   0          113s
call-record-aggregator-cdr-generator2-driver                 1/1     Running        0          2m25s
call-record-aggregator-cdr-ingress-c45b677c9-jjkfx           1/1     Running        0          2m40s
call-record-aggregator-console-egress-5f5cb59c44-hf7rt       1/1     Running        0          2m39s
call-record-aggregator-error-egress-649897b6f8-8689t         1/1     Running        0          2m40s
call-record-aggregator-split-68d98cd56f-r9ksw                1/1     Running        0          2m40s

Now with the new image, the executor pods can run:

$ k -n call-record-aggregator get po
NAME                                                         READY   STATUS    RESTARTS   AGE
call-record-aggregator-cdr-aggregator-1594750370482-exec-1   1/1     Running   0          2m25s
call-record-aggregator-cdr-aggregator-driver                 1/1     Running   0          2m57s
call-record-aggregator-cdr-generator1-1594750370691-exec-1   1/1     Running   0          2m23s
call-record-aggregator-cdr-generator1-driver                 1/1     Running   0          2m57s
call-record-aggregator-cdr-generator2-1594750369600-exec-1   1/1     Running   0          2m25s
call-record-aggregator-cdr-generator2-driver                 1/1     Running   0          2m58s
call-record-aggregator-cdr-ingress-66cb57877d-hmw46          1/1     Running   0          3m13s
call-record-aggregator-console-egress-7854dbf96f-4bd2d       1/1     Running   0          3m13s
call-record-aggregator-error-egress-6849c69b79-dxblv         1/1     Running   0          3m13s
call-record-aggregator-split-6555449494-v7kx6                1/1     Running   0          3m13s

And I've verified that they have the right service account.

@yuchaoran2011 yuchaoran2011 self-assigned this Jul 13, 2020
@yuchaoran2011 yuchaoran2011 marked this pull request as ready for review July 14, 2020 18:25
Copy link
Contributor

@RayRoestenburg RayRoestenburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants