run `analyze`, `restore`, and `export` in an ephemeral image #631

natalieparellano · 2020-05-15T16:55:18Z

Fixes #528

Not ready to be merged until the "lifecycle image" is actually published. The const lifecycleImageRepo must be changed to point to buildpacksio.

codecov · 2020-05-15T16:57:40Z

Codecov Report

Merging #631 into master will increase coverage by 0.10%.
The diff coverage is 75.83%.

@@            Coverage Diff             @@
##           master     #631      +/-   ##
==========================================
+ Coverage   70.52%   70.61%   +0.10%     
==========================================
  Files          67       67              
  Lines        4687     4763      +76     
==========================================
+ Hits         3305     3363      +58     
- Misses       1063     1075      +12     
- Partials      319      325       +6

Flag	Coverage Δ
#os_linux	`73.29% <79.46%> (+0.10%)`	⬆️
#os_macos	`69.36% <79.46%> (+0.16%)`	⬆️
#os_windows	`69.18% <75.83%> (+0.12%)`	⬆️
#unit	`70.61% <75.83%> (+0.10%)`	⬆️

acceptance/acceptance_test.go

build.go

build_test.go

internal/build/build.go

internal/build/phase_config_provider_test.go

internal/build/phases.go

natalieparellano · 2020-05-15T17:49:25Z

internal/build/phases.go

+		return "", errors.Wrapf(err, "writing stack.toml tempfile: %s", stackFile.Name())
+	}
+
+	// Override umask


umask (needed for Linux) and symlinks (needed for Mac) requirements were uncovered in our acceptance tests for volume mounts.

natalieparellano · 2020-05-15T18:34:16Z

This is blocked on buildpacks/lifecycle#294

natalieparellano · 2020-05-15T20:29:52Z

acceptance/acceptance_test.go

+								err := cmd.Wait()
+								h.AssertNotNil(t, err)
+
+								h.AssertContains(t, buf.String(), "[detector]")


I removed a check that the log output contains a warning if no lifecycle image is available, because this is now unit tested.

build_test.go

natalieparellano · 2020-05-15T20:32:22Z

build_test.go

@@ -194,7 +126,7 @@ func testBuild(t *testing.T, when spec.G, it spec.S) {
 			it("is required", func() {
 				h.AssertError(t, subject.Build(context.TODO(), BuildOptions{
 					Image:   "",
-					Builder: builderName,


This var name seemed too ambiguous and confused me a couple of times...

build_test.go

natalieparellano · 2020-05-15T20:46:27Z

GHA seem stuck :/

You can see here here and here that everything went green two commits ago, and the changes since then are quite minimal.

Hopefully GHA will get un-stuck when I push additional commits per feedback.

natalieparellano · 2020-05-18T15:42:01Z

acceptance/acceptance_test.go

+							})
+						})
+
+						when("builder is trusted", func() {


On thinking about this further, maybe we don't need this test now that this is unit tested in build_test.go... I do think we'll want at least one "happy path" that runs and verifies the 5 phases (as all the other tests, since they are run without --publish, will use the creator).

I am going to remove this test for now. I updated the test assertions in the --publish case to verify that the 5 phases were used.

build.go

internal/build/build.go

internal/commands/build.go

internal/build/phases.go

acceptance/acceptance_test.go

build.go

internal/build/build.go

internal/build/phases.go

Only run creator if builder is trusted. For running the acceptance tests, use TEST_TRUST_BUILDER env variable to determine if the builder should be trusted. Signed-off-by: Natalie Arellano <[email protected]>

See if this fixes the error on the Linux runner. Signed-off-by: Natalie Arellano <[email protected]>

Signed-off-by: Natalie Arellano <[email protected]>

- Update variable declarations to be the way they were previously Signed-off-by: Natalie Arellano <[email protected]>

Signed-off-by: Natalie Arellano <[email protected]>

It makes the logic around creator support + trusted builders + publish easier to test and reason about. Signed-off-by: Natalie Arellano <[email protected]>

It's not used in other scopes, so don't make it global Signed-off-by: Natalie Arellano <[email protected]>

Signed-off-by: Natalie Arellano <[email protected]>

natalieparellano · 2020-05-19T20:47:02Z

Updated per feedback. The compat tests are failing due to something on master unrelated to this...

Still waiting on buildpacksio/lifecycle:0.7.5 to be published before we can un-draft this.

Signed-off-by: Natalie Arellano <[email protected]>

build.go

Signed-off-by: Natalie Arellano <[email protected]>

internal/builder/lifecycle.go

yaelharel · 2020-05-19T23:39:01Z

build.go

+		return c.lifecycle.Execute(ctx, lifecycleOpts)
+	}
+
+	lifecycleImageSupported := !ephemeralBuilder.LifecycleDescriptor().Info.Version.LessThan(semver.MustParse("0.7.5"))


I think it should be 0.7.4.

At this time we will start creating lifecycle images starting at 0.7.5. I wasn't part of the discussion but I'm not sure it makes sense to provide older lifecycle images. A quick look at the current suggested builders the all use 0.7.5 with the exception of the heroku builder (version 0.6.2).

I don't think we intend to publish a 0.7.4 lifecycle image - that version of the lifecycle is broken when you use the 5 phases with --publish (under certain conditions) and that is the exact scenario when we want to use the lifecycle image.

If someone provides lifecycle 0.7.4 and they trust their builder or they are not publishing, then great - we'll use the creator. But if not then we'll warn them (since lifecycle image is not available) and we'll use that lifecycle with the 5 phases. Their build might fail due to the bug, but sadly there isn't anything we can do about that.

acceptance/testdata/pack_fixtures/report_output.txt

jromero · 2020-05-20T05:13:51Z

✔️ Accepted

`pack build` with builder containing old lifecycle

Output

$ ./out/pack inspect-builder cnbs/sample-builder:alpine
Inspecting builder: cnbs/sample-builder:alpine
...
Lifecycle:
  Version: 0.7.2
  Buildpack API: 0.2
  Platform API: 0.3
...


$ ./out/pack build my-app-586 -B cnbs/sample-builder:alpine -p ~/dev/buildpacks/samples/apps/bash-script/ -v
Using project descriptor located at /Users/javier.romero/dev/buildpacks/samples/apps/bash-script/project.toml
Pulling image index.docker.io/cnbs/sample-builder:alpine
alpine: Pulling from cnbs/sample-builder
Digest: sha256:b12fef6427956b670aa3b9ff91daee19b927200dbf6bbb6171bf98be3360e8d7
Status: Image is up to date for cnbs/sample-builder:alpine
Selected run image cnbs/sample-stack-run:alpine
Pulling image cnbs/sample-stack-run:alpine
alpine: Pulling from cnbs/sample-stack-run
Digest: sha256:ff5952abb5476176ddd68790c1ba69ef0114f701c0565acc94ad067100571449
Status: Image is up to date for cnbs/sample-stack-run:alpine
Adding buildpack samples/bash-script version 0.0.1 to builder
Setting custom order
Saving builder with the following buildpacks:
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
Warning: Lifecycle 0.7.2 does not have an associated lifecycle image.
Using build cache volume pack-cache-e3d7b7b3e3c7.build
===> DETECTING
[detector] ======== Output: samples/[email protected] ========
[detector] ---> Hello Bash Script buildpack
[detector] 
[detector] Here are the contents of the current working directory:
[detector] .:
[detector] total 16
[detector] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:06 .
[detector] drwxr-xr-x    1 root     root          4096 May 20 05:06 ..
[detector] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[detector] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
[detector] ======== Results ========
[detector] pass: samples/[email protected]
[detector] Resolving plan... (try #1)
[detector] samples/bash-script 0.0.1
===> ANALYZING
[analyzer] Analyzing image "68706b2783e5157738e2a0a009430aabaca7dedb8ec6cc5e824194d7ffe3e99d"
===> RESTORING
===> BUILDING
[builder] ---> Hello World buildpack
[builder] ---> Hello Bash Script buildpack
[builder] 
[builder] Here are the contents of the current working directory:
[builder] .:
[builder] total 16
[builder] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:06 .
[builder] drwxr-xr-x    1 root     root          4096 May 20 05:06 ..
[builder] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[builder] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
===> EXPORTING
[exporter] no project metadata found at path './project-metadata.toml', project metadata will not be exported
[exporter] Reusing layers from image with id '68706b2783e5157738e2a0a009430aabaca7dedb8ec6cc5e824194d7ffe3e99d'
[exporter] Writing tarball for layer "launcher"
[exporter] Adding layer 'launcher'
[exporter] Layer 'launcher' SHA: sha256:c189102796b1e6d38ccb28796cf6b65e1a5541f619367d0a7cd479785bf21a7c
[exporter] Layer 'app' SHA: sha256:4c6ff16e28deaf437a1abe5c926cdfb8afcd2c31ea544f6992ed7b665965b3f4
[exporter] Reusing 1/1 app layer(s)
[exporter] Writing tarball for layer "config"
[exporter] Reusing layer 'config'
[exporter] Layer 'config' SHA: sha256:ed89d623eb7cf47a03403af0b6d3723a967ca4f73985b97056246193d29eb3b0
[exporter] *** Images (beb3e2343e11):
[exporter]       index.docker.io/library/my-app-586:latest
[exporter] 
[exporter] *** Image ID: beb3e2343e11028a5be3a10c5fc17031be3a6eea92235eeceea2b16f48f8938d
Successfully built image my-app-586

`pack build ... --publish` w/ builder containing lifecycle 0.7.5

Output

$ ./out/pack inspect-builder sample-builder
Inspecting builder: sample-builder
...
Lifecycle:
  Version: 0.7.5
  Buildpack API: 0.2
  Platform API: 0.3
...

$ ./out/pack build my-app-586 -B sample-builder -p ~/dev/buildpacks/samples/apps/bash-script/ -v --publish
Using project descriptor located at /Users/javier.romero/dev/buildpacks/samples/apps/bash-script/project.toml
Selected run image cnbs/sample-stack-run:alpine
Adding buildpack samples/bash-script version 0.0.1 to builder
Setting custom order
Saving builder with the following buildpacks:
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
Pulling image buildpacksio/lifecycle:0.7.5
0.7.5: Pulling from buildpacksio/lifecycle
Digest: sha256:246e6c365f53306583286bcb307b6dd7700e66188d4e9aabf1e4d3139af0f8e1
Status: Image is up to date for buildpacksio/lifecycle:0.7.5
Using build cache volume pack-cache-e3d7b7b3e3c7.build
===> DETECTING
[detector] ======== Output: samples/[email protected] ========
[detector] ---> Hello Bash Script buildpack
[detector] 
[detector] Here are the contents of the current working directory:
[detector] .:
[detector] total 16
[detector] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:11 .
[detector] drwxr-xr-x    1 root     root          4096 May 20 05:11 ..
[detector] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[detector] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
[detector] ======== Results ========
[detector] pass: samples/[email protected]
[detector] Resolving plan... (try #1)
[detector] samples/bash-script 0.0.1
===> ANALYZING
[analyzer] Previous image with name "index.docker.io/library/my-app-586:latest" not found
===> RESTORING
===> BUILDING
[builder] ---> Hello World buildpack
[builder] ---> Hello Bash Script buildpack
[builder] 
[builder] Here are the contents of the current working directory:
[builder] .:
[builder] total 16
[builder] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:11 .
[builder] drwxr-xr-x    1 root     root          4096 May 20 05:11 ..
[builder] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[builder] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
===> EXPORTING
[exporter] no project metadata found at path './project-metadata.toml', project metadata will not be exported
[exporter] Writing tarball for layer "launcher"
[exporter] Adding layer 'launcher'
[exporter] Layer 'launcher' SHA: sha256:32608bc6d97e0193edb0360555b3e08dc6dfe1187833d8548cdd9e662213935b
[exporter] Layer 'app' SHA: sha256:4c6ff16e28deaf437a1abe5c926cdfb8afcd2c31ea544f6992ed7b665965b3f4
[exporter] Adding 1/1 app layer(s)
[exporter] Writing tarball for layer "config"
[exporter] Adding layer 'config'
[exporter] Layer 'config' SHA: sha256:9790af0612365f9299ec555ed3b78aa1d71aeca28676b9dc86ba67934f629fbc
[exporter] *** Images (sha256:1b2790f001373b10c08425ba289b0d1bb5674299dbdc27ee81562fb0121447b2):
[exporter]       index.docker.io/library/my-app-586:latest

`pack build ... --publish` w/ builder containing lifecycle 0.7.5

Output

$ ./out/pack inspect-builder sample-builder
Inspecting builder: sample-builder
...
Lifecycle:
  Version: 0.7.5
  Buildpack API: 0.2
  Platform API: 0.3
...

$ ./out/pack build my-app-586 -B sample-builder -p ~/dev/buildpacks/samples/apps/bash-script/ -v
Using project descriptor located at /Users/javier.romero/dev/buildpacks/samples/apps/bash-script/project.toml
Selected run image cnbs/sample-stack-run:alpine
Pulling image cnbs/sample-stack-run:alpine
alpine: Pulling from cnbs/sample-stack-run
Digest: sha256:ff5952abb5476176ddd68790c1ba69ef0114f701c0565acc94ad067100571449
Status: Image is up to date for cnbs/sample-stack-run:alpine
Adding buildpack samples/bash-script version 0.0.1 to builder
Setting custom order
Saving builder with the following buildpacks:
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
Using build cache volume pack-cache-e3d7b7b3e3c7.build
===> CREATING
[creator] ---> DETECTING
[creator] ======== Output: samples/[email protected] ========
[creator] ---> Hello Bash Script buildpack
[creator] 
[creator] Here are the contents of the current working directory:
[creator] .:
[creator] total 16
[creator] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:12 .
[creator] drwxr-xr-x    1 root     root          4096 May 20 05:12 ..
[creator] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[creator] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
[creator] ======== Results ========
[creator] pass: samples/[email protected]
[creator] Resolving plan... (try #1)
[creator] samples/bash-script 0.0.1
[creator] ---> ANALYZING
[creator] Analyzing image "beb3e2343e11028a5be3a10c5fc17031be3a6eea92235eeceea2b16f48f8938d"
[creator] ---> RESTORING
[creator] ---> BUILDING
[creator] ---> Hello World buildpack
[creator] ---> Hello Bash Script buildpack
[creator] 
[creator] Here are the contents of the current working directory:
[creator] .:
[creator] total 16
[creator] drwxr-xr-x    2 cnb      cnb           4096 May 20 05:12 .
[creator] drwxr-xr-x    1 root     root          4096 May 20 05:12 ..
[creator] -rwxr-xr-x    1 cnb      cnb            738 Apr  6 23:13 app.sh
[creator] -rw-r--r--    1 cnb      cnb            202 Apr  6 23:13 project.toml
[creator] ---> EXPORTING
[creator] no project metadata found at path './project-metadata.toml', project metadata will not be exported
[creator] Reusing layers from image with id 'beb3e2343e11028a5be3a10c5fc17031be3a6eea92235eeceea2b16f48f8938d'
[creator] Writing tarball for layer "launcher"
[creator] Adding layer 'launcher'
[creator] Layer 'launcher' SHA: sha256:32608bc6d97e0193edb0360555b3e08dc6dfe1187833d8548cdd9e662213935b
[creator] Layer 'app' SHA: sha256:4c6ff16e28deaf437a1abe5c926cdfb8afcd2c31ea544f6992ed7b665965b3f4
[creator] Reusing 1/1 app layer(s)
[creator] Writing tarball for layer "config"
[creator] Reusing layer 'config'
[creator] Layer 'config' SHA: sha256:ed89d623eb7cf47a03403af0b6d3723a967ca4f73985b97056246193d29eb3b0
[creator] *** Images (68706b2783e5):
[creator]       index.docker.io/library/my-app-586:latest
[creator] 
[creator] *** Image ID: 68706b2783e5157738e2a0a009430aabaca7dedb8ec6cc5e824194d7ffe3e99d

acceptance/testdata/pack_fixtures/report_output.txt

jromero · 2020-05-20T05:21:03Z

build.go

+		return c.lifecycle.Execute(ctx, lifecycleOpts)
+	}
+
+	lifecycleImageSupported := !ephemeralBuilder.LifecycleDescriptor().Info.Version.LessThan(semver.MustParse("0.7.5"))


At this time we will start creating lifecycle images starting at 0.7.5. I wasn't part of the discussion but I'm not sure it makes sense to provide older lifecycle images. A quick look at the current suggested builders the all use 0.7.5 with the exception of the heroku builder (version 0.6.2).

internal/builder/lifecycle.go

Signed-off-by: Javier Romero <[email protected]>

"ANALYZING" etc. is also printed to the logs when creator runs, so we need to check the binary name. Signed-off-by: Natalie Arellano <[email protected]>

natalieparellano force-pushed the feature/528-published-lifecycle-image branch from 94b2abd to c44cbe0 Compare May 15, 2020 17:14