Missing layers when building image with --cache-from

[akshatflx]

In one of the images I'm building with cache-from flag, the resultant image doesn't have all the expected layers. I'm using Docker version 19.03.12, build 48a66213fe

Here's how I found this issue -

1. Pull latest image from remote registry, run docker run <image> ls and check that all files are present.
2. Build the same image from the exact same Dockerfile, like so - DOCKER_BUILDKIT=1 docker build -t <imagename>:latest --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from=gcr.io/...<imagename>:latest .... Notice that all the build steps are CACHED according to the logs
3. Run docker run <image> ls again, on the newly built image. Notice that some of the files are missing (which are clearly added as per Dockerfile instructions).
4. On doing docker inspect on both the images, I found that the newly built image doesn't have all the expected layers. Out of 14 instructions in the Dockerfile, it only contains the output of the first 6. New image is smaller in size as well.

Here's a log file containing the output of above steps -
BuildLogs.txt

Some more tests/observations -

• Tried to chain this image into smaller ones. Like separating the first 6 & the remaining instructions into 2 different Dockerfiles (tried splitting into 6-8, 5-9, 7-7 as well) - Didn't make any difference, all layers were missing after the 6th instruction.
• Re-ordering some of the instructions. I swapped instructions 6 & 7. This time the image contained the output of instruction#6, but not the 7th one. So overall it included the same number of instructions.

[adamf]

+1 to this issue - we're seeing it as well. It breaks caching for us.

[tonistiigi]

seems to match description of #1568

[akshatflx]

@tonistiigi okay, I can try with Docker version 19.03.13-beta2 to see if it fixes.

Should the image in cache source also be built with the new version, or can I test it only while using --cache-from?

[tonistiigi]

You do need to export cache again.

[gordz]

I've been able to reproduce a similar problem with Buildkit, running on docker version 19.03.8, build afacb8b7f0, with an example at https://github.com/gordz/moby-buildkit-cache-from-bug with Dockerfiles' and a script that reproduces the problem.

I've included a test.sh script in the above repo that will:

• clear out and prune local images and containers, to start from a clean state.
• build the parent image, and push to a registry
• clear local state
• build a child image using secrets, with inline cache layers enabled, inheriting from the parent image. Push to a registry.
• list files in the child image
• clear local state
• re-build the child image, using --cache-from to take advantage of the layer cache from the previous build.
• list files in the child image

On the final rebuild, the resulting image is missing several layers, confirmed by running an ls inside the running container. We expect to see several files that should have been added by Dockerfile-child, which are missing.

If you plan to run the script, please be aware that it will remove local containers and images, and perform a prune, and you will need to replace the 2* FIXME placeholders with your own registry.

Attached are logs from running the test I've scripted up - notice that in the final ls a bunch of files are missing (child-1.txt, child-2.txt, child-3.txt, child-4.txt)

build.log

[akshatflx]

With Docker version 19.03.13-beta2 it works fine.

[gordz]

@akshatflx thanks, I've tested with 19.03.13-beta2 and my tests now pass.