failed to unpack: : waiting for the reexec command failed

[Bean900]

Actual behavior
I am currently migrating my Dockerfiles to be build with Kaniko (V16 and V19 have been tried). The following error occurs in two of my Dockerfiles when starting in Cloud Foundry with cf push (Dockerimage build with kaniko: thebohne/kaniko:kaniko):

15:22:15.270: [CELL.0] Cell 7412a641-2c74-4f99-9cc0-788f7c4536fd failed to create container for instance 640101cd-af6c-467a-4b4e-d704: running image plugin create: pulling the image: unpacking layer `sha256:33db33d935721dfdf882423ee0e4d68410850032bc4e45779f7d746816710787`: failed to unpack: : waiting for the reexec command failed: unpacking-failed: opening target whiteout directory: open test: no such file or directory
15:22:15.270: [CELL.0] : exit status 1

Expected behavior
The expected behavior looks like this when building with Docker (docker build .) and starting with cf push (Dockerimage build with docker: thebohne/kaniko:docker):

15:49:29.176: [CELL.0] Cell f3551e37-379f-454a-81d3-389a4c6d12a1 successfully created container for instance 77902562-a6aa-4aaf-7659-d8b3
15:49:31.431: [APP/PROC/WEB.0] Exit status 143

To Reproduce
So I wrote a minimal Dockerfile:

FROM debian:stretch
RUN mkdir /test
RUN echo "TEST" > /test/testfile
RUN mv /test/testfile /usr/local/bin

The same error should occur with other base images as well.
I am running the following commands in my Jenkinspipeline (there should be no difference when running on your local maschine):

1. /kaniko/executor --destination=thebohne/kaniko:kaniko --context=${WORKSPACE} --dockerfile=src/main/docker/Dockerfile or with --cache=true
2. CF_DOCKER_PASSWORD=${DOCKER_PASSWORD} cf push ${NAME} --docker-image thebohne/kaniko:kaniko --docker-username ${USERNAME} -u process

EDIT
Here a little Testmatrix:

	Run with Docker	Run in Cloud Foundry
Build with Docker	✔️	✔️
Build with Kaniko	✔️	❌

There appears to be a connection between Cloud Foundry and Kaniko. Since there is a possibility that it can also be a bug in Cloud Foundry, I also created an issue there: cloudfoundry/grootfs#71

[tejal29]

@Bean900 I am not sure how we can help more with this.
Would it possible for you to send trace logs -v=trace to see if something jumps out.
We recently fixed a bug where parent dirs where not added.

Judging from this line here https://github.com/cloudfoundry/grootfs/blob/master/base_image_puller/unpacker/whiteout_handler_linux.go#L22
Looks for a whiteout file in test directory, kaniko did not add the whiteout directory.

We recently fixed a bug to add parent dirs for files
in commit e5585fd
However, this is not done for whiteouts.
Can you add the same code here in the whiteout block?https://github.com/GoogleContainerTools/kaniko/blob/master/pkg/snapshot/snapshot.go#L225

[Bean900]

Thanks for the answer :)
Hopefully this tracelog will help you (Run with Kaniko 0.22):
consoleText.txt

[MichaelChan-Pong]

Hi all, are there any plans to fix this if not already fixed? I've run into this issue recently and would love to see it resolved :)

[imjasonh]

Hi all, are there any plans to fix this if not already fixed? I've run into this issue recently and would love to see it resolved :)

Kaniko is effectively unstaffed at this time, so any improvements tend to come from motivated and capable contributors from the community. If you're interested in investigating and fixing this issue, I'll happily review and approve any PRs that get sent my way.