Running kaniko from inside a container doesn't seem to work

[xinau]

HI.
I have the following problem:
I'm unable to build my Dockerfile with kaniko from inside the kaniko container.

$ tree
.
├── Dockerfile
└── main.go
$ docker run --rm -it --entrypoint="/busybox/sh" -v $PWD:/workspace gcr.io/kaniko-project/executor:debug
/ # /kaniko/executor -d test.hub

Stops at with following line and raising 0 errors

...
INFO[0027] Taking snapshot of full filesystem...        
INFO[0029] Executing 0 build triggers                   
INFO[0029] cmd: Add [.]                                 
INFO[0029] dest: /src                                   
INFO[0029] cmd: copy [.]                                
INFO[0029] dest: /src                                   
INFO[0029] Creating directory /src                      
INFO[0029] Copying file /workspace/Dockerfile to /src/Dockerfile 
INFO[0029] Copying file /workspace/main.go to /src/main.go 
INFO[0029] cmd: /bin/sh                                 
INFO[0029] args: [-c cd /src && go build -o app]        
Killed

Doing the same without using /busybox/sh does work

$ docker run --rm -it --entrypoint="/kaniko/executor" -v $PWD:/workspace gcr.io/kaniko-project/executor:debug -d test.hub

The contents of the Dockerfile and app.go.

FROM golang:1.10.3-alpine AS build
ADD . /src
RUN cd /src && go build -o app

FROM alpine
WORKDIR /app
COPY --from=build /src/app /app/
CMD [ "./app" ]

package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

[JensGutermuth]

+1 on this. Everything after 6fe9ea4 is completely broken. The only difference from the failing log below and the successful one is the commit kaniko was built from.

6fe9ea4

$ kaniko-executor --dockerfile Dockerfile --context `pwd`/images/$CI_JOB_NAME --verbosity warn --single-snapshot --destination "[REDACTED]"
2018/08/01 12:40:59 No matching credentials were found, falling back on anonymous
+ apt-get update -qq
+ apt-get upgrade -yqq
+ apt-get install -y -qq --no-install-recommends rsync
debconf: delaying package configuration, since apt-utils is not installed
+ sh -c rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* /usr/share/doc/* /usr/share/man/*/*
Killed
ERROR: Job failed: exit code 137

8cad6d0

$ kaniko-executor --dockerfile Dockerfile --context `pwd`/images/$CI_JOB_NAME --verbosity warn --single-snapshot --destination "[REDACTED]"
2018/08/01 13:58:17 No matching credentials were found, falling back on anonymous
+ apt-get update -qq
+ apt-get upgrade -yqq
+ apt-get install -y -qq --no-install-recommends rsync
debconf: delaying package configuration, since apt-utils is not installed
+ sh -c rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* /usr/share/doc/* /usr/share/man/*/*
2018/08/01 13:58:35 pushed blob sha256:4f065d0d2edfbcbbfc404bd7fe369e45101adec2f36444994c2cf7e0005242c9
2018/08/01 13:58:35 pushed blob sha256:7c40dfdae5ca05848deee472dbe664058fdef1bcaac831594aa34bc2ed72f5b5
2018/08/01 13:58:36 pushed blob sha256:7d1463d31d7e5ad679ea175cd72afede858628ca49598925a17410e452d5ccec
2018/08/01 13:58:36 [REDACTED]: digest: sha256:9ed4b483684cc4b6586f5fb51f4386835a202277cc22a6113eaef447fb7660cd size: 592
Job succeeded

[dlorenc]

I was able to just build the first example dockerfile fine at head. I stuck the Dockerfile and main.go into a folder named foo:

$ ls foo
Dockerfile	main.go

$ cat foo/Dockerfile
FROM golang:1.10.3-alpine AS build
ADD . /src
RUN cd /src && go build -o app

FROM alpine
WORKDIR /app
COPY --from=build /src/app /app/
CMD [ "./app" ]

$ cat foo/main.go
package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

Then ran:

/run_in_docker.sh Dockerfile $(pwd)/foo $IMG

From the root of kaniko and everything worked.

Do you have any other repro instructions?

@Delphinator can you share the Dockerfile you used?

[JensGutermuth]

@dlorenc Are you sure your local gcr.io/kaniko-project/executor:latest image is up-to-date? I have these. run_in_docker.sh uses gcr.io/kaniko-project/executor:latest, which currently has the id 96d976b13c08.

$ docker images | grep gcr.io/kaniko-project
gcr.io/kaniko-project/executor   debug                   d2d2b30d6618        21 hours ago        54.5MB
gcr.io/kaniko-project/executor   debug-v0.3.0            d2d2b30d6618        21 hours ago        54.5MB
gcr.io/kaniko-project/executor   latest                  96d976b13c08        21 hours ago        53.6MB
gcr.io/kaniko-project/executor   v0.3.0                  96d976b13c08        21 hours ago        53.6MB
gcr.io/kaniko-project/executor   debug-v0.2.0            f2005807912a        3 weeks ago         41.1MB
gcr.io/kaniko-project/executor   v0.2.0                  0b9efc52ab49        3 weeks ago         40.2MB

I used the following Dockerfile, although I so far had all Dockerfile I've tried fail.

FROM debian:stable-slim

RUN set -ex; \
    \
    apt-get update -qq > /dev/null; apt-get upgrade -yqq > /dev/null; \
    apt-get install -y -qq --no-install-recommends \
        rsync \
        > /dev/null; \
    sh -c 'rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* /usr/share/doc/* /usr/share/man/*/*'

CMD rsync --daemon --no-detach --log-file /dev/stdout

[dlorenc]

Yeah I'm trying to repro but having trouble. I built from head with:

$ docker rmi -f gcr.io/kaniko-project/executor
$ make images

The image appears to have been rebuilt correctly, and exists in my daemon:

$ docker images | grep kaniko
gcr.io/kaniko-project/executor                 latest                                                             4779434dffc1        39 seconds ago       53.6MB

Then run_in_docker works with your Dockerfile:

$ ./run_in_docker.sh Dockerfile $(pwd)/foo $IMG

I've also done it with the image currently at latest, by removing it from my local daemon cache and doing run_in_docker again.

[priyawadhwa]

@Delphinator I did find a bug when building your Dockerfile (PR #270), but I was still able to build it successfully.

[xinau]

@dlorenc the way you describe works for me as well, since the ./run_in_docker.sh executes directly the executor in the container. The problem I describe happens, when you enter the container with /busybox/sh as entrypoint and than use the executor. :)

P.s. Not sure if I did understand your point correctly.

[dlorenc]

Ah, ok! That helps. I missed that difference in execution. There might be a difference in the way process groups are handled inside the busybox sh binary vs. when docker executes the executor directly.

[JensGutermuth]

There might be a difference in the way process groups are handled inside the busybox sh binary vs. when docker executes the executor directly.

There is definitely a difference. Just did a few experiments starting the executor in different ways and only some failed. I used the files from @dlorenc above, but I don't think the exact Dockerfile used makes any difference.

$ cat foo/Dockerfile 

FROM golang:1.10.3-alpine AS build
ADD . /src
RUN cd /src && go build -o app

FROM alpine
WORKDIR /app
COPY --from=build /src/app /app/
CMD [ "./app" ]
$ cat foo/main.go 
package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

working invocations

$ ./run_in_docker.sh Dockerfile $(pwd)/foo test

$ docker run -it --rm -v $(pwd)/foo:/workspace --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug
[container] $ exec /kaniko/executor -c /workspace -f Dockerfile --tarPath /kaniko/test.tar -d test

$ echo "/kaniko/executor -c /workspace -f Dockerfile --tarPath /kaniko/test.tar -d test" | docker run -i --rm -v $(pwd)/foo:/workspace --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug

failing invocation

$ docker run -it --rm -v $(pwd)/foo:/workspace --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug
[container] $ /kaniko/executor -c /workspace -f Dockerfile --tarPath /kaniko/test.tar -d test
[snip]
INFO[0022] Taking snapshot of full filesystem...        
INFO[0026] Executing 0 build triggers                   
INFO[0026] cmd: Add [.]                                 
INFO[0026] dest: /src                                   
INFO[0026] cmd: copy [.]                                
INFO[0026] dest: /src                                   
INFO[0026] Creating directory /src                      
INFO[0026] Copying file /workspace/Dockerfile to /src/Dockerfile 
INFO[0026] Copying file /workspace/main.go to /src/main.go 
INFO[0026] cmd: /bin/sh                                 
INFO[0026] args: [-c cd /src && go build -o app]        
Killed
[container] $ echo $?  # echo exit code of last command
137