/busybox PATH not in ignoreList, causing multistage builds to timeout in Jenkins Pipelines

[jeanfabrice]

Actual behavior
This pipeline times out on Jenkins:

podTemplate(name: "jenkinsbuild-${JOB_NAME}-${BUILD_NUMBER}", yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: jnlp
    image: jenkins/inbound-agent
    imagePullPolicy: Always
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug-v1.2.0
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /kaniko/.docker
  volumes:  
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: jenkins-secrets
          items:
            - key: dockerconfigjson
              path: config.json
"""
) {
    node(POD_LABEL) {
        stage("Build with Kaniko") {
            container('kaniko') {
              writeFile file: "Dockerfile", text: """
FROM alpine AS stage1
RUN touch /file1
FROM alpine
COPY --from=stage1 /file1 /tmp
RUN touch /file2
              """

                sh ''' 
echo $PATH
/busybox/ps -ef
/busybox/ls /
/kaniko/executor --dockerfile=`pwd`/Dockerfile --context `pwd` --verbosity trace --no-push
/bin/ps -ef
/bin/ls /
                '''
            }
        }
    }
}

Expected behavior
Pipeline to success

To Reproduce
Run provided groovy pipeline on Jenkins

Personal analysis
/busybox is getting deleted between both Docker stage (Look at the provided logs, specifically the results of the ls command before and after the execution of kaniko.

In the meantime, the sh Jenkins step spawns the following process to detect the end and the result of the step (see logs):

sh -c ({ while [ -d '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635' -a \! -f '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-log.txt'; sleep 3; done } & jsc=durable-16856647925e219f4405aa6c51dc26b2; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe  '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/script.sh' > '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt.tmp'; mv '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt'; wait) >&- 2>&- &

If you look at this command line, you will see there's a call to mv which is supposed to be found in the PATH. This particular call is responsible for creating the jenkins-result.txt, telling sh step that the shell has terminated. Since /busybox is not there anymore, the mv command is not in the PATH neither and never succeed.

Possible solution:
1/ Persist /busybox between stages
2/ use withEnv(['PATH=/busybox:/kaniko:/bin']) for Jenkins shstep (adding alpine /bin to the PATH)
3/ Jump into the kaniko pod before it times out and manually /bin/mv '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt'

My guess is that 2/ and 3/ are not reliable ;)

Additional Information

• Dockerfile

FROM alpine AS stage1
RUN touch /file1
FROM alpine
COPY --from=stage1 /file1 /tmp
RUN touch /file2

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:debug-v1.2.0

• Jenkins Logs
  see https://gist.github.com/jeanfabrice/6d363e0181a5e8091c7b5a2cc3e6e294

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[jeanfabrice]

Maybe duplicates of #1275

[sodared]

@jeanfabrice Not sure if your problem has been solved ， here is one way to Persist /busybox between stages : /kaniko/executor --ignore-path /busybox (refer : #1622)
By the way , How do you get this ?
sh -c ({ while [ -d '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635' -a \! -f '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-log.txt'; sleep 3; done } & jsc=durable-16856647925e219f4405aa6c51dc26b2; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/script.sh' > '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt.tmp'; mv '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/test@tmp/durable-ca5ae635/jenkins-result.txt'; wait) >&- 2>&- &