Kaniko cannot copy hidden directory

[achmudas]

Actual behavior

When building docker image Kaniko cannot copy hidden directory:

[36mINFO [0m[0258] RUN mkdir ./backend/.test                    
[36mINFO [0m[0258] cmd: /bin/sh                                 
[36mINFO [0m[0258] args: [-c mkdir ./backend/.test]             
[36mINFO [0m[0258] Taking snapshot of full filesystem...        
[36mINFO [0m[0258] RUN touch ./backend/.test/simplefile.txt     
[36mINFO [0m[0258] cmd: /bin/sh                                 
[36mINFO [0m[0258] args: [-c touch ./backend/.test/simplefile.txt] 
[36mINFO [0m[0258] Taking snapshot of full filesystem...        
[36mINFO [0m[0259] RUN ls -la ./backend/.test/                  
[36mINFO [0m[0259] cmd: /bin/sh                                 
[36mINFO [0m[0259] args: [-c ls -la ./backend/.test/]           
total 8
drwxr-sr-x    2 root     node          4096 Jul 19 13:57 .
drwxr-sr-x    4 root     node          4096 Jul 19 13:57 ..
-rw-r--r--    1 root     node             0 Jul 19 13:57 simplefile.txt
[36mINFO [0m[0259] Taking snapshot of full filesystem...        
[36mINFO [0m[0260] No files were changed, appending empty layer to config. No layer added to image. 
error building image: error building stage: lstat /home/jenkins/workspace/tests/backend/.test: no such file or directory

Expected behavior
It should see all the content in hidden directory and copy it to destination.

Additional Information

• Dockerfile

FROM node:8.11.3-alpine as base

WORKDIR /home/node

RUN mkdir ./backend/.test
RUN touch ./backend/.test/simplefile.txt

RUN ls -la ./backend/.test/

COPY ./backend/.test ./backend

• Jenkins slave Pod

kind: Pod
metadata:
  name: kaniko
spec:
  serviceAccountName: jenkins
  automountServiceAccountToken: true
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
    - name: jenkins-docker-cfg
      mountPath: /kaniko/.docker/
  - name: node
    image: node:8.15.0-alpine
    tty: true
    command:
    - cat
  - name: kubectl
    image: k8s-kubectl:v1.14.4
    imagePullPolicy: Always
    command:
    - cat
    tty: true
  imagePullSecrets:
    - name: secret
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: secret
          items:
          - key: .dockerconfigjson
            path: config.json

--- Jenkinsfile

...
stage('Create Docker image') {
      container(name: 'kaniko', shell: '/busybox/sh') {
        withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
          sh '''#!/busybox/sh
            /kaniko/executor -f `pwd`/Dockerfile -c `pwd` --skip-tls-verify --destination=<dest>
          '''
        }
      }
    }   
...

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor@sha256:7cb20b4b9b23f3f5997db20adc97c04889efaf1cd57bc00b87b9da9a1754b69f

[tejal29]

Thanks @achmudas. This has nothing to do the hidden file. I think the issue is the copy command

FROM busybox

RUN mkdir /.test
RUN touch /.test/simplefile.txt

RUN ls -la /.test/

RUN mkdir /not
RUN touch /not/test

RUN mkdir /test1
COPY /not /test1
#COPY /.test/ /test1/

Even copying from a dir which was created in previous docker directives fails.
Right now, we resolve sources to copy from build context. Since this dir does not exists in build context, the build fails.

The offending code is here

kaniko/pkg/util/command_util.go

Line 227 in c8fabdf

fi, err := os.Lstat(filepath.Join(root, resolvedSources[0]))

We need to account for sources which were created in previous docker directives if src dir is not found in the build context.

[tejal29]

This might be related to #742

[subesokun]

Same issue here. To be sure I've removed the folder ./dist, which I want to copy into the container image, from the .gitignore file but still the build is failing when Kaniko tries to extract the cached layer.

[dani29]

Hi @achmudas,
After taking a look, the issue seems to be what @tejal29 described: the source is resolved at the start of the build, and files created subsequently aren't picked by the ADD/COPY command. This seem to be consistent to the behavior of docker, and not a bug in Kaniko.

Trying to docker build the Dockerfile you supplied fails as well:

Step 1/6 : FROM node:8.11.3-alpine as base
 ---> ca148a52ea10
Step 2/6 : WORKDIR /home/node
 ---> Running in c2888d8f376f
Removing intermediate container c2888d8f376f
 ---> 5146db187e27
Step 3/6 : RUN mkdir ./backend/.test
 ---> Running in bae617543038
mkdir: can't create directory './backend/.test': No such file or directory
The command '/bin/sh -c mkdir ./backend/.test' returned a non-zero code: 1

Likewise, docker build on the example supplied by @tejal29 also fails on the COPY step:

COPY failed: stat /usr/local/docker/tmp/docker-builder806188802/not: no such file or directory

[tejal29]

@dani29 Thanks a lot for confirming!

@achmudas i am going to close this bug as won't fixed. Please re-open if you still have any questions.