Kaniko doesn't recognize file changes to modified files

[filesnate]

Actual behavior
When modifying an existing file in an image, kaniko doesn't recognize the file has been changed.

Expected behavior
Specifically, we're modifying the file /kaniko/ssl/certs/ca-certificates.crt, but it's not being identified as being changed, so the resulting image does not contain the updated file change.

To Reproduce
Steps to reproduce the behavior:

1. Create a new directory

% mkdir ~/issue-1156
% cd ~/issue-1156

2. Create a cert [ Totally optional to use a cert, but I wanted an example for the issue that was completely self-contained and a valid use-case. ]

% openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt -subj "/C=US/ST=Montana/L=Bozeman/O=Company Name/OU=Org/CN=www.example.com"

3. Create a Dockerfile

FROM gcr.io/kaniko-project/executor:debug

COPY certificate.crt /tmp/

# /bin/sh doesn't exist in kaniko
RUN ["/busybox/sh", "-c", "/busybox/cat /tmp/certificate.crt >> /kaniko/ssl/certs/ca-certificates.crt" ]

4. Build image using kaniko

[ Note, building an image hosted in a public gcr.io registry depends on the the fix for #1122 is merged, so a local image built from master is used below. ]

% docker run -v `pwd`:/workspace localhost:5000/executor:debug --context=/workspace --no-push
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0001] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0002] Unpacking rootfs as cmd COPY certificate.crt /tmp/ requires it. 
INFO[0009] Taking snapshot of full filesystem...        
INFO[0009] Resolving paths                              
INFO[0009] COPY certificate.crt /tmp/                   
INFO[0009] Resolving paths                              
INFO[0009] Taking snapshot of files...                  
INFO[0009] RUN ["/busybox/sh", "-c", "/busybox/cat /tmp/certificate.crt >> /kaniko/ssl/certs/ca-certificates.crt" ] 
INFO[0009] cmd: /busybox/sh                             
INFO[0009] args: [-c /busybox/cat /tmp/certificate.crt >> /kaniko/ssl/certs/ca-certificates.crt] 
INFO[0009] Taking snapshot of full filesystem...        
INFO[0009] Resolving paths                              
INFO[0009] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0009] Skipping push to container registry due to --no-push flag 
%

Specfically, the info line is indicating that there is an issue.

INFO[0009] No files were changed, appending empty layer to config. No layer added to image. 

5. This is more obvious if you actually push the image and send it to a registry

% docker run -v `pwd`:/workspace localhost:5000/executor:debug --context=/workspace --destination=localhost:5000/kaniko_local:latest --insecure
...
% docker pull localhost:5000/kaniko_local:latest

6. Build using docker

% docker build -t kaniko_docker:latest .

7. Compare file sizes of the certificates file in the original image, and in both new builds.

Base image:

% docker run -it --entrypoint='' gcr.io/kaniko-project/executor:debug ls -l /kaniko/ssl/certs
total 260
-rw-r--r--    1 0        0           262622 Mar 18 18:06 ca-certificates.crt

^^ Last week's date (expected)

Docker-generated image:

% docker run -it --entrypoint='' kaniko_docker ls -l /kaniko/ssl/certs
total 260
-rw-r--r--    1 0        0           263846 Mar 24 20:03 ca-certificates.crt

^^ Today's date (expected)

Kaniko-build image:

% docker run -it --entrypoint='' localhost:5000/kaniko_local:latest ls -l /kaniko/ssl/certs
total 260
-rw-r--r--    1 0        0           262622 Mar 18 18:06 ca-certificates.crt

^^ Last week's date (same as the original image, unexpected and an error). :(

Basically, the change to the file /kaniko/ssl/certs/ca-certificates.crt is not being recognized, so the resulting image does not contain the updated changes/layer.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	N
Please check if the build works in docker but not in kaniko	Y
Please check if this error is seen when you use --cache flag	Y
Please check if your dockerfile is a multistage dockerfile	N

[filesnate]

I suspect this is related to having /kaniko whitelisted, which I don't believe is configurable.

[ ie; it would be nice to be able to un-whitelist certain files such as the certificates, vs. having to have one-off solutions like done in #1011 ]

[tejal29]

@filesnate can you please create the file in another directory? We can't remove the whitelist kaniko dir because it stores all the files needed across multiple stages for mutlistage builds in /kaniko/$idx

[filesnate]

I can, but I can't see a way to update the certificates to add a self-signed certificate. Can you think of a way to update the list of certificates to kaniko to allow downloads from a registry that has a self-signed certificate?

[filesnate]

Never-mind, I believe I can use the feature from #1037 to solve my issue. This can be closed.