Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

horizonaz · 2020-09-10T03:23:01Z

Actual behavior
The kaniko build failed while unpacking and whiteouting file systems. (Debug log is available in Additional Information section)
The build is working if doing with "docker build".

INFO[0002] Unpacking rootfs as cmd COPY test.lic test.lic requires it.
error building image: error building stage: failed to get filesystem from image: removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system

Expected behavior
Kaniko build should be successful if there is COPY command in Dockerfile.

To Reproduce
Run the below kaniko build on Docker

docker run -v "$HOME"/.config/gcloud:/root/.config/gcloud -v /home/ec2-user:/workspace/ gcr.io/kaniko-project/executor:latest --context dir:///workspace/ --no-push --verbosity debug

Additional Information

Dockerfile

FROM pingidentity/pingfederate
COPY test.lic test.lic

Build Context

ls -l
total 8
-rw-rw-r-- 1 ec2-user ec2-user 54 Sep 8 10:28 Dockerfile
-rw-rw-r-- 1 ec2-user ec2-user 5 Sep 8 10:31 test.lic

Kaniko Image (fully qualified with digest)
gcr.io/kaniko-project/executor:latest
Full Debug Log

docker run -v "$HOME"/.config/gcloud:/root/.config/gcloud -v /home/ec2-user:/workspace/ gcr.io/kaniko-project/executor:latest --context dir:///workspace/ --no-push --verbosity debug
DEBU[0000] Getting source context from dir:///workspace/
DEBU[0000] Build context located at /workspace/
DEBU[0000] Copying file /workspace/Dockerfile to /kaniko/Dockerfile
DEBU[0000] Skip resolving path /kaniko/Dockerfile
DEBU[0000] Skip resolving path /workspace/
DEBU[0000] Skip resolving path /cache
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Built stage name to index map: map[]
INFO[0000] Retrieving image manifest pingidentity/pingfederate
INFO[0000] Retrieving image pingidentity/pingfederate
DEBU[0000] No file found for cache key sha256:9cbf21e56fa11e140e68f95fb9d855d7b5a6468e1ccebb647c5078d169eed9d2 stat /cache/sha256:9cbf21e56fa11e140e68f95fb9d855d7b5a6468e1ccebb647c5078d169eed9d2: no such file or directory
DEBU[0000] Image pingidentity/pingfederate not found in cache
INFO[0000] Retrieving image manifest pingidentity/pingfederate
INFO[0000] Retrieving image pingidentity/pingfederate
INFO[0000] Built cross stage deps: map[]
INFO[0000] Retrieving image manifest pingidentity/pingfederate
INFO[0000] Retrieving image pingidentity/pingfederate
DEBU[0001] No file found for cache key sha256:9cbf21e56fa11e140e68f95fb9d855d7b5a6468e1ccebb647c5078d169eed9d2 stat /cache/sha256:9cbf21e56fa11e140e68f95fb9d855d7b5a6468e1ccebb647c5078d169eed9d2: no such file or directory
DEBU[0001] Image pingidentity/pingfederate not found in cache
INFO[0001] Retrieving image manifest pingidentity/pingfederate
INFO[0001] Retrieving image pingidentity/pingfederate
INFO[0002] Executing 0 build triggers
INFO[0002] Unpacking rootfs as cmd COPY test.lic test.lic requires it.
DEBU[0002] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/blkio false} {/dev/mqueue false} {/dev/shm false} {/workspace false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/root/.config/gcloud false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/latency_stats false} {/proc/timer_list false} {/proc/sched_debug false} {/sys/firmware false}]
DEBU[0002] Whiting out /opt/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0002] Whiting out /bin/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0002] Not adding /dev because it is ignored
DEBU[0002] Whiting out /dev/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0002] Whiting out /etc/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0002] Not adding /etc/hostname because it is ignored
DEBU[0002] Not adding /etc/hosts because it is ignored
DEBU[0002] Not adding /etc/mtab because it is ignored
DEBU[0002] Whiting out /home/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0002] Whiting out /lib/.wh..wh..opq
DEBU[0002] not including whiteout files
DEBU[0003] Whiting out /media/.wh..wh..opq
DEBU[0003] not including whiteout files
DEBU[0003] Whiting out /mnt/.wh..wh..opq
DEBU[0003] not including whiteout files
DEBU[0003] Whiting out /opt/backup/.wh..wh..opq
DEBU[0003] not including whiteout files
DEBU[0003] Whiting out /opt/in/.wh..wh..opq
DEBU[0003] not including whiteout files
DEBU[0003] Whiting out /opt/java/.wh..wh..opq
DEBU[0003] not including whiteout files
DEBU[0005] Whiting out /opt/logs/.wh..wh..opq
DEBU[0005] not including whiteout files
DEBU[0005] Whiting out /opt/out/.wh..wh..opq
DEBU[0005] not including whiteout files
DEBU[0005] Whiting out /opt/server/.wh..wh..opq
DEBU[0005] not including whiteout files
DEBU[0012] Whiting out /opt/staging/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Not adding /proc because it is ignored
DEBU[0012] Whiting out /proc/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Whiting out /root/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Whiting out /run/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Whiting out /sbin/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Whiting out /srv/.wh..wh..opq
DEBU[0012] not including whiteout files
DEBU[0012] Not adding /sys because it is ignored
DEBU[0012] Whiting out /sys/.wh..wh..opq
error building image: error building stage: failed to get filesystem from image: removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system

This is working if doing via Docker build

docker build -t ping:1 .
Sending build context to Docker daemon 24.06kB
Step 1/2 : FROM pingidentity/pingfederate
---> 898156c692f6
Step 2/2 : COPY test.lic test.lic
---> Using cache
---> 34b88c8834fe
Successfully built 34b88c8834fe
Successfully tagged ping:1

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use `--cache` flag	Yes
Please check if your dockerfile is a multistage dockerfile	No

The text was updated successfully, but these errors were encountered:

shaolinyang2009 · 2021-02-18T20:48:17Z

We got same error

claudioweiler · 2021-04-09T17:10:41Z

I can confirm this error on k8s too.
Appears to be related to issue #1073.

amirna2 · 2021-05-10T05:40:48Z

I am running into the same issue with several different base docker images when trying to build an image in Gitlab CI pipeline.
This happens with cache disabled and DOCKER_DRIVER set to overlayFS.
I am able to fix some of the permission denied by using the --ignore-path, but in some cases there are just too many paths to ignore.
The trace below happens when using golang:1.16-stretch as a base image and simply installing wget

INFO[0000] Retrieving image manifest golang:1.16-stretch 
INFO[0000] Retrieving image golang:1.16-stretch from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest golang:1.16-stretch 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
WARN[0001] maintainer is deprecated, skipping           
INFO[0001] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y wget requires it. 
DEBU[0001] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/run/systemd false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/perf_event false} {/dev/mqueue false} {/cache false} {/builds false} {/busybox false} {/certs/client false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false}] 
DEBU[0001] Not adding /dev because it is ignored        
DEBU[0001] Not adding /etc/hostname because it is ignored 
DEBU[0001] Not adding /etc/resolv.conf because it is ignored 
DEBU[0001] Not adding /proc because it is ignored       
DEBU[0001] Not adding /sys because it is ignored        
DEBU[0004] Not adding /var/run because it is ignored    
DEBU[0004] Whiting out /etc/ca-certificates/.wh..wh..opq 
error building image: error building stage: failed to get filesystem from image: removing whiteout etc/ca-certificates/.wh..wh..opq: fstatat /etc/ca-certificates/.wh..opq: operation not permitted

jrevillard · 2021-09-24T17:10:31Z

Dear all, I'm stuck with it also in gitlab runners... any workaround ?

DS-KrzJon · 2022-04-13T20:57:07Z

We got it too. First it creates symlink
{"level":"trace","msg":"creating file /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0","time":"2022-04-13T20:46:32Z"}
{"level":"trace","msg":"symlink from libcap.so.2.32 to /usr/lib/x86_64-linux-gnu/libcap.so.2","time":"2022-04-13T20:46:32Z"}
Then it removes it
{"level":"debug","msg":"Whiting out /usr/lib/x86_64-linux-gnu/.wh.libXmuu.so.1.0.0","time":"2022-04-13T20:46:38Z"}
{"level":"debug","msg":"not including whiteout files","time":"2022-04-13T20:46:38Z"}
{"level":"debug","msg":"Whiting out /usr/lib/x86_64-linux-gnu/.wh.libbsd.so.0","time":"2022-04-13T20:46:38Z"}

Without that files build fails
/usr/lib/x86_64-linux-gnu/libgtk-3-0/gtk-query-immodules-3.0: error while loading shared libraries: libbsd.so.0: cannot open shared object file: No such file or directory

Any advice or idea for quick fix? It's really destroing some of impotant builds ;(

gabyx · 2022-04-19T12:59:03Z

Relates directly to #1944

gabyx · 2022-04-20T07:02:18Z

The code here is just really strange:

// GetFlattenedPathsForWhiteOut returns all paths in the current FS
func (l *LayeredMap) getFlattenedPathsForWhiteOut() map[string]struct{} {
	paths := map[string]struct{}{}
	for _, l := range l.layers {
		for p := range l {
			if strings.HasPrefix(filepath.Base(p), ".wh.") {
				delete(paths, p)
			}
			paths[p] = struct{}{} // For what is then the "iffed" delete above??????
		}
	}
	return paths
}

Shouldn't that be: delete(paths, p.RemovePrefix(archive.WhiteoutPrefix))... otherwise it does not make sense or does it?

gabyx mentioned this issue Apr 20, 2022

Fix: Flatten layer function needs to return existing files in the layer correctly #2057

Merged

3 tasks

imjasonh closed this as completed in #2057 Apr 26, 2022

gabyx mentioned this issue Apr 28, 2022

fix: Refactor LayersMap to correct old strange code behavior #2066

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

horizonaz commented Sep 10, 2020 •

edited

Loading

shaolinyang2009 commented Feb 18, 2021

claudioweiler commented Apr 9, 2021

amirna2 commented May 10, 2021

jrevillard commented Sep 24, 2021

DS-KrzJon commented Apr 13, 2022

gabyx commented Apr 19, 2022

gabyx commented Apr 20, 2022 •

edited

Loading

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

Comments

horizonaz commented Sep 10, 2020 • edited Loading

shaolinyang2009 commented Feb 18, 2021

claudioweiler commented Apr 9, 2021

amirna2 commented May 10, 2021

jrevillard commented Sep 24, 2021

DS-KrzJon commented Apr 13, 2022

gabyx commented Apr 19, 2022

gabyx commented Apr 20, 2022 • edited Loading

horizonaz commented Sep 10, 2020 •

edited

Loading

gabyx commented Apr 20, 2022 •

edited

Loading