failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat

[pschichtel]

Actual behavior
Build of simple image with kaniko/executor:debug in docker with sysbox-runc runtime fails with:

error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.10.0-8-amd64/build: read-only file system

Expected behavior

image builds just fine

To Reproduce
Steps to reproduce the behavior:

1. install standard docker (20.10.8)
2. install sysbox-ce (0.4)
3. docker run --rm -it --entrypoint sh gcr.io/kaniko-project/executor:debug
4. create Dockerfile and packages
5. try building with /kaniko/executor --context . --dockerfile Dockerfile --destination "docker.repo.lbaum.eu/test123:latest" --cache=true --force

Additional Information

• Dockerfile

  FROM fedora:34
  RUN dnf clean all
  ADD packages /tmp/packages
  RUN dnf --assumeyes install $(cat /tmp/packages)

• Build Context
  packages:

  autoconf
  automake
  binutils
  bzip2
  clang
  curl
  gettext
  libcurl
  libcurl-devel
  libtool
  libxml2
  libxml2-devel
  llvm
  llvm-devel
  make
  maven
  mingw-binutils-generic
  mingw-filesystem-base
  mingw-w64-tools
  mingw32-binutils
  mingw32-cpp
  mingw32-crt
  mingw32-curl-static
  mingw32-expat-static
  mingw32-filesystem
  mingw32-gcc
  mingw32-gcc-c++
  mingw32-headers
  mingw32-libidn-static
  mingw32-libidn2-static
  mingw32-libssh2-static
  mingw32-nsis
  mingw32-openssl-static
  mingw32-pkg-config
  mingw32-win-iconv-static
  mingw32-winpthreads-static
  mingw32-zlib-static
  openssl
  openssl-devel
  patch
  upx
  uuid
  uuid-devel
  vim-common
  wget
  xar
  xar-devel
  xmlstarlet
  

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor@sha256:fcccd2ab9f3892e33fc7f2e950c8e4fc665e7a4c66f6a9d70b300d7a2103592f

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[pschichtel]

podman also works just fine:

podman --storage-driver=vfs build -t test .

[pschichtel]

this seems to fail when a base image has a "/lib" folder.

so far it happened for the library/fedora images and the jboss/keycloak (based on redhat's ubi8) image.

[pschichtel]

A workaround: --ignore-path=/lib

probably doesn't have an impact, but not entirely sure

[pschichtel]

It seems that --force is causing this (necessary with the latest :debug tag since the fix for container detection is not released yet).

[pschichtel]

On master this particular issue still happens and also a SIGSEGV is triggered when using the --ignore-path=/lib workaround:

# /kaniko/executor --context . --no-push --force --ignore-path=/lib
INFO[0000] Retrieving image manifest fedora:34          
INFO[0000] Retrieving image fedora:34 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest fedora:34          
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Unpacking rootfs as cmd RUN findmnt requires it. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7ff5e7]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/util.mkdirAllWithPermissions(0xc0000530c0, 0xf, 0x800001fd, 0x0, 0xc, 0x1, 0x1)
	/home/phillip/Code/kaniko/pkg/util/fs_util.go:776 +0x467
github.com/GoogleContainerTools/kaniko/pkg/util.ExtractFile(0x21d6f5c, 0x1, 0xc000e55a40, 0x2204180, 0xc0005f66c0, 0x0, 0x0)
	/home/phillip/Code/kaniko/pkg/util/fs_util.go:344 +0xe3c
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromLayers(0x21d6f5c, 0x1, 0xc00062d2e0, 0x1, 0x1, 0xc00071eee0, 0x1, 0x1, 0x0, 0x0, ...)
	/home/phillip/Code/kaniko/pkg/util/fs_util.go:205 +0x78d
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromImage(0x21d6f5c, 0x1, 0x2265948, 0xc00041bb20, 0x20056d0, 0xc000238c00, 0x0, 0xc00071ef60, 0x479525, 0x0)
	/home/phillip/Code/kaniko/pkg/util/fs_util.go:131 +0x147
github.com/GoogleContainerTools/kaniko/pkg/executor.(*stageBuilder).build.func1(0xc00071ef80, 0x47915c)
	/home/phillip/Code/kaniko/pkg/executor/build.go:311 +0x65
github.com/GoogleContainerTools/kaniko/pkg/util.Retry(0xc000e07278, 0x0, 0x3e8, 0x2f85d60, 0xc00071f288)
	/home/phillip/Code/kaniko/pkg/util/util.go:165 +0x2e
github.com/GoogleContainerTools/kaniko/pkg/executor.(*stageBuilder).build(0xc0004ad1e0, 0x0, 0x0)
	/home/phillip/Code/kaniko/pkg/executor/build.go:315 +0x147d
github.com/GoogleContainerTools/kaniko/pkg/executor.DoBuild(0x2f54860, 0x1, 0x0, 0x0, 0x1c86360)
	/home/phillip/Code/kaniko/pkg/executor/build.go:600 +0x645
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x2f3b8c0, 0xc000178000, 0x0, 0x5)
	/home/phillip/Code/kaniko/cmd/executor/cmd/root.go:141 +0xef
github.com/spf13/cobra.(*Command).execute(0x2f3b8c0, 0xc00004c1f0, 0x5, 0x5, 0x2f3b8c0, 0xc00004c1f0)
	/home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x2f3b8c0, 0xd, 0xc000798a50, 0x0)
	/home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/home/phillip/Code/kaniko/cmd/executor/main.go:31 +0x72

[fuomag9]

same error even without --force

error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.4.0-104-generic/initrd: read-only file system

[aaron-prindle]

possible dupe of #1619