error building image: error building stage GCC

[dllozhkin]

I try build gcc image with kaniko
and get error

INFO[0003] Unpacking rootfs as cmd RUN cmake -DCMAKE_BUILD_TYPE=Release requires it. 
error building image: error building stage: removing whiteout usr/bin/cmake/.wh..wh..opq: lstat /usr/bin/cmake/.wh..opq: operation not permitted

kaniko image : gcr.io/kaniko-project/executor:debug

stack:

k8s, gitlab, gitlab-runner(into k8s)

[donmccasland]

Are you using cacheing? Might be related to #742

[tejal29]

@denitol Can you provide more information on your docker file?

[z0mb1ek]

same problem

INFO[0030] RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev
INFO[0030] Found cached layer, extracting to filesystem
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

[z0mb1ek]

my docker image

FROM nginx:1.17.8

COPY devops/nginx/ssl/certbundle.pem /etc/ssl/certbundle.pem

COPY devops/nginx/ssl/1.key /etc/ssl/1.key

COPY devops/nginx/nginx.conf /etc/nginx/nginx.conf

COPY devops/nginx/default.conf /etc/nginx/conf.d/default.conf

RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev

RUN git clone https://github.com/kaltura/nginx-vod-module.git

RUN wget 'https://nginx.org/download/nginx-1.17.8.tar.gz' && tar xvzf nginx-1.17.8.tar.gz

WORKDIR nginx-1.17.8

RUN ./configure --add-dynamic-module=/nginx-vod-module \
	--with-file-aio \
	--with-threads \
	--with-cc-opt="-O3"

RUN ls -la /nginx-vod-module

[ribx]

We are also on gitlab/gitlab-runner + k8s.

I can reproduce this sometimes always (if cache is present) with this simple Dockerfile:

FROM nginx:1.17

RUN echo "server {listen 80; root /html/; \
          location /api/ {return 500;} \
          location /static/ {} \
          location / {try_files \$uri /index.html;}}" > /etc/nginx/conf.d/default.conf

COPY ./build /html

with this command:

/kaniko/executor
      --context "./"
      --dockerfile Dockerfile
      --destination "$REGISTRY/$GROUP/$PROJECT/frontend:$TAG" # maybe this is important?
      --cache=true

inside the image "gcr.io/kaniko-project/executor:debug" (sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3)
and also "gcr.io/kaniko-project/executor:debug-v0.16.0"

When the cache is cleared, the build works fine.

[Flauschbaellchen]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

[cvgw]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

You probably need to clear your cache so that it doesn't keep re-using the bad cached layer.

[Flauschbaellchen]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

[cvgw]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

Apologies if I'm misunderstanding you; my suggestion was to clear the cache when going from a known bad version of kaniko to a good version

E.G

image was previously built with kaniko v0.17.0 and cached
switch to kaniko v0.17.1
clear cache
build image with v0.17.1
at that point the cache is good to use for v0.17.1

[z0mb1ek]

Apologies if I'm misunderstanding you; my suggestion was to clear the cache when going from a known bad version of kaniko to a good version

E.G

image was previously built with kaniko v0.17.0 and cached
switch to kaniko v0.17.1
clear cache
build image with v0.17.1
at that point the cache is good to use for v0.17.1

try this, not working too

[Flauschbaellchen]

With kaniko 0.24.0 this problem still persists.

First run without cache is successful, but on the second run, with the cached layer, it fails:

INFO[0013] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: error removing /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info to make way for new file.: fstatat /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info: operation not permitted

[Laski]

I was thinking: would it be possible to write (perhaps outside kaniko) some script that retires the job without cache if the first one fails? Probably using the --cleanup flag

[Narretz]

Can confirm that this is still an issue with Kaniko 1.2.0 (and Gitlab Runner 12.8.0 + kubernetes executor)

[gajus]

Is there a workaround?

[Keramblock]

Same

[deepakkt-a]

Can confirm that this is still an issue with Kaniko 1.2.0 (and Gitlab Runner 12.8.0 + kubernetes executor)

And with v1.3

[pdefreitas]

I've experienced the same issue while using warmer. It occurs when the executor is creating a .tag.gz file when running under Docker.

error building image: error building stage: failed to execute command: extracting fs from image: unexpected EOF

Edit: I'm using the latest debug image.

[Flauschbaellchen]

I'm still experience this issue with Kaniko executor:v1.3.0-debug:

First run works fine, however when using the cached layer it throws an error:

INFO[0008] RUN pip install --no-cache --no-cache-dir awscli==${AWSCLI_VERSION} 
INFO[0008] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: error removing /usr/local/lib/python3.7/site-packages/.wh.botocore-1.20.0.dist-info to make way for new file.: fstatat /usr/local/lib/python3.7/site-packages/.wh.botocore-1.20.0.dist-info: operation not permitted

Running with docker-executor on gitlab-runner 13.8.0

[Flauschbaellchen]

Tested again with version v1.5.1-debug, same failure.

[Flauschbaellchen]

I've changed the docker storage driver from the default aufs to overlay2 as mentioned in this comment.

As a note if you use a gitlab-runner with docker executor:
I needed to change it within /etc/docker/daemon.json for the whole docker process.
The configuration in /etc/gitlab-runner/config.toml using environment= ["DOCKER_DRIVER=overlay2"] or within the CI was not sufficient (as suggested in the gitlab documentation)

This has fixed my issues I faced currently.

[dclong]

I encountered a similar when building a Docker image using Kaniko on k8s. Unfortunately, the Dockerfile refers to enterprise Docker images, so that I could post steps to reproduce it here.

error building image: error building stage: failed to get filesystem from image: unexpected EOF

It fails after

INFO[2021-09-15T00:51:01Z] Unpacking rootfs

Seems to me that unpacking rootfs failed.

Note: The Dockerfile can be built successfully using Docker.

[piyoki]

I encountered a similar when building a Docker image using Kaniko on k8s. Unfortunately, the Dockerfile refers to enterprise Docker images, so that I could post steps to reproduce it here.

error building image: error building stage: failed to get filesystem from image: unexpected EOF

It fails after

INFO[2021-09-15T00:51:01Z] Unpacking rootfs

Seems to me that unpacking rootfs failed.

Note: The Dockerfile can be built successfully using Docker.

I experienced exactly the same issue. Did you resolve it?

[chris-ng-scmp]

In my case, taking out registry mirror setting can fix the issue...

Don't know how this would related.

[csh995426531]

I encountered a similar when building a Docker image using Kaniko on k8s. Unfortunately, the Dockerfile refers to enterprise Docker images, so that I could post steps to reproduce it here.

error building image: error building stage: failed to get filesystem from image: unexpected EOF

It fails after

INFO[2021-09-15T00:51:01Z] Unpacking rootfs

Seems to me that unpacking rootfs failed.
Note: The Dockerfile can be built successfully using Docker.

I experienced exactly the same issue. Did you resolve it?

Have you solved this problem yet? I'm having the same trouble

[tooptoop4]

getting same issue

[nmichlo]

Same issue. Caching issue?

[omidmazraati]

same issue with caching, v1.9.1-debug

[mouuii]

same issue with caching, v1.9.1-debu

[omidmazraati]

I found a workaround for my issue
set --compressed-caching=false flag for kaniko
@mouuii

[servier-wajdibenabderahim]

any updates ? I'm facing the same issue when using the cache. the first build works fine but the second fails

[Narretz]

I've had this problem in gitlab runner for a long time, but just now tested it again and it works for me with the following config / versions:

• latest kaniko:debug image
• gitlab enterprise 16.1
• gitlab-runner 15.4.0 with kubernetes executor
• cache layers pushed to gitlab registry

[Enderric]

On configuration

• GitLab CE 13.2.1
• kaniko/executor:v1.13.0-debug
• gitlab-runner 16.1.0 docker executor
• cache layers pushed to nexus docker registry

I have a problem

error building image: error building stage: failed to execute command: extracting fs from image: unexpected EOF

Error appears аfter I change some in the dockerfile. If nothing changes in dockerfile, build works, cache is used.
I would be grateful for any hints