error building image: error building stage: gzip: invalid header

[drpmma]

Actual behavior
The dockerfile command RUN and USER will lead to "unpack rootfs", which causing "error building image: error building stage: gzip: invalid header".
But the error only happens in some specific images.

Expected behavior
It should work fine.

To Reproduce
Steps to reproduce the behavior:
build the image with kaniko.

Additional Information

• Dockerfile works wrong

FROM myimage

USER root
RUN chmod 1777 /tmp

ADD test.txt /tmp/

• Dockerfile works right

FROM myimage

ADD test.txt /tmp/

• log

INFO[0000] Executing 0 build triggers
INFO[0000] Unpacking rootfs as cmd RUN chmod 1777 /tmp requires it.
error building image: error building stage: gzip: invalid header

• Kaniko Image v0.10.0

[tejal29]

@drpmma Can you confirm your base image has /tmp dir?

[drpmma]

@drpmma Can you confirm your base image has /tmp dir?

Yes, and its permission is 1777. But only some images have this issue.

[vm86]

@drpmma
Need for more information.
I check with kaniko version 0.12.0

FROM busybox

USER root
RUN chmod 1777 /tmp

ADD test.txt /tmp/

Bug doesn't reproduce:

INFO[0006] Downloading base image busybox               
INFO[0007] Unpacking rootfs as cmd RUN chmod 1777 /tmp requires it. 
INFO[0008] Taking snapshot of full filesystem...        
INFO[0010] USER root                                    
INFO[0010] cmd: USER                                    
INFO[0010] RUN chmod 1777 /tmp                          
INFO[0010] cmd: /bin/sh                                 
INFO[0010] args: [-c chmod 1777 /tmp]                   
INFO[0010] Taking snapshot of full filesystem...        
INFO[0010] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0010] Using files from context: [/workspace/test.txt] 
INFO[0010] ADD test.txt /tmp/                           
INFO[0010] Taking snapshot of files...                  
INFO[0010] Skipping push to container registry due to --no-push flag 

[drpmma]

@drpmma
Need for more information.
I check with kaniko version 0.12.0

FROM busybox

USER root
RUN chmod 1777 /tmp

ADD test.txt /tmp/

Bug doesn't reproduce:

INFO[0006] Downloading base image busybox               
INFO[0007] Unpacking rootfs as cmd RUN chmod 1777 /tmp requires it. 
INFO[0008] Taking snapshot of full filesystem...        
INFO[0010] USER root                                    
INFO[0010] cmd: USER                                    
INFO[0010] RUN chmod 1777 /tmp                          
INFO[0010] cmd: /bin/sh                                 
INFO[0010] args: [-c chmod 1777 /tmp]                   
INFO[0010] Taking snapshot of full filesystem...        
INFO[0010] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0010] Using files from context: [/workspace/test.txt] 
INFO[0010] ADD test.txt /tmp/                           
INFO[0010] Taking snapshot of files...                  
INFO[0010] Skipping push to container registry due to --no-push flag 

This issue only happens in some specific images which may be inconvenient to be made public.
I am trying to reproduce it in some insensitive images.

[drpmma]

I push a problem image to docker hub and pull it with kaniko. The image pulled from docker hub extracts with no problem. This issue seems causes by improper compression or the unofficial image registry.

[tejal29]

So the error is coming from here https://github.com/GoogleContainerTools/kaniko/blob/master//pkg/util/fs_util.go#L73.
Can you check if the layers belonging to image are correct?

Is image, built by kaniko?

[tejal29]

@drpmma Can you try with -v debug flag with the latest image

[drpmma]

@drpmma Can you try with -v debug flag with the latest image

The image is not built by kaniko. And the image can be extracted and run correctly by docker.

Below are the logs.
The error is here.
https://github.com/GoogleContainerTools/kaniko/blob/master//pkg/util/fs_util.go#L87

...
DEBU[0000] Read the following line from /proc/self/mountinfo:
DEBU[0000] Reached end of file /proc/self/mountinfo
DEBU[0000] Mounted directories: [{/kaniko false} {/var/run false} {/etc/mtab false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/cpuset,cpu,cpuacct false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/net_cgroup false} {/dev/mqueue false} {/build-test false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/kaniko/.docker/config.json false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/timer_stats false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}]
DEBU[0000] Extracting layer %!d(types.MediaType=application/vnd.docker.image.rootfs.diff.tar.gzip) of media type %!s(MISSING)
error building image: error building stage: gzip: invalid header

[codevbus]

This is currently failing when running make test as well.

[Silvanoc]

@drpmma could you fix the issue? I'm getting the same error, and trying to build with Buildah works properly... but I need Kaniko for CI.

[vivekpd15]

+1

[vivekpd15]

Any updates on this?

I can't build using Kaniko on CI because of this issue. I would really appreciate some help. Thank you.

[savemuri]

Came across same error and could not find a workaround. Wondering if anyone has solved it?

[MatrixManAtYrService]

I get this error when I provide a raw string as build context:

echo -e 'FROM alpine \nRUN echo "created from standard input"' \
    | kubectl run build --rm --restart=Never -i --image=gcr.io/kaniko-project/executor:latest -- \
    --destination someplace \
    --context tar://stdin

But it goes away if I tar and gzip the contents first:

echo -e 'FROM alpine \nRUN echo "created from standard input"' \
    > Dockerfile \
    | tar -cf - Dockerfile \
    | gzip -9 \
    | kubectl run build --rm --restart=Never -i --image=gcr.io/kaniko-project/executor:latest -- \
        --destination someplace \
        --context tar://stdin

Something must be triggering kaniko to expect gzipped content, yet the content it receives isn't gzipped.

[siwyd]

Seems like I'm running into this as well. Tracked it down to trying to run a command in an Ubuntu 20.04 image pulled from Amazon's new public registry.

So basically, this image fails: public.ecr.aws/lts/ubuntu@sha256:aed3a67ccca8f308fa5e367918f91dde88fb79c5d72e0d139e799da89634bd9e

And this image works: ubuntu@sha256:376209074d481dca0a9cf4282710cd30a9e7ff402dea8261acdaaf57a18971dd

I hope that helps! I was using Kaniko v1.6.0.

[lepirlouit]

if it helps, here's a public image that fails too

FROM public.ecr.aws/ubuntu/ubuntu:20.04
RUN chmod 1777 /tmp

I was able to test a local Dockerfile by running this command in the folder containing the Dockerfile

docker run -it --rm \          
    -v ${PWD}:/workspace \
    gcr.io/kaniko-project/executor:latest \
    --dockerfile /workspace/Dockerfile \
    --no-push \
    --context dir:///workspace/

trace debug both output

TRAC[0001] Extracting layer 0 of media type application/vnd.docker.image.rootfs.diff.tar.gzip

kaniko is downloading the layers from the resgistry,
layers from docker.io are in gzip format while layers from public.ecr.aws are in tarball format directly
kaniko should Unzip only if gzip header is present
unzip is done in file vendor/github.com/google/go-containerregistry/pkg/v1/partial/compressed.go:56

[DrissiReda]

Any updates on this?