Kaniko documentation should be more clear regarding required Linux Capabilities

[kravvcu]

Request summary
It needs to be explicitly stated in the documentation that some Linux capabilities might be required (apart from the permission to run Kaniko as root) in order to successfully build docker images using Kaniko (this is relevant for CI systems build on a PodSecurityPolicy(PSP)-secured Kubernetes cluster).

Description
There are not too many resources available online on the subject on how to setup a CI system using Kaniko on a PSP-secured Kubernetes cluster. One of such resources is this article, which suggests that Kaniko would only need the permission to run as root user, which, at least for me, is not the case. It so turns out that the rule of dropping all capabilities prevents even simple builds (example below). The question is: what is the minimal capability set which allows all (or most) Kaniko builds.

To Reproduce
Steps to reproduce the behavior:
docker run --rm --name kaniko --cap-drop=all -v $(pwd)/Dockerfile:/Dockerfile -v $(pwd):/kaniko-context -it gcr.io/kaniko-project/executor:latest -f /Dockerfile -c /kaniko-context --no-push

What am I seeing
Depending on which capabilities are missing:

• Without CAP_CHOWN

error building image: error building stage: chown /etc/gshadow: operation not permitted

• Without CAP_SETUID

E: setegid 65534 failed - setegid (1: Operation not permitted)

• Without CAP_SETGID

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)

• Without CAP_FOWNER

W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)

• Without CAP_DAC_OVERRIDE

W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)
E: can not open /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - fopen (13: Permission denied)
E: The repository 'http://security.debian.org/debian-security buster/updates InRelease' provides only weak security information.

Working command
docker run --rm --name kaniko --cap-drop=all --cap-add CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=FOWNER --cap-add=DAC_OVERRIDE -v $(pwd)/Dockerfile:/Dockerfile -v $(pwd):/kaniko-context -it gcr.io/kaniko-project/executor:latest -f /Dockerfile -c /kaniko-context --no-push

Additional Information

• Dockerfile

FROM debian:10-slim
RUN apt update && apt install -y libcap2-bin
COPY file1 /file1
COPY file2 /file2

• Build Context

$ ls .
Dockerfile file1 file2

file1 and file2 are regular files with irrelevant contents

• Kaniko Image (fully qualified with digest)

gcr.io/kaniko-project/executor:latest (sha256:584a8d90679211d9b09465d778990ec15965cf78f57f197e973d57d14b08eb81)

[tejal29]

Thanks @kravvcu for the detailed bug report.
Would you be up for writing this documentation??

Thanks
Tejal

[kravvcu]

Hey, @tejal29

Thanks for the answer. I'll put it on my to-do list and try to contribute in the coming days / weeks if that's ok.

Regards,
kravvcu

[tejal29]

sounds good!

[ddgenome]

This comment may be of help: #1020 (comment) . One mentioned there that is not listed above is CAP_SETFCAP , but that may have been specific to the build in question at that time. That demonstrates the difficulty of documenting this, as what it done in the build can change the requirements significantly.

It may be worth also pointing to the default set of capabilities the Docker daemon provides:

• https://docs.docker.com/engine/security/security/#linux-kernel-capabilities
• https://github.com/moby/moby/blob/f6a5ccf492e8eab969ffad8404117806b4a15a35/oci/caps/defaults.go

[arielshulman]

I think this would be helpful.

We've just upgraded our ver from 1.6.0 to 1.8.1 in our openshift env and suddenly build process broke with security.capability error.
It is probably cause by #1838 change and lack of SETFCAP in our env, but I couldn't find a list of required cap for kaniko to work.

[aaron-prindle]

This article describes the set of capabilities kaniko uses when run from docker - so different IIUC than what capabilities would be needed for PSP but possibly useful for documentation:
https://blog.rewanthtammana.com/hardening-kaniko-build-process-with-linux-capabilities

From the article:
Dockerfile used

FROM alpine
ENTRYPOINT ["/bin/sh", "-c", "echo hello"]

Min set of capabilities needed for the analyzed run-with-docker + minimal-dockerfile:

• CHOWN
• SETUID
• SETGID
• FOWNER
• DAC_OVERRIDE