Whiteout files present after image unpacking

[JakubLedworowski]

Describe the bug

For confidential VM, whiteout files (.wh) are present in the container's filesystem, while they should not, according to OCI image spec.

Runtime classes tested:

• kata-qemu-coco-dev - .wh files are present
• kata-qemu-tdx - .wh files are present
• kata-qemu - not present
• no runtime class - not present

How to reproduce

Repro steps

1. Install CoCo Operator and sample runtime classes v0.11.0:

kubectl apply -k "github.com/confidential-containers/operator/config/release?ref=v0.11.0"
kubectl apply -k "github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.11.0"

2. Prepare the following whiteout-file-test.yaml file for deployment:

apiVersion: v1
kind: Pod
metadata:
  name: six-qemu-coco-dev
  labels:
    app: six
spec:
  runtimeClassName: kata-qemu-coco-dev
  containers:
    - name: dummy
      image: quay.io/jakub_ledworowski/test/six:latest
      command:
        - /bin/sh
        - -c
        - sleep 1000000
      imagePullPolicy: Always
---
apiVersion: v1
kind: Pod
metadata:
  name: six-runc
  labels:
    app: six
spec:
  containers:
    - name: dummy
      image: quay.io/jakub_ledworowski/test/six:latest
      command:
        - /bin/sh
        - -c
        - sleep 1000000
      imagePullPolicy: Always

3. Deploy:

kubectl apply -f whiteout-file-test.yaml

4. Execute the command on both pods to list the python pip directory:

kubectl exec -it six-runc -- ls -la /usr/local/lib/python3.10/site-packages/pip
kubectl exec -it six-qemu-coco-dev -- ls -la /usr/local/lib/python3.10/site-packages/pip

Output is as follows, note the .wh..wh..opq present only on the six-qemu-coco-dev pod:

$ kubectl exec -it six-runc -- ls -la /usr/local/lib/python3.10/site-packages/pip
total 36
drwxr-xr-x 1 root root 4096 Jan 10 13:07 .
drwxr-xr-x 1 root root 4096 Jan 10 13:08 ..
-rw-r--r-- 1 root root  357 Aug 16  2023 __init__.py
-rw-r--r-- 1 root root 1198 Aug 16  2023 __main__.py
(...)

$ kubectl exec -it six-qemu-coco-dev -- ls -la /usr/local/lib/python3.10/site-packages/pip
total 16
drwxr-xr-x 1 root root  100 Jan 10 13:07 .
drwxr-xr-x 1 root root  140 Jan 10 13:08 ..
-rwxr-xr-x 1 root root    0 Jan  1  1970 .wh..wh..opq
-rw-r--r-- 1 root root  357 Aug 16  2023 __init__.py
-rw-r--r-- 1 root root 1198 Aug 16  2023 __main__.py
(...)

How to build the image

Image is prebuilt on the quay.io/jakub_ledworowski/test/six:latest. If you want to build it yourself use the following Dockerfile:

FROM python:3.10.12-slim

RUN pip install six

More complicated example

The above example is just a showcase of much bigger problem. For more complicated Dockerfiles the .wh files present on the filesystem might interfere with the deployed applications, see the example output below:

$ kubectl exec -it more-complicated-pod -- ls -la /usr/lib/python3/dist-packages/
total 0
drwxr-xr-x 1 root root 120 Nov  7 17:37 .
drwxr-xr-x 1 root root  60 Nov  7 17:37 ..
-rw------- 1 root root   0 Nov  7 17:37 .wh._distutils_hack
-rw------- 1 root root   0 Nov  7 17:37 .wh.pkg_resources
-rw------- 1 root root   0 Nov  7 17:37 .wh.setuptools
-rw------- 1 root root   0 Nov  7 17:37 .wh.setuptools-59.6.0.egg-info
drwxr-xr-x 3 root root 100 Nov  7 17:37 _distutils_hack
drwxr-xr-x 5 root root 160 Nov  7 17:37 pip
(...)

# Error log from the deployed app
PermissionError: Permission denied: '/usr/lib/python3/dist-packages/.wh.setuptools-59.6.0.egg-info'

CoCo version information

operator v0.11.0

What TEE are you seeing the problem on

Sample

Failing command and relevant log output

No response

[JakubLedworowski]

I think this issue is related, so adding for reference: opea-project/GenAIExamples#861

[fitzthum]

Ok, let's see if we can reproduce just with image-rs.

[fitzthum]

Well when I pull quay.io/jakub_ledworowski/test/six:latest locally with image-rs I get this

rootfs/usr/local/lib/python3.10/site-packages/pip$ ls -la
total 36
drwxr-xr-x 1 root root 4096 Jan 10 07:07 .
drwxr-xr-x 1 root root 4096 Jan 10 07:08 ..
-rw-r--r-- 1 root root  357 Aug 16  2023 __init__.py
drwxr-xr-x 1 root root 4096 Jan 10 07:07 _internal
-rw-r--r-- 1 root root 1198 Aug 16  2023 __main__.py
-rw-r--r-- 1 root root 1444 Aug 16  2023 __pip-runner__.py
drwxr-xr-x 2 root root 4096 Jan 10 07:07 __pycache__
-rw-r--r-- 1 root root  286 Aug 16  2023 py.typed
drwxr-xr-x 1 root root 4096 Jan 10 07:07 _vendor

Which isn't what you show in either of your cases. I get the same result with the latest and with v0.10.0 (which is what we used in coco v0.11.0). Am I looking at the right container image?