Snyk can't scan kaniko-produced images

[markszabo]

Actual behavior

Snyk fails to scan docker images produced by kaniko:

➜ cat Dockerfile 
FROM alpine
RUN echo 'abc' > /test.txt
➜ /kaniko/executor -f Dockerfile --no-push --destination=outimage --tarPath=out.tar
INFO[0000] Resolved base name alpine to alpine          
...
INFO[0011] Taking snapshot of full filesystem...
➜ snyk test --docker docker-archive:out.tar
Invalid Docker archive
➜ snyk test --docker oci-archive:out.tar
Invalid OCI archive

However docker loading the same image make snyk work:

➜ docker load -i out.tar
8aa28e0db101: Loading layer [==================================================>]     152B/152B
Loaded image: outimage:latest
➜ snyk test --docker outimage:latest       

Testing outimage:latest...
...
✔ Tested 14 dependencies for known issues, no vulnerable paths found.

According to our scan, you are currently using the most secure version of the selected base image

Expected behavior

I expect kaniko to produce images that can be scanned with snyk

To Reproduce

See above.

Additional Information

• Dockerfile

  FROM alpine
  RUN echo 'abc' > /test.txt
  

• Build Context
  nothing
• Snyk's blog post on container formats
• Same issue has been described on the mailing list

Workaround

We found that if the archive is copied with skopeo, then snyk can work with it:

➜  snyk test --docker docker-archive:out.tar                                                                         
Invalid Docker archive
➜  skopeo copy docker-archive:out.tar docker-archive:out2.tar
Getting image source signatures
Copying blob 8d3ac3489996 done  
Copying blob 93ebe7ef63a5 done  
Copying config 52727ba68e done  
Writing manifest to image destination
Storing signatures
➜  snyk test --docker docker-archive:out2.tar

Testing docker-archive:out2.tar...
...
✔ Tested 14 dependencies for known issues, no vulnerable paths found.

According to our scan, you are currently using the most secure version of the selected base image

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[imjasonh]

Can you share the invalid tar file, and the valid one produced by skopeo copying? Diffing them may help uncover what the problem is.

[markszabo]

Sure! Here is the file produced by kaniko: out.tar.gz and here is after copying with skopeo: out2.tar.gz (both got renamed to .tar.gz from .tar as Github doesn't support .tar files)

[tejal29]

I think I had a meeting with developers form Snyk . We spoke about changing the kaniko tar format to adhere to https://snyk.io/blog/container-image-formats.

However, we havent seen any movement from them.

[tejal29]

Related to https://groups.google.com/g/kaniko-users/c/TKE1xy7Neyo/m/2nMYl4W0AAAJ

[idoham]

I think this issue is also related to scanning docker images using Synopsis Black Duck, I am getting a similar issue " Exception thrown while getting image packages: Could not parse the image manifest file"

[lewisboon]

There was a potentially similar issue with Dive (wagoodman/dive#318).

Docker and Kaniko use different naming conventions for the config files. Dive assumed all config files would end in .json and would filter based on name. Kaniko uses the format sha254:<hash> as the file name and was filtered. Additionally, Kaniko stores layers as tar.gz, rather than plain tar files.

Looking at @markszabo's example files, we can see similar differences:

Kaniko file structure

> tar xvf out.tar.gz -C kaniko/
sha256:6f4eeeb6af6975a3a005943d019f2149d3d90815b6403ab11d527f8b21baf26a
59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3.tar.gz
39c46de6fa0d18af0c4239463ac9525d38f3d4bc1bd373a703f3699c31aa68df.tar.gz
manifest.json

Skopeo file structure

> tar xvf out2.tar.gz -C skopeo/
8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759.tar
5d8df4285410a06dc1639e7e808e90b0a25508ed5d01ad9d02e868c95299f924.tar
6f4eeeb6af6975a3a005943d019f2149d3d90815b6403ab11d527f8b21baf26a.json
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/layer.tar
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/VERSION
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/json
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/layer.tar
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/VERSION
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/json
manifest.json
repositories

[MatthiasRoels]

Is there any further update on this issue?

[R3DRUN3]

Any news on this??

[skull-squadron]

snyk appears to fail on ~50% of images. Another: docker.io/library/busybox:latest via Docker Desktop UI on Apple silicon.

[svilambi]

does Synk supports or scans kaniko produced images ? any further update on this ?

[kevin-secops-lt]

I'm seeing the same issue on my end, any updates?

[cmanfre4]

I would like to see this fixed, one way or another, as well please.

[kevin-secops-lt]

The only way I was able to get this to work is by doing the below, the solution assumed you are using aws ecr as the repo & kaniko image to build out your docker images.

Automation Pipelines General Overview:

1. Build Image w/ Kaniko
2. Skopeo - this tools here will allow you to pull from the ecr via a tar file (this is compatible w/ snyk)
3. Snyk Cli Command -snyk container test --severity-threshold=high -fail-on=all oci-archive:image.tar
4. Retag mechanism - tag it as a temp tag first, if scan is succesfull (i.e no vulns), retag back to what is the production like docker tag

This is an obvious work around, but since the team has no direction as to when it will get fixed, this is the solution I'm leveraging for automated docker scans in the pipeline.

The entire solution can be put into a docker container btw, i.e install the snyk cli + skopeo tool on a docker container.

Command for skopeo to get the ECR Token and make the call to aws ecr to pull the image layers:
def ECR_TOKEN = sh(script: "aws ecr get-login-password --region ${DOCKER_REGION} --profile ${AWS_PROFILE}", returnStdout: true).trim()
sh "set +x && skopeo copy --src-creds=AWS:${ECR_TOKEN} docker://${DOCKER_REGISTRY}/${DOCKER_REPOSITORY}:${IMAGE_TAG} oci-archive:image.tar"

@cmanfre4 @svilambi @skull-squadron @R3DRUN3 @MatthiasRoels

[jameshartig]

I was able to get it to work by doing skopeo copy docker-archive:$tar_from_kaniko oci-archive:$oci_archive_out which doesn't need ECR or any remote registry.

[sflanker]

If anybody is looking for an example repro, have a kaniko built image which snyk container test fails to scan. The skopeo copy workaround does work, but I have to use docker-archive:... for both the input and output (if I try oci-archive: for the output snyk still cannot scan the resulting tar file).

Source: https://github.com/CMS-Enterprise/batcave-knight-light/tree/main/node-server
Build with gcr.io/kaniko-project/executor:v1.23.2 on an arm64 arch using the command --context=/app --dockerfile=/app/Dockerfile --verbosity=info --no-push --tar-path=/out/image.tar

Attempting to scan with a locally build version of the snky/snky:alpine image derived from v1.1291.1 (see https://github.com/sflanker/snyk-cli/tree/feat/alpine-arm64-support), using the CLI: snyk container test --exclude-app-vulns --severity-threshold=high docker-archive:image.tar).

Initially I received Invalid OCI archive.

After running skopeo copy docker-archive:image.tar docker-archive:image.docker-to-docker.tar the scan succeeded.

Original archive: https://drive.google.com/file/d/12k0ovaUfDfhnzqkPzksTIqi3gs_7eA4R/view?usp=drive_link
Converted archive: https://drive.google.com/file/d/1AeAT01ZZwX8UxEc0A_dA9siv_G0D2bjm/view?usp=sharing

The issue is also reproducible in a CI environment running on amd64 instead of arm.

[sflanker]

It seems likely that this is a problem with snyk, not with Kaniko. Snyk has the same issue with images built using buildx with buildkit and save/exported using the --output type=docker,dest=./image.tar argument.