Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kind Load Image Digest Doesn't Work #2394

Open
midnightconman opened this issue Jul 28, 2021 · 10 comments
Open

Kind Load Image Digest Doesn't Work #2394

midnightconman opened this issue Jul 28, 2021 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@midnightconman
Copy link

What happened:
I loaded a docker image by digest and kind seemed to load the image correctly. But, unfortunately the image wasn't loaded.

Here is a redacted snippet of the load command and response:

]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...
]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...
]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...

What you expected to happen:
I expected the image digest to be loaded into kind and to be accessible via it's digest.

How to reproduce it (as minimally and precisely as possible):
Try to load an image digest into kind and try to use that image digest, or try to look for that image digest via crictl.

Anything else we need to know?:

Environment:

  • kind version: (use kind version): kind v0.11.1 go1.16.4 linux/amd64
  • Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version: (use docker info):
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df35096d
 Built:             Fri Jun  4 08:14:39 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.4
  Git commit:       b0f5bc36fe
  Built:            Fri Jun  4 08:14:24 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.2
  GitCommit:        36cc874494a56a253cd181a1a685b44b58a2e34a.m
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  • OS (e.g. from /etc/os-release):
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux
@midnightconman midnightconman added the kind/bug Categorizes issue or PR as related to a bug. label Jul 28, 2021
@BenTheElder
Copy link
Member

I expected the image digest to be loaded into kind and to be accessible via it's digest.

You can load by a digest but you cannot reference it by the digest you pulled. You can list the images in the cluster and see that there is an image loaded, but not with the digest you desired.

This is because image digests hash the entire image including the metadata that contains the repo tags in the image, and they will not match.

Another common issue is referencing an image by the digest of a manifest list, but manifest lists can only be stored in a registry, not in docker/containerd.

Basically you're seeing what happens when you docker save + ctr images import, kind does nothing itself to break anything.

@BenTheElder
Copy link
Member

You can push to a local registry instead (see the docs for how to set one up), but the digests will again change because the hostname will change.

@BenTheElder
Copy link
Member

I think it may be possible to make this work with some manipulation of the image as loaded to have identical metadata but this has not ever worked in kind and is not a straightforward bug.

@mitar
Copy link
Contributor

mitar commented Mar 8, 2022

Lol, this is really strange design that digest of Docker images are registry-bound.

@BenTheElder
Copy link
Member

possibly related 🤦 #2734

have not had time to dig back into this.

You can push to a local registry instead (see the docs for how to set one up), but the digests will again change because the hostname will change.

That's not accurate, I was thinking of the docker save format, not the registry API.

However, digests are frequently pinned to multi-arch images, but kind load docker-image must be for a particular architecture and not the manifest, because images in local storage / saved by docker save are for a specific arch (one of the manifests, behind the manifest list indirection).

@BenTheElder
Copy link
Member

#2756 🤦‍♂️

See also: #2955 (not released yet, but fixing a regression in image loading)

@aclevername
Copy link

Hey @BenTheElder 👋

Just to chime in I think I'm seeing the same issue with kind v0.17.0 (I also tried building off main at 4a0de915). If I load the image via load docker-image:

kind load docker-image gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6 --name kind

Image: "" with ID "sha256:1e7e67348c2fce975e89c1670537a68ff1e1131467d94f42d8d8fb8f9a15cb4b" not yet present on node "kind-control-plane", loading...

or from a tarball via load image-archive (note: kind show nothing on stdout, but does exit 0)

kind load image-archive gcr.io__knative-releases__knative.dev__operator__cmd__operator.tar --name kind

Neither work, as the image isn't present:

docker exec -it kind-control-plane crictl images
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20221004-44d545d1   7ba9b35cf55e6       23.7MB
docker.io/kindest/local-path-helper        v20220607-9a4d8d2a   304f67e47fc80       2.75MB
ghcr.io/fluxcd/helm-controller             v0.27.0              5fd6014d29477       71.4MB
registry.k8s.io/coredns/coredns            v1.9.3               b19406328e70d       13.4MB
registry.k8s.io/kube-controller-manager    v1.25.3              05b17bba8656e       62.3MB
docker.io/library/import-2023-01-19        <none>               1e7e67348c2fc       73.7MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       7902f9a1c54fa       15.6MB
registry.k8s.io/etcd                       3.5.4-0              8e041a3b0ba8b       81.1MB
registry.k8s.io/kube-apiserver             v1.25.3              feafd6a91eb52       74.2MB
registry.k8s.io/kube-proxy                 v1.25.3              aa31a9b19ccdf       59.6MB
registry.k8s.io/kube-scheduler             v1.25.3              253d0aeea8c69       50.6MB
registry.k8s.io/pause                      3.7                  e5a475a038057       268kB

I know this issue isn't closed so it makes sense it still doesn't work, just wanted to +1 the issue. I'd also be happy to contribute a fix if you can point me in the right direction of what changes need to be made 😄

@BenTheElder
Copy link
Member

BenTheElder commented Jan 19, 2023

The problem is that gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6 is a multi-arch manifest-list, which is resolved by docker pull / docker save to the host-platform specific manifest / image.

$ crane manifest gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:a71bad45fe686851238a86408e9cdfcb4dda14b6974e131911bc7fda27e3cc98","platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:403cb4e0b91443365813de9df862073302e47122f5441e63188748256fbd3426","platform":{"architecture":"arm","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:e87490b5b47027452692589cf995c1432ee773293f91b2b0e87c3e40cf22a7f9","platform":{"architecture":"arm64","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:c6ec9ae95baad76e76a02542bdc7a9d216cbec53909c34d724a10ba55fe83f4a","platform":{"architecture":"ppc64le","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:22b8326043ba1173ee1a72c5ebf936a5e3f839225d1937d1dd5b842049d1672c","platform":{"architecture":"s390x","os":"linux"}}]}

kind load docker-image cannot load a multi-arch image manifest, docker pull / docker save do not support this. Docker stores one platform per ref locally.

I haven't had time to dig into if there's a reasonable approach to solve this in an expected and compatible way.

@BenTheElder
Copy link
Member

I think to do this sort of thing properly we'd need a different command with some custom logic to:

  • pull to some cache directory instead of docker's existing image store
  • record the original multi-arch image digest
  • attach the multi-arch digest to the imported image

Which could get tricky fast ...

The OCI layout format using someting like crane pull has this information, but it's natively a directory structure not a single file like the docker archive tars, so plumbing it to the running node container is going to be clunky, and again we lose the expectation of de-duping storage on the host at least into your container runtime's image store (along with the existing controls for listing, removing, and pruning images etc)


We might be able to do something hackier and whenever we see we're loading a digest we instruct containerd to attach the digest ref to the imported image. Maybe by exec-ing ctr images tag.

@jglick
Copy link

jglick commented Apr 10, 2024

Related to #1140? I am having trouble preloading images from the ingress-nginx chart since the chart specifies both a tag and a hash. If I preload the images with just the tag, then it looks like the layers are there on the node, but Kind still has to pull the manifest (taking ~1s but of course requiring the network); if I preload the images with both the tag and hash (verbatim as in the chart) then the pods fail to start ever (it seems like CRI gets confused). Can try to come up with a self-contained test case if this is unknown.

abhijeet-dhumal added a commit to abhijeet-dhumal/kueue that referenced this issue Sep 16, 2024
abhijeet-dhumal added a commit to abhijeet-dhumal/kueue that referenced this issue Sep 16, 2024
k8s-ci-robot pushed a commit to kubernetes-sigs/kueue that referenced this issue Sep 16, 2024
#2570)

* Use env variable for the test sleep image in case of using other image reference or same image but with digest in case of disconnected environment

* Update E2e test sleep image to use image digest instead of version tag

* Incase of kind cluster, manually create tag for image with digest which is already pulled. For more information : kubernetes-sigs/kind#2394
kannon92 pushed a commit to openshift-kannon92/kubernetes-sigs-kueue that referenced this issue Nov 19, 2024
kubernetes-sigs#2570)

* Use env variable for the test sleep image in case of using other image reference or same image but with digest in case of disconnected environment

* Update E2e test sleep image to use image digest instead of version tag

* Incase of kind cluster, manually create tag for image with digest which is already pulled. For more information : kubernetes-sigs/kind#2394
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants