containerd cannot pull kube-apiserver-healthcheck

[hakman]

1. What kops version are you running? The command kops version, will display
this information.

kops create cluster \
  --name e2e-kops-aws-containerd.test-cncf-aws.k8s.io \
  --ssh-public-key /workspace/.ssh/kube_aws_rsa.pub \
  --node-count 4 \
  --node-volume-size 48 \
  --master-volume-size 48 \
  --master-count 1 \
  --zones ap-southeast-1a \
  --master-size c5.large \
  --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.18.2 \
  --admin-access 35.224.122.237/32 \
  --cloud aws \
  --container-runtime=containerd \
  --networking=calico \
  --image=099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200423 \
  --override cluster.spec.nodePortAccess=0.0.0.0/0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.18.0-alpha.4+de7cc70c3

3. What cloud provider are you using?
aws

4. What commands did you run? What is the simplest way to reproduce this issue?
n/a

5. What happened after the commands executed?
https://storage.googleapis.com/kubernetes-jenkins/logs/e2e-kops-aws-misc-containerd/1258780879564574720/artifacts/ip-172-20-48-115.ap-southeast-1.compute.internal/containerd.log

time="2020-05-08T15:33:38.838738725Z" level=info msg="PullImage \"kope/kube-apiserver-healthcheck:1.18.0-alpha.3\""
time="2020-05-08T15:33:43.206611091Z" level=error msg="PullImage \"kope/kube-apiserver-healthcheck:1.18.0-alpha.3\" failed" error="failed to pull and unpack image \"docker.io/kope/kube-apiserver-healthcheck:1.18.0-alpha.3\": failed to resolve reference \"docker.io/kope/kube-apiserver-healthcheck:1.18.0-alpha.3\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"

6. What did you expect to happen?
Repo should be public.

9. Anything else do we need to know?
There are 2 issues here:

1. the kope/kube-apiserver-healthcheck should be public
2. containerd should handle images that are preloaded

https://storage.googleapis.com/kubernetes-jenkins/logs/e2e-kops-aws-misc-containerd/1258780879564574720/artifacts/ip-172-20-48-115.ap-southeast-1.compute.internal/kops-configuration.log

I0508 15:32:05.716396    1838 http.go:77] Downloading "https://storage.googleapis.com/kops-ci/bin/1.18.0-alpha.4+de7cc70c3/images/kube-apiserver-healthcheck.tar.gz"
W0508 15:32:05.715150    1838 load_image.go:86] LoadImageTask checking if image present not yet implemented
I0508 15:32:06.285378    1838 files.go:100] Hash matched for "/var/cache/nodeup/sha256:7d972b9cade8071142da40b097c29acbb23b3acfd7c94b8421fd7bfcff2ca728_https___storage_googleapis_com_kops-ci_bin_1_18_0-alpha_4_de7cc70c3_images_kube-apiserver-healthcheck_tar_gz": sha256:7d972b9cade8071142da40b097c29acbb23b3acfd7c94b8421fd7bfcff2ca728
I0508 15:32:07.224699    1838 load_image.go:168] running command ctr images import /tmp/loadimage453482291/https___storage_googleapis_com_kops-ci_bin_1_18_0-alpha_4_de7cc70c3_images_kube-apiserver-healthcheck_tar_gz