Use HTTP for Registry Pull

[albal]

Environmental Info:
K3s Version:
v1.30.2+k3s1

Node(s) CPU architecture, OS, and Version:

Linux k3-hpc-dev-1 5.14.0-427.40.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Oct 16 14:57:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 Servers and 6 Agents - otherwise functioning.

Describe the bug:

I'm trying to get a pull through proxy working with Harbor. My client doesn't want to use publicly signed certificates and they have approved that the connection from the kubelet to the Harbor registry can be over http.

I have created a registries.yaml file in /etc/etc/rancher/k3s:

mirrors:
  "harbor.redacted.uk:80":
    endpoint:
      - "http://harbor.redacted.uk:80"

configs:
  "harbor.redacted.uk:80":
    tls:
      # Tells containerd not to attempt TLS
      insecure_skip_verify: true

My mutating webhook is working as expected and when the image is being pulled by the kubelet it is still trying https:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  45m                  default-scheduler  Successfully assigned default/ubby3 to k3-hpc-dev-1
  Normal   Pulling    43m (x4 over 45m)    kubelet            Pulling image "harbor.redacted.uk:80/proxy/library/ubuntu:22.04"
  Warning  Failed     43m (x4 over 45m)    kubelet            Failed to pull image "harbor.redacted.uk:80/proxy/library/ubuntu:22.04": failed to pull and unpack image "harbor.redacted.uk:80/proxy/library/ubuntu:22.04": failed to resolve reference "harbor.redacted.uk:80/proxy/library/ubuntu:22.04": failed to do request: Head "https://harbor.redacted.uk:80/v2/proxy/library/ubuntu/manifests/22.04": http: server gave HTTP response to HTTPS client
  Warning  Failed     43m (x4 over 45m)    kubelet            Error: ErrImagePull
  Warning  Failed     43m (x6 over 45m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    14s (x198 over 45m)  kubelet            Back-off pulling image "harbor.redacted.uk:80/proxy/library/ubuntu:22.04"

Steps To Reproduce:

• Installed K3s:
• installed harbor
• setup proxy registry to dockerhub
• created regsitries.yaml with contents above
• create daemonset to deploy file on all nodes
• verify registries.yaml is on some nodes

Expected behavior:
Kubelet should pull over http and https or TLS should not come into play

Actual behavior:
Kubelet tries https connection, even on port 80.

[brandond]

      # Tells containerd not to attempt TLS
      insecure_skip_verify: true

That is not what that does. It tells it not to verify TLS certs. It does NOT tell it to use plaintext HTTP.

Check containerd.log. I suspect that it is also failing to pull via HTTP, and is failing back to the default endpoint (which uses HTTPS). The kubelet only shows the terminal failure message. See https://docs.k3s.io/installation/private-registry#default-endpoint-fallback

[albal]

Thank you for your swift response and a pointer to the correct documentation. I have changed my registry.yaml to this and it is all working:

mirrors:
  docker.io:
    endpoint:
      - "http://harbor.redacted.uk:80"