Mirrors configured in /etc/rancher/k3s/registries.yaml do not take effect.

[flyfax]

Environmental Info:
K3s Version:
k3s version v1.30.5+k3s1 (9b58670)
go version go1.22.6

Node(s) CPU architecture, OS, and Version:
Linux qb-test5-m1.fyre.ibm.com 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
single node k3s for testing

Describe the bug:
Originally it will pull image from icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433

But I set up registries.yaml to set up mirror registry for k3s

mirrors:
  icr.io:
    endpoint:
      - "https://docker-na-public.artifactory.test.com"
configs:
  icr.io:
    auth:
      username: <username>
      password: <pwd>

But it failed pull image from the mirror registry I configured

  Warning  Failed     9s    kubelet            Failed to pull image "icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433": failed to pull and unpack image "icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433": failed to resolve reference "icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://docker-na-public.artifactory.test.com/artifactory/api/docker/null/v2/token?scope=repository%3Acpopen%2Fedge-operator-catalog%3Apull&service=docker-na-public.artifactory.test.com: 401

But I could pull the image from mirror artifactory registry manually in k3s using

ctr images pull --user <username>:<pwd> docker-na-public.artifactory.test.com/next-gen-docker-local/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433

Is there something wrong in registries.yaml configuration? Thanks

Steps To Reproduce:

• Installed K3s:

Expected behavior:
pull image from mirror registry.
Actual behavior:
pull image from mirror registry failed.
Additional context / logs:

[brandond]

If you want to pull images from docker-na-public.artifactory.test.com with authentication, then you should add credentials for docker-na-public.artifactory.test.com instea of icr.io.

Your current configuration only contains credentials for icr.io - so it is trying your registry as a mirror for icr.io, getting an error because it does not have credentials for that registry, then trying to pull directly from icr.io with the credentials you configured for it.