Unable to push images to local registry from Kubernetes Pod

[kameshsampath]

Following the document I created k3s cluster and registry. While my testing works from the local machine using the url http://k3d-myregistry.localhost:5001, but when I try to build and push image to the registry from another Kubernetes Pod I am not able to do resulting in the following error,'

/apps # nslookup k3d-myregistry.localhost
Server:         10.43.0.10
Address:        10.43.0.10:53


Name:   k3d-myregistry.localhost
Address: 192.168.144.6

/apps # curl 192.168.144.6:5001/v2
curl: (7) Failed to connect to 192.168.144.6 port 5001 after 0 ms: Connection refused
/apps # curl http://192.168.144.6:5001/v2
curl: (7) Failed to connect to 192.168.144.6 port 5001 after 0 ms: Connection refused

k3s config

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: dag
servers: 1
agents: 2
image: rancher/k3s:v1.24.3-k3s1
ports:
  # Drone CI
  - port: 127.0.0.1:30980:30980
    nodeFilters:
      - agent:*
  # Gitea
  - port: 127.0.0.1:30950:30950
    nodeFilters:
      - agent:*
  # Argo CD
  - port: 127.0.0.1:30080:30080
    nodeFilters:
      - agent:*
registries:
  create:
    name: "k3d-myregistry.localhost"
    host: "0.0.0.0"
    hostPort: "5001"
    volumes:
      - ".k3s/registry:/var/lib/registry"
  config: |
    mirrors:
      "docker.io": 
         endpoint:
           - https://registry-1.docker.io/

I used the following command to create the cluster,

k3d cluster create -c "k3s-cluster-config.yaml"

The coredns config map looks like the following after creation,

Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
    192.168.65.2 host.k3d.internal
    192.168.144.4 k3d-dag-agent-1
    192.168.144.2 k3d-dag-tools
    192.168.144.7 k3d-dag-serverlb
    192.168.144.6 k3d-myregistry.localhost
    192.168.144.3 k3d-dag-server-0
    192.168.144.5 k3d-dag-agent-0

I see the k3s pods/containers are in different subnet than docker network, is that an issue ?

[iwilltry42]

Hi @kameshsampath , thanks for starting this discussion.

First off

I see the k3s pods/containers are in different subnet than docker network, is that an issue ?

No, Kubernetes uses its own networking, unrelated to Docker's networking (which would translate to node-level networking in k3d, since containers are nodes).

But actually, I think the solution is trivial and I just didn't notice it in our Slack chat: Please try with port 5000 from within the cluster.
The hostPort: 5001 setting only changes the port exposed to your host system, but not the port that the registry is listening on.

[kameshsampath]

thanks @iwilltry42

But actually, I think the solution is trivial and I just didn't notice it in our Slack chat: Please try with port 5000 from within the cluster.
The hostPort: 5001 setting only changes the port exposed to your host system, but not the port that the registry is listening on.

so the push urls from with pods should be like k3d-myregistry.localhost:5000 ?

[iwilltry42]

Yes

[kameshsampath]

This is some thing I tried now which seem to work

REGISTRY_NAME=k3d-myregistry.localhost

docker tag gcr.io/google-samples/hello-app:1.0 "${REGISTRY_NAME}:5001/hello-app:1.0"
docker push "${REGISTRY_NAME}:5001/hello-app:1.0"
kubectl create deployment hello-server --image="${REGISTRY_NAME}:5000/hello-app:1.0"
kubectl rollout status deployment.apps/hello-server --timeout=30s
kubectl delete deployment.apps/hello-server