cannot create cluster with latest (5.8.3) k3d version

[altfatterz]

What did you do

• How was the cluster created?

 k3d cluster create

What did you expect to happen

Start the cluster.

Screenshots or terminal output

Stuck in

INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created image volume k3d-k3s-default-images
INFO[0000] Starting new tools node...
INFO[0000] Starting node 'k3d-k3s-default-tools'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting node 'k3d-k3s-default-tools'
INFO[0002] Starting cluster 'k3s-default'
INFO[0002] Starting servers...
INFO[0002] Starting node 'k3d-k3s-default-server-0'

$ docker ps 

CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS         PORTS     NAMES
fb0581bc4cc0   ghcr.io/k3d-io/k3d-tools:5.8.3   "/app/k3d-tools noop"    4 minutes ago   Up 4 minutes             k3d-k3s-default-tools
6a54226c0bc8   rancher/k3s:v1.31.5-k3s1         "/bin/k3d-entrypoint…"   4 minutes ago   Up 4 minutes             k3d-k3s-default-server-0

$ docker logs -f 6a54226c0bc8

time="2025-02-24T13:57:18Z" level=info msg="Starting k3s v1.31.5+k3s1 (56ec5dd4)"
time="2025-02-24T13:57:18Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2025-02-24T13:57:18Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
E0224 13:57:18.291306      77 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
time="2025-02-24T13:57:18Z" level=info msg="Database tables and indexes are up to date"
E0224 13:57:18.293677      77 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0224 13:57:18.295495      77 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0224 13:57:18.297575      77 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
time="2025-02-24T13:57:18Z" level=info msg="Kine available at unix://kine.sock"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
time="2025-02-24T13:57:18Z" level=info msg="Bootstrap key locked for initial create"
time="2025-02-24T13:57:18Z" level=info msg="generated self-signed CA certificate CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18.310313311 +0000 UTC notAfter=2035-02-22 13:57:18.310313311 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:k3s-supervisor,O=system:masters signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="generated self-signed CA certificate CN=k3s-server-ca@1740405438: notBefore=2025-02-24 13:57:18.314877077 +0000 UTC notAfter=2035-02-22 13:57:18.314877077 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="generated self-signed CA certificate CN=k3s-request-header-ca@1740405438: notBefore=2025-02-24 13:57:18.316658366 +0000 UTC notAfter=2035-02-22 13:57:18.316658366 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1740405438: notBefore=2025-02-24 13:57:18.317522163 +0000 UTC notAfter=2035-02-22 13:57:18.317522163 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1740405438: notBefore=2025-02-24 13:57:18.318261761 +0000 UTC notAfter=2035-02-22 13:57:18.318261761 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=info msg="Saving cluster bootstrap data to datastore"
time="2025-02-24T13:57:18Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1740405438: notBefore=2025-02-24 13:57:18 +0000 UTC notAfter=2026-02-24 13:57:18 +0000 UTC"
time="2025-02-24T13:57:18Z" level=warning msg="dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2025-02-24T13:57:18Z" level=info msg="Active TLS secret / (ver=) (count 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.18.0.3:172.18.0.3 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-k3s-default-server-0:k3d-k3s-default-server-0 listener.cattle.io/cn-k3d-k3s-default-serverlb:k3d-k3s-default-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=5665947546F73F0E1CA290F0E42C8289BB09D560]"
time="2025-02-24T13:57:18Z" level=info msg="Bootstrap key lock is held"
time="2025-02-24T13:57:18Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2025-02-24T13:57:18Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2025-02-24T13:57:18Z" level=info msg="Waiting for API server to become available"
W0224 13:57:18.712510      76 registry.go:256] calling componentGlobalsRegistry.AddFlags more than once, the registry will be set by the latest flags
time="2025-02-24T13:57:18Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
W0224 13:57:18.714927      76 registry.go:256] calling componentGlobalsRegistry.AddFlags more than once, the registry will be set by the latest flags
time="2025-02-24T13:57:18Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
time="2025-02-24T13:57:18Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
I0224 13:57:18.717203      76 options.go:228] external host was not specified, using 172.18.0.3
time="2025-02-24T13:57:18Z" level=info msg="To join server node to cluster: k3s server -s https://172.18.0.3:6443 -t ${SERVER_NODE_TOKEN}"
time="2025-02-24T13:57:18Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
time="2025-02-24T13:57:18Z" level=info msg="To join agent node to cluster: k3s agent -s https://172.18.0.3:6443 -t ${AGENT_NODE_TOKEN}"
time="2025-02-24T13:57:18Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2025-02-24T13:57:18Z" level=info msg="Run: k3s kubectl"
I0224 13:57:18.719947      76 server.go:150] Version: v1.31.5+k3s1
time="2025-02-24T13:57:18Z" level=warning msg="Failed to find cpuset cgroup, you may need to add \"cgroup_enable=cpuset\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
time="2025-02-24T13:57:18Z" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
I0224 13:57:18.720015      76 server.go:152] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
time="2025-02-24T13:57:18Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
E0224 13:57:18.831281      99 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0224 13:57:18.833696      99 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0224 13:57:18.835433      99 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0224 13:57:18.837570      99 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Which OS & Architecture

$  k3d runtime-info
arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 27.5.1


$ k3d version

k3d version v5.8.3
k3s version v1.31.5-k3s1 (default)

Which version of docker

$ docker version
Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:37:19 2025
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[altfatterz]

I get the following in the logs:

$ docker logs -f k3d-k3s-default-server-0

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
E0226 20:49:03.297508     121 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0226 20:49:03.299996     121 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0226 20:49:03.302115     121 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0226 20:49:03.304136     121 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

I see others have similar issue: https://slack-archive.rancher.com/t/26982322/hey-community-i-have-a-bit-strange-issue-with-k3d-im-trying-

Does anybody know how to solve it?

[altfatterz]

The issue was with my Docker runtime: Docker Desktop on Mac.

The following steps fixed my issue:

• open the Docker Desktop app and head to the dashboard.
• select the Troubleshoot icon and click Reset to factory defaults.
• Next, click the Yes, reset anyway and the task is done.