DNS Problem after Servercrash

[patrik-upspot]

Version (k3OS / kernel)
k3os version v0.21.1-k3s1r0
5.4.0-73-generic #82 SMP Thu Jun 3 02:29:43 UTC 2021

Architecture
x86_64

Describe the bug
Hello,
i have some problems wiht my Kubernetes cluster. I use k3os version v0.21.1-k3s1r0. Last week my vHost goes offline for a short time. My provider had a failure.
Since then, my cluster does not boot up properly. Nothing has changed in the configurations.

kubectl get pods --all-namespaces
NAMESPACE                  NAME                                                     READY   STATUS                  RESTARTS   AGE
default                    svclb-ingress-nginx-controller-n9vvp                     0/2     Unknown                 0          86d
default                    ingress-nginx-controller-68649d49b8-g25vh                0/1     Unknown                 0          85d
test                       order-management-api-postgresql-postgresql-0             0/1     Unknown                 0          3d19h
k3os-system                system-upgrade-controller-8bf4f84c4-fsnbq                0/1     NodeAffinity            0          85d
test                       document-management-api-mongodb-5f46494477-nqzbl         0/1     Unknown                 0          16d
test                       notification-management-api-mongodb-8458655556-m86xx     0/1     Unknown                 0          16d
test                       document-management-api-7b6664966f-2grtb                 0/1     Unknown                 0          16d
test                       order-management-api-axon-5c4f84d668-njwzh               0/1     Unknown                 0          13d
keycloak                   keycloak-postgresql-0                                    0/1     Unknown                 0          3d19h
test                       order-management-api-mongodb-774f8549cd-cnvd6            0/1     Unknown                 0          13d
test                       user-management-api-mongodb-7cfcc95977-4jr6q             0/1     Unknown                 0          3d19h
cattle-resources-system    rancher-backup-d9d984f74-h6cn7                           0/1     Unknown                 0          16d
default                    busybox                                                  0/1     Pending                 0          28h
k3os-system                system-upgrade-controller-8bf4f84c4-gbsw7                0/1     Pending                 0          28h
default                    dnsutils                                                 0/1     Pending                 0          27h
longhorn-system            csi-snapshotter-69f8bc8dcf-dh85b                         1/1     Running                 1          36d
longhorn-system            csi-resizer-5c88bfd4cf-tfqf2                             1/1     Running                 1          36d
kube-system                coredns-7448499f4d-plr2q                                 1/1     Running                 1          85d
rancher-operator-system    rancher-operator-595ddc6db9-w4jsj                        1/1     Running                 1          85d
longhorn-system            csi-attacher-75588bff58-pdmt7                            1/1     Running                 1          36d
longhorn-system            longhorn-driver-deployer-5fc857649c-mgp8m                0/1     Init:0/1                1          36d
cattle-monitoring-system   rancher-monitoring-operator-9779869d-6r4mh               1/1     Running                 1          33d
longhorn-system            longhorn-manager-ll268                                   1/1     Running                 1          36d
longhorn-system            engine-image-ei-0f7c4304-d5l2p                           1/1     Running                 1          36d
cert-manager               cert-manager-webhook-7dc86765b4-kdsvv                    1/1     Running                 1          85d
test                       planung-ui-959b9d47c-x8djr                               1/1     Running                 1          3d9h
cattle-monitoring-system   alertmanager-rancher-monitoring-alertmanager-0           2/2     Running                 2          33d
cattle-monitoring-system   pushprox-k3s-server-client-qwhvq                         1/1     Running                 1          33d
longhorn-system            csi-resizer-5c88bfd4cf-ghxfz                             1/1     Running                 1          36d
kube-system                metrics-server-86cbb8457f-tkq6f                          1/1     Running                 1          85d
fleet-system               fleet-controller-79554fcbf5-9zw2d                        1/1     Running                 3          85d
cattle-monitoring-system   rancher-monitoring-grafana-6c9976ccc6-njtt6              3/3     Running                 3          33d
longhorn-system            engine-image-ei-a5a44787-7q78r                           1/1     Running                 1          86d
cert-manager               cert-manager-cainjector-595fc4d55-ljzn2                  1/1     Running                 1          85d
cattle-system              rancher-webhook-6cccfd96b5-v5v24                         1/1     Running                 1          85d
longhorn-system            csi-snapshotter-69f8bc8dcf-cjvnk                         1/1     Running                 1          36d
longhorn-system            csi-attacher-75588bff58-rr57b                            1/1     Running                 1          36d
longhorn-system            csi-provisioner-6968cf94f9-wlrql                         1/1     Running                 1          36d
kube-system                local-path-provisioner-5ff76fc89d-d4pdw                  1/1     Running                 1          85d
longhorn-system            csi-provisioner-6968cf94f9-fb894                         1/1     Running                 1          36d
longhorn-system            csi-provisioner-6968cf94f9-vgxks                         1/1     Running                 1          36d
cert-manager               cert-manager-58bbf5798-cs5x2                             1/1     Running                 1          85d
cattle-monitoring-system   pushprox-k3s-server-proxy-6946957f87-r75k8               1/1     Running                 1          33d
longhorn-system            csi-attacher-75588bff58-rlk42                            1/1     Running                 1          36d
longhorn-system            csi-snapshotter-69f8bc8dcf-wcfbz                         1/1     Running                 1          36d
cattle-monitoring-system   rancher-monitoring-kube-state-metrics-5f56fc44ff-vfp4c   1/1     Running                 1          33d
longhorn-system            csi-resizer-5c88bfd4cf-mqwj4                             1/1     Running                 1          36d
fleet-system               gitjob-568c57cfb9-c4mww                                  1/1     Running                 1          85d
test                       notification-management-api-bd7d67c5d-wfs4q              1/1     Running                 1          35d
fleet-system               fleet-agent-d59db746-h7nqt                               1/1     Running                 1          85d
cattle-monitoring-system   rancher-monitoring-prometheus-node-exporter-rmz2w        1/1     Running                 1          33d
cattle-monitoring-system   rancher-monitoring-prometheus-adapter-58d5b96d85-wwh45   1/1     Running                 1          33d
cattle-system              rancher-6d8598477d-h8cf2                                 1/1     Running                 1          85d
kube-system                coredns-8577c5cb4c-gw9dc                                 0/1     Pending                 0          52m
longhorn-system            longhorn-ui-64b96684bc-8jbmk                             0/1     CrashLoopBackOff        44         36d
longhorn-system            longhorn-csi-plugin-gxqb2                                1/2     CrashLoopBackOff        45         36d
test                       user-management-api-76c74dc97b-bxjtk                     0/1     CrashLoopBackOff        44         25d
keycloak                   keycloak-0                                               0/1     Init:CrashLoopBackOff   45         15d
cattle-monitoring-system   prometheus-rancher-monitoring-prometheus-0               2/3     CrashLoopBackOff        48         33d
test                       order-managament-api-7d654d8d84-jvnf5                    1/1     Running                 44         13d

kubectl -n cattle-system logs -f rancher-6d8598477d-h8cf2
2021/11/01 21:48:55 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:03 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:11 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:19 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:23 [ERROR] error syncing 'p-8mlxr/akomljen-charts': handler projectCatalog: Timeout in HTTP GET to [https://raw.githubusercontent.com/komljen/helm-charts/master/charts/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:23 [ERROR] error syncing 'p-g7r2v/wg-access-server': handler projectCatalog: Timeout in HTTP GET to [https://place1.github.io/wg-access-server/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:27 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:36 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:42 [ERROR] error syncing 'p-8mlxr/elastic': handler projectCatalog: Timeout in HTTP GET to [https://helm.elastic.co/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:42 [ERROR] error syncing 'p-g7r2v/bitnami': handler projectCatalog: Timeout in HTTP GET to [https://charts.bitnami.com/bitnami/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:44 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-7448499f4d-plr2q 1/1 Running 0 85d

kubectl describe pod  coredns-7448499f4d-plr2q --namespace=kube-system
Name:                 coredns-7448499f4d-plr2q
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 upspot-cluster/***.***.***.***
Start Time:           Sun, 08 Aug 2021 12:36:18 +0000
Labels:               k8s-app=kube-dns
                      pod-template-hash=7448499f4d
Annotations:          <none>
Status:               Running
IP:                   10.42.0.76
IPs:
  IP:           10.42.0.76
Controlled By:  ReplicaSet/coredns-7448499f4d
Containers:
  coredns:
    Container ID:  containerd://0620e642d29c8014e130153dfb461d4f2ed6c66d18d3591b69d94289288aa53e
    Image:         rancher/coredns-coredns:1.8.3
    Image ID:      docker.io/rancher/coredns-coredns@sha256:cd0f6ef34a9ee6145c0bcbc0a4511a04f81dc2c91582140f0f0250be834b5f27
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Mon, 01 Nov 2021 18:38:38 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rwbk8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-rwbk8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                   From             Message
  ----     ------                  ----                  ----             -------
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51c9ae6a65d704a3a4040261f56158523d29dfd1292e3b0cbade6d9e9bd579e4": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b72d7dbff055840ff68f3d364795d23ff6235e5db889131e20b8241f6138e8ab": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ca3f773654508caddcc648cda24d9fe10e34547b9bb515860ffe266915b4b4c4": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d01b13574e05d65dcb8656b1c7041356f93510711068d21572071525c94d6000": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f570ffd13e461cdc0853a990281617c5091cc4e004a28c9ae67c8b3d26f098ae": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe6a0c549e77626942ad9e35ac4562ed95cb1508cb56c2d625ff5ca2386e982e": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ab0b2c71df867d1e95351b8841ad89ff75266fc4362e370b7aa0aef28bb10d33": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "41da419b6c354b515f27b35548c4a95c4fc9ea9e064317126ebce7690d243332": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ed13b48e3715625a62f60f25d487790075deeec66d85998a227f54fef954be3e": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h (x3 over 31h)     kubelet          (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0e7febedb6943de8b7c295cc10afdafb0c1f848aeb85f4cc8b0ece0159eb14ef": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          27h (x1149 over 31h)  kubelet          Pod sandbox changed, it will be killed and re-created.
  Warning  FailedMount             27h                   kubelet          MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Created                 27h                   kubelet          Created container coredns
  Normal   Pulled                  27h                   kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Started                 27h                   kubelet          Started container coredns
  Warning  NodeNotReady            27h                   node-controller  Node is not ready
  Normal   Pulled                  27h                   kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 27h                   kubelet          Created container coredns
  Normal   Started                 27h                   kubelet          Started container coredns
  Warning  Unhealthy               27h                   kubelet          Readiness probe failed: Get "http://10.42.0.29:8181/ready": dial tcp 10.42.0.29:8181: connect: connection refused
  Warning  NodeNotReady            131m                  node-controller  Node is not ready
  Normal   Pulled                  125m                  kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 125m                  kubelet          Created container coredns
  Normal   Started                 125m                  kubelet          Started container coredns

kubectl describe nodes
Name:               upspot-cluster
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    k3os.io/mode=local
                    k3os.io/version=v0.21.1-k3s1r0
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=upspot-cluster
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"***"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: *.*.*.*
                    k3s.io/hostname: upspot-cluster
                    k3s.io/internal-ip: *.*.*.*
                    k3s.io/node-args: ["server","--no-deploy","traefik","--node-label","k3os.io/mode=local","--node-label","k3os.io/version=v0.21.1-k3s1r0"]
                    k3s.io/node-config-hash: L3CO4RG3V7U4SLPXOQEKR6RYZMQJODZUCK2BVNSCXA7HTV36QU7Q====
                    k3s.io/node-env:
                      {"K3S_CLUSTER_SECRET":"********","K3S_DATA_DIR":"/var/lib/rancher/k3s/data/77457d0b09d8b94d6f5029bcbc70f94b7ae9c50a08b539b76612e713ea81825...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 31 Oct 2021 12:51:56 +0000
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  upspot-cluster
  AcquireTime:     <unset>
  RenewTime:       Mon, 01 Nov 2021 20:56:50 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 01 Nov 2021 20:50:45 +0000   Mon, 01 Nov 2021 20:50:45 +0000   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:50:45 +0000   KubeletNotReady              PLEG is not healthy: pleg has yet to be successful
Addresses:
  InternalIP:  *.*.*.*
  Hostname:    upspot-cluster
Capacity:
  cpu:                8
  ephemeral-storage:  827392920Ki
  hugepages-2Mi:      0
  memory:             32886628Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  804887831945
  hugepages-2Mi:      0
  memory:             32886628Ki
  pods:               110
System Info:
  Machine ID:                 4a9d40c633f866838c2c04d460b91f75
  System UUID:                0a0f4d22-8e23-45b4-aa30-79f8b85f2989
  Boot ID:                    9773ef9f-9974-4931-b20e-b3e90fe51216
  Kernel Version:             5.4.0-73-generic
  OS Image:                   k3OS v0.21.1-k3s1r0
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.4.4-k3s2
  Kubelet Version:            v1.21.1+k3s1
  Kube-Proxy Version:         v1.21.1+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://upspot-cluster
Non-terminated Pods:          (55 in total)
  Namespace                   Name                                                      CPU Requests  CPU Limits   Memory Requests  Memory Limits  Age
  ---------                   ----                                                      ------------  ----------   ---------------  -------------  ---
  default                     svclb-ingress-nginx-controller-n9vvp                      0 (0%)        0 (0%)       0 (0%)           0 (0%)         86d
  default                     ingress-nginx-controller-68649d49b8-g25vh                 100m (1%)     0 (0%)       90Mi (0%)        0 (0%)         85d
  test                        order-management-api-postgresql-postgresql-0              250m (3%)     0 (0%)       256Mi (0%)       0 (0%)         3d18h
  test                        document-management-api-mongodb-5f46494477-nqzbl          0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        notification-management-api-mongodb-8458655556-m86xx      0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        document-management-api-7b6664966f-2grtb                  0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        order-management-api-axon-5c4f84d668-njwzh                0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
  keycloak                    keycloak-postgresql-0                                     250m (3%)     0 (0%)       256Mi (0%)       0 (0%)         3d18h
  test                        order-management-api-mongodb-774f8549cd-cnvd6             0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
  test                        user-management-api-mongodb-7cfcc95977-4jr6q              0 (0%)        0 (0%)       0 (0%)           0 (0%)         3d18h
  cattle-resources-system     rancher-backup-d9d984f74-h6cn7                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  longhorn-system             csi-snapshotter-69f8bc8dcf-dh85b                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-resizer-5c88bfd4cf-tfqf2                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 coredns-7448499f4d-plr2q                                  100m (1%)     0 (0%)       70Mi (0%)        170Mi (0%)     85d
  rancher-operator-system     rancher-operator-595ddc6db9-w4jsj                         0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-attacher-75588bff58-pdmt7                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             longhorn-driver-deployer-5fc857649c-mgp8m                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cattle-monitoring-system    rancher-monitoring-operator-9779869d-6r4mh                100m (1%)     200m (2%)    100Mi (0%)       500Mi (1%)     33d
  longhorn-system             longhorn-manager-ll268                                    0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             engine-image-ei-0f7c4304-d5l2p                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cert-manager                cert-manager-webhook-7dc86765b4-kdsvv                     0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  test                        planung-ui-959b9d47c-x8djr                                0 (0%)        0 (0%)       0 (0%)           0 (0%)         3d8h
  cattle-monitoring-system    alertmanager-rancher-monitoring-alertmanager-0            200m (2%)     1100m (13%)  150Mi (0%)       550Mi (1%)     33d
  cattle-monitoring-system    pushprox-k3s-server-client-qwhvq                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  longhorn-system             csi-resizer-5c88bfd4cf-ghxfz                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 metrics-server-86cbb8457f-tkq6f                           0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  fleet-system                fleet-controller-79554fcbf5-9zw2d                         0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    rancher-monitoring-grafana-6c9976ccc6-njtt6               100m (1%)     200m (2%)    100Mi (0%)       200Mi (0%)     33d
  longhorn-system             engine-image-ei-a5a44787-7q78r                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         86d
  cert-manager                cert-manager-cainjector-595fc4d55-ljzn2                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-system               rancher-webhook-6cccfd96b5-v5v24                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-snapshotter-69f8bc8dcf-cjvnk                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-attacher-75588bff58-rr57b                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-provisioner-6968cf94f9-wlrql                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 local-path-provisioner-5ff76fc89d-d4pdw                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-provisioner-6968cf94f9-fb894                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-provisioner-6968cf94f9-vgxks                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cert-manager                cert-manager-58bbf5798-cs5x2                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    pushprox-k3s-server-proxy-6946957f87-r75k8                0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  longhorn-system             csi-attacher-75588bff58-rlk42                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-snapshotter-69f8bc8dcf-wcfbz                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cattle-monitoring-system    rancher-monitoring-kube-state-metrics-5f56fc44ff-vfp4c    100m (1%)     100m (1%)    130Mi (0%)       200Mi (0%)     33d
  longhorn-system             csi-resizer-5c88bfd4cf-mqwj4                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  fleet-system                gitjob-568c57cfb9-c4mww                                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  test                        notification-management-api-bd7d67c5d-wfs4q               0 (0%)        0 (0%)       0 (0%)           0 (0%)         35d
  fleet-system                fleet-agent-d59db746-h7nqt                                0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    rancher-monitoring-prometheus-node-exporter-rmz2w         100m (1%)     200m (2%)    30Mi (0%)        50Mi (0%)      33d
  cattle-monitoring-system    rancher-monitoring-prometheus-adapter-58d5b96d85-wwh45    0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  cattle-system               rancher-6d8598477d-h8cf2                                  0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    prometheus-rancher-monitoring-prometheus-0                850m (10%)    1100m (13%)  1800Mi (5%)      2550Mi (7%)    33d
  longhorn-system             longhorn-ui-64b96684bc-8jbmk                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  keycloak                    keycloak-0                                                100m (1%)     0 (0%)       0 (0%)           0 (0%)         15d
  longhorn-system             longhorn-csi-plugin-gxqb2                                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  test                        user-management-api-76c74dc97b-bxjtk                      0 (0%)        0 (0%)       0 (0%)           0 (0%)         25d
  test                        order-managament-api-7d654d8d84-jvnf5                     0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                2250m (28%)  2900m (36%)
  memory             2982Mi (9%)  4220Mi (13%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:
  Type     Reason                   Age                     From        Message
  ----     ------                   ----                    ----        -------
  Warning  InvalidDiskCapacity      32h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  32h                     kubelet     Updated Node Allocatable limit across pods
  Normal   Starting                 32h                     kubelet     Starting kubelet.
  Normal   NodeHasNoDiskPressure    32h (x7 over 32h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     32h (x7 over 32h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   Starting                 32h                     kube-proxy  Starting kube-proxy.
  Normal   NodeHasSufficientMemory  27h (x105183 over 32h)  kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   Starting                 27h                     kube-proxy  Starting kube-proxy.
  Normal   Starting                 27h                     kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      27h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  27h                     kubelet     Updated Node Allocatable limit across pods
  Normal   NodeReady                27h (x3 over 27h)       kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientMemory  27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Warning  Rebooted                 27h (x92 over 27h)      kubelet     Node upspot-cluster has been rebooted, boot id: 7cb8d273-5e32-47f1-9914-cc731dc83bda
  Normal   Starting                 27h                     kube-proxy  Starting kube-proxy.
  Normal   Starting                 27h                     kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      27h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  27h                     kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             27h (x2 over 27h)       kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeReady                27h                     kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientPID     27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   NodeHasNoDiskPressure    27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientMemory  27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Warning  Rebooted                 26h (x211 over 27h)     kubelet     Node upspot-cluster has been rebooted, boot id: f9d367b9-bf7a-4998-886f-3fb4fc1a83be
  Normal   Starting                 138m                    kube-proxy  Starting kube-proxy.
  Normal   Starting                 138m                    kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      138m                    kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  138m                    kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             138m (x2 over 138m)     kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeReady                138m                    kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientMemory  138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Warning  Rebooted                 8m22s (x777 over 138m)  kubelet     Node upspot-cluster has been rebooted, boot id: 1f0cbe8e-20a6-4c87-ab20-5dd1a4a8fc68
  Normal   Starting                 6m13s                   kube-proxy  Starting kube-proxy.
  Normal   Starting                 6m9s                    kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      6m9s                    kubelet     invalid capacity 0 on image filesystem
  Normal   NodeHasNoDiskPressure    6m9s                    kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     6m9s                    kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  6m9s                    kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             6m9s (x2 over 6m9s)     kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeHasSufficientMemory  6m9s                    kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Warning  Rebooted                 64s (x32 over 6m9s)     kubelet     Node upspot-cluster has been rebooted, boot id: 9773ef9f-9974-4931-b20e-b3e90fe51216

If you need more/other infomation let me know!
Thanks for your help.