New Node (arm64), Ready but not used, Warning: InvalidDiskCapacity

[Masegi]

Hi,

I'm trien to scale up my cluster, when I increment the instance_count and rerun the script everything looks fine so far, but when I check my pods, none is actually running on the new workers.

So I ran describe command for the nodes and I am getting this:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                130m (0%)   0 (0%)
  memory             150Mi (0%)  400Mi (1%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
  hugepages-32Mi     0 (0%)      0 (0%)
  hugepages-64Ki     0 (0%)      0 (0%)
Events:
  Type     Reason                   Age                    From                   Message
  ----     ------                   ----                   ----                   -------
  Normal   Starting                 4m25s                  kube-proxy
  Normal   Synced                   4m27s                  cloud-node-controller  Node synced successfully
  Normal   RegisteredNode           4m27s                  node-controller        Node tdhcrm-cax41-pool-static2-fsn1-worker1 event: Registered Node tdhcrm-cax41-pool-static2-fsn1-worker1 in Controller
  Normal   Starting                 4m27s                  kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity      4m27s                  kubelet                invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  4m27s (x2 over 4m27s)  kubelet                Node tdhcrm-cax41-pool-static2-fsn1-worker1 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    4m27s (x2 over 4m27s)  kubelet                Node tdhcrm-cax41-pool-static2-fsn1-worker1 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     4m27s (x2 over 4m27s)  kubelet                Node tdhcrm-cax41-pool-static2-fsn1-worker1 status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  4m27s                  kubelet                Updated Node Allocatable limit across pods
  Normal   NodeReady                4m26s                  kubelet                Node tdhcrm-cax41-pool-static2-fsn1-worker1 status is now: NodeReady

How can I make my cluster use this worker please?

I have tried it multiple times, I also tried adding a new node pool instead of increment the count, but was same result.

[vitobotta]

It seems that the node is fine according to the output of the describe command. On top of my head two scenarios I can think of are 1) you are running some image not built for arm, 2) perhaps there was still capacity on the other nodes so this was simply hasn't been picked up yet. Which application are you expecting to run on this node? Have you configured cpu and memory requests? If not that might also be a reason.

Edit: I confused the comment a bit so forget the arm part.

[Masegi]

Hi I have this app already running on this cluster - all nodes are arm64, and I just incremented the replication number but all new pods spawned on old workers.
I have not done anything cpu/memory related, all I did was choosing what server type (cax41)

[vitobotta]

If you haven't set reequests for CPU and memory, Kubernetes doesn't know how much resources the workload needs to it may just still add pods to the existing nodes. Please see https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.

Converting into a discussion since it's not an issue with the tool itself.

[Masegi]

Sorry, its solved now, I dont want to set cpu/mem limits or requests, so I did add this to my spec for only one pod of this application per node:

    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: my-app