High IOPS on K3s Agent Causes Node to Become Unavailable

[gaecfov]

Environmental Info:
K3s Version:
k3s version v1.31.1+k3s1 (452dbbc)
go version go1.22.6

Node(s) CPU architecture, OS, and Version:
Linux je-k3s-test-node-05 5.10.134-17.2.al8.x86_64 #1 SMP Fri Aug 9 15:49:42 CST 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 servers，6 agents

Describe the bug:

• Using iotop on the affected agent nodes, it was observed that the k3s-agent process generates a significant amount of read IO.
• This spike in IO activity coincides with the node becoming unavailable.

Steps To Reproduce:

1. Deploy a K3s cluster with multiple agent nodes.
2. Monitor the agent nodes for a period of time under normal workload.
3. Observe random occurrences of high IOPS on one of the agent nodes, leading to node unavailability.

Additional context / logs:

Timestamp: Thu Dec 5 02:03:09 PM CST 2024
Total DISK READ : 68.16 M/s | Total DISK WRITE : 122.51 K/s
Actual DISK READ: 20.26 M/s | Actual DISK WRITE: 747.97 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
b' 54680 be/4 root 9.31 M/s 0.00 B/s 0.00 % 47.29 % node_exporter --web.listen-address=:9101'
b' 1702 be/4 root 42.12 M/s 0.00 B/s 0.00 % 42.09 % k3s agent'
b' 1665 be/4 root 0.00 B/s 0.00 B/s 0.00 % 13.10 % containerd'
b' 1316 be/4 root 15.81 M/s 0.00 B/s 0.00 % 12.45 % k3s agent'
b' 1654 be/4 root 838.24 K/s 0.00 B/s 0.00 % 11.52 % containerd'

What factors are involved in the io of k3s to disk? Has anyone encountered this problem?