Duplicate iptables rules cause 100% ksoftirqd usage after long uptimes

[kruton]

Version (k3OS / kernel)

k3os version v1.26.15-k3s1-r1

5.15.0-124-generic #134 SMP Sat Oct 19 12:32:03 UTC 2024

Architecture

x86_64

Describe the bug

kube-router synchronizes rules periodically and uses iptables -C (check) to determine if a rule is already present in the rules table. However, the version of iptables used in the base OS of Alpine 3.19, iptables 1.8.10, has a broken check command (see cloudnativelabs/kube-router#1789).

Since the check functionality is broken, this causes kube-router to insert duplicate rules every time it runs. Eventually after thousands or tens of thousands of duplicate rules are present, ksoftirqd takes up 100% CPU continuously and any network traffic slows to a crawl as every single packet traverses the duplicate rules.

To Reproduce

Run some pods on a node and check:

sudo iptables -S | grep -E 'allow LOCAL (TCP|UDP) traffic' | sort | uniq -c

If the count for each rule is greater than 1, then this bug is affecting you.

Expected behavior

iptables -C should return the correct exit code when the rule exists and duplicate rules should not be created.

Actual behavior

Duplicate rules are created every time kube-router's sync is run.

Additional context

This is probably exacerbated by pods that start up and stop frequently. It frequently hits worker nodes which are running as CI builders that start and stop hundreds of pods a day.