Built-in Traefik crashes

[AllardKrings]

Hello,

I have installed the kubernetes binary on my 2 riscv-sbc’s. Fantastic that you have built them!

However I have noticed a problem with the built-in traefic.

The traefik pod itself runs fine, however the loadbalancers crash.

kube-system helm-install-traefik-crd-989n8 0/1 Completed 0 127m
kube-system helm-install-traefik-wcfzz 0/1 Completed 2 127m
kube-system traefik-8657d6b9f4-zzbhb 1/1 Running 1 (110m ago) 124m
kube-system coredns-97b598894-gxwfr 1/1 Running 1 (110m ago) 127m
kube-system local-path-provisioner-6d44f4f9d7-2tvz5 1/1 Running 2 (109m ago) 127m
kube-system metrics-server-7c55d89d5d-zmqt8 1/1 Running 2 (109m ago) 127m
kube-system svclb-traefik-c167a2e3-f2wsl 0/2 CrashLoopBackOff 44 (4m52s ago) 93m

When I look in the logs of the containers it says:

• trap exit TERM INT
• BIN_DIR=/sbin
• check_iptables_mode
• set +e
• lsmod
• grep nf_tables
• '[' 1 '=' 0 ]
• mode=legacy
• set -e
• info 'legacy mode detected'
• echo '[INFO] ' 'legacy mode detected'
• set_legacy
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables
  [INFO] legacy mode detected
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables-save
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables-restore
• ln -sf /sbin/xtables-legacy-multi /sbin/ip6tables
• start_proxy
• echo 0.0.0.0/0
• grep -Eq :
• iptables -t filter -I FORWARD -s 0.0.0.0/0 -p TCP --dport 80 -j ACCEPT
  /usr/bin/entry: line 46: iptables: not found

I am running

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=23.10
DISTRIB_CODENAME=mantic
DISTRIB_DESCRIPTION="Ubuntu 23.10"

On 2 Starfive Visionfive SBC’s

Both have the same problem.

Maybe it has nothing to do with the K3S-binaries but with my system-configuration.

I would apprectie some tips/help.

Kind Regards Allard Krings

[chazapis]

@AllardKrings, thanks for the feedback. I will try this out and let you know if it crashes. Can you check whether the svclb-traefik-c167a2e3-f2wsl mounts the iptables binary from the system or it is supposed to include it and it is missing?

[AllardKrings]

hello Anthony,

issuing kubectl get pod svclb-traefik-82f5b39b-kmvj5 -n kube-system -o yaml

gives:

apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-04-19T07:41:16Z"
generateName: svclb-traefik-82f5b39b-
labels:
app: svclb-traefik-82f5b39b
controller-revision-hash: 8666c56fb8
pod-template-generation: "1"
svccontroller.k3s.cattle.io/svcname: traefik
svccontroller.k3s.cattle.io/svcnamespace: kube-system
name: svclb-traefik-82f5b39b-kmvj5
namespace: kube-system
ownerReferences:

• apiVersion: apps/v1
  blockOwnerDeletion: true
  controller: true
  kind: DaemonSet
  name: svclb-traefik-82f5b39b
  uid: 3eec9cbd-a16c-4eb3-92f8-dcd2bfcb1b18
  resourceVersion: "784"
  uid: f4be8419-82d7-48cd-a0c4-68c00c595a9d
  spec:
  affinity:
  nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchFields:
  - key: metadata.name
  operator: In
  values:
  - rvsvrwsv02
  automountServiceAccountToken: false
  containers:
• env:
  • name: SRC_PORT
    value: "80"
  • name: SRC_RANGES
    value: 0.0.0.0/0
  • name: DEST_PROTO
    value: TCP
  • name: DEST_PORT
    value: "80"
  • name: DEST_IPS
    value: 10.43.226.66
    image: carvicsforth/klipper-lb:v0.4.4
    imagePullPolicy: IfNotPresent
    name: lb-tcp-80
    ports:
  • containerPort: 80
    hostPort: 80
    name: lb-tcp-80
    protocol: TCP
    resources: {}
    securityContext:
    capabilities:
    add:
    • NET_ADMIN
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
• env:
  • name: SRC_PORT
    value: "443"
  • name: SRC_RANGES
    value: 0.0.0.0/0
  • name: DEST_PROTO
    value: TCP
  • name: DEST_PORT
    value: "443"
  • name: DEST_IPS
    value: 10.43.226.66
    image: carvicsforth/klipper-lb:v0.4.4
    imagePullPolicy: IfNotPresent
    name: lb-tcp-443
    ports:
  • containerPort: 443
    hostPort: 443
    name: lb-tcp-443
    protocol: TCP
    resources: {}
    securityContext:
    capabilities:
    add:
    • NET_ADMIN
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      nodeName: rvsvrwsv02
      preemptionPolicy: PreemptLowerPriority
      priority: 0
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
      sysctls:
  • name: net.ipv4.ip_forward
    value: "1"
    serviceAccount: svclb
    serviceAccountName: svclb
    terminationGracePeriodSeconds: 30
    tolerations:
• effect: NoSchedule
  key: node-role.kubernetes.io/master
  operator: Exists
• effect: NoSchedule
  key: node-role.kubernetes.io/control-plane
  operator: Exists
• key: CriticalAddonsOnly
  operator: Exists
• effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
• effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
• effect: NoSchedule
  key: node.kubernetes.io/disk-pressure
  operator: Exists
• effect: NoSchedule
  key: node.kubernetes.io/memory-pressure
  operator: Exists
• effect: NoSchedule
  key: node.kubernetes.io/pid-pressure
  operator: Exists
• effect: NoSchedule
  key: node.kubernetes.io/unschedulable
  operator: Exists
  status:
  conditions:
• lastProbeTime: null
  lastTransitionTime: "2024-04-19T07:41:16Z"
  status: "True"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2024-04-19T07:41:16Z"
  message: 'containers with unready status: [lb-tcp-80 lb-tcp-443]'
  reason: ContainersNotReady
  status: "False"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: "2024-04-19T07:41:16Z"
  message: 'containers with unready status: [lb-tcp-80 lb-tcp-443]'
  reason: ContainersNotReady
  status: "False"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2024-04-19T07:41:16Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
• containerID: containerd://75ab4e8a9fea5b582acb1783cc195c81f3033c97be3e5472055c59b775a18130
  image: docker.io/carvicsforth/klipper-lb:v0.4.4
  imageID: docker.io/carvicsforth/klipper-lb@sha256:d7cb10c117e39056f577ddbc8cca0af1110b5c1c4a8dd3150a51cf9413bf51a9
  lastState:
  terminated:
  containerID: containerd://75ab4e8a9fea5b582acb1783cc195c81f3033c97be3e5472055c59b775a18130
  exitCode: 127
  finishedAt: "2024-04-19T07:47:16Z"
  reason: Error
  startedAt: "2024-04-19T07:47:16Z"
  name: lb-tcp-443
  ready: false
  restartCount: 6
  started: false
  state:
  waiting:
  message: back-off 5m0s restarting failed container=lb-tcp-443 pod=svclb-traefik-82f5b39b-kmvj5_kube-system(f4be8419-82d7-48cd-a0c4-68c00c595a9d)
  reason: CrashLoopBackOff
• containerID: containerd://35e2a17cf6032868e0648484db0bc2b7b6b1b0a67f9ed72e67df11d2c91813e9
  image: docker.io/carvicsforth/klipper-lb:v0.4.4
  imageID: docker.io/carvicsforth/klipper-lb@sha256:d7cb10c117e39056f577ddbc8cca0af1110b5c1c4a8dd3150a51cf9413bf51a9
  lastState:
  terminated:
  containerID: containerd://35e2a17cf6032868e0648484db0bc2b7b6b1b0a67f9ed72e67df11d2c91813e9
  exitCode: 127
  finishedAt: "2024-04-19T07:47:16Z"
  reason: Error
  startedAt: "2024-04-19T07:47:16Z"
  name: lb-tcp-80
  ready: false
  restartCount: 6
  started: false
  state:
  waiting:
  message: back-off 5m0s restarting failed container=lb-tcp-80 pod=svclb-traefik-82f5b39b-kmvj5_kube-system(f4be8419-82d7-48cd-a0c4-68c00c595a9d)
  reason: CrashLoopBackOff
  hostIP: 192.168.2.27
  phase: Running
  podIP: 10.42.0.7
  podIPs:
• ip: 10.42.0.7
  qosClass: BestEffort
  startTime: "2024-04-19T07:41:16Z"

The host is running: iptables v1.8.9 (legacy)

lsmod gives:

Module Size Used by
xt_limit 12288 5
xt_NFLOG 12288 5
nfnetlink_log 28672 5
xt_physdev 12288 10
xt_multiport 20480 3
ip_set 86016 0
ipt_REJECT 12288 5
nf_reject_ipv4 16384 1 ipt_REJECT
ip6table_filter 12288 1
ip6table_nat 12288 1
ip6table_mangle 12288 1
ip6_tables 36864 3 ip6table_filter,ip6table_nat,ip6table_mangle
xt_comment 12288 170
tls 208896 0
ip_vs_rr 12288 23
xt_ipvs 16384 4
ip_vs 294912 26 ip_vs_rr,xt_ipvs
xt_REDIRECT 16384 8
xt_nat 16384 69
veth 53248 0
vxlan 204800 0
ip6_udp_tunnel 12288 1 vxlan
udp_tunnel 36864 1 vxlan
xt_policy 16384 0
iptable_mangle 12288 5
xt_mark 12288 69
xt_u32 12288 0
xt_tcpudp 20480 144
rpcsec_gss_krb5 49152 0
xt_conntrack 16384 59
xt_MASQUERADE 16384 10
nf_conntrack_netlink 77824 0
nfnetlink 24576 4 nf_conntrack_netlink,ip_set,nfnetlink_log
xfrm_user 73728 1
xfrm_algo 20480 1 xfrm_user
xt_addrtype 16384 16
iptable_filter 12288 7
iptable_nat 12288 12
nf_nat 86016 5 ip6table_nat,xt_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT
nf_conntrack 270336 7 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs,xt_REDIRECT
nf_defrag_ipv6 36864 2 nf_conntrack,ip_vs
nf_defrag_ipv4 12288 1 nf_conntrack
bpfilter 12288 0
nfsv4 1392640 1
nfs 774144 2 nfsv4
fscache 434176 1 nfs
netfs 86016 2 fscache,nfs
overlay 278528 16
rtw88_8821cu 12288 0
rtw88_8821c 90112 1 rtw88_8821cu
rtw88_usb 28672 1 rtw88_8821cu
rtw88_core 421888 2 rtw88_8821c,rtw88_usb
mac80211 2019328 2 rtw88_core,rtw88_usb
btusb 118784 0
btrtl 53248 1 btusb
btbcm 28672 1 btusb
btintel 90112 1 btusb
btmtk 16384 1 btusb
binfmt_misc 36864 1
bluetooth 1859584 6 btrtl,btmtk,btintel,btbcm,btusb
cfg80211 1634304 2 rtw88_core,mac80211
cdns3 200704 0
ecdh_generic 16384 1 bluetooth
libarc4 12288 1 mac80211
ecc 57344 1 ecdh_generic
cdns_usb_common 40960 1 cdns3
udc_core 110592 1 cdns3
ofpart 20480 0
cmdlinepart 16384 0
jh7110_tdm 20480 0
snd_soc_core 512000 1 jh7110_tdm
spi_nor 196608 0
nls_iso8859_1 12288 1
snd_compress 40960 1 snd_soc_core
cdns3_starfive 16384 0
ac97_bus 12288 1 snd_soc_core
dw_axi_dmac_platform 45056 4
mtd 143360 7 spi_nor,cmdlinepart,ofpart
axp20x_pek 16384 0
snd_pcm_dmaengine 20480 1 snd_soc_core
snd_pcm 233472 4 snd_compress,snd_soc_core,jh7110_tdm,snd_pcm_dmaengine
sfctemp 16384 0
snd_timer 65536 1 snd_pcm
pwm_starfive 16384 0
snd 167936 4 snd_timer,snd_compress,snd_soc_core,snd_pcm
soundcore 20480 1 snd
uio_pdrv_genirq 20480 0
uio 32768 1 uio_pdrv_genirq
nfsd 1105920 5
dm_multipath 61440 0
br_netfilter 40960 0
bridge 544768 1 br_netfilter
auth_rpcgss 225280 2 nfsd,rpcsec_gss_krb5
stp 12288 1 bridge
llc 16384 2 bridge,stp
nfs_acl 16384 1 nfsd
drm 946176 0
lockd 208896 2 nfsd,nfs
grace 16384 2 nfsd,lockd
efi_pstore 16384 0
backlight 36864 1 drm
sunrpc 1036288 24 nfsd,nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl,nfs
ip_tables 36864 3 iptable_filter,iptable_nat,iptable_mangle
x_tables 65536 24 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,xt_multiport,xt_NFLOG,xt_tcpudp,xt_addrtype,xt_physdev,xt_nat,xt_ipvs,xt_comment,xt_policy,ip6_tables,xt_u32,ipt_REJECT,ip_tables,iptable_nat,xt_limit,ip6table_mangle,xt_MASQUERADE,iptable_mangle,xt_REDIRECT,xt_mark
autofs4 94208 2
btrfs 2924544 0
blake2b_generic 24576 0
raid10 110592 0
raid456 356352 0
async_raid6_recov 24576 1 raid456
async_memcpy 16384 2 raid456,async_raid6_recov
async_pq 24576 2 raid456,async_raid6_recov
async_xor 24576 3 async_pq,raid456,async_raid6_recov
async_tx 20480 5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov
xor 20480 2 async_xor,btrfs
raid6_pq 102400 4 async_pq,btrfs,raid456,async_raid6_recov
libcrc32c 12288 5 nf_conntrack,nf_nat,btrfs,raid456,ip_vs
raid1 90112 0
raid0 40960 0
multipath 24576 0
linear 20480 0
motorcomm 36864 1
axp20x_regulator 65536 6
xhci_pci 32768 0
dwmac_starfive 12288 0
nvme 73728 3
stmmac_platform 40960 1 dwmac_starfive
stmmac 471040 4 dwmac_starfive,stmmac_platform
xhci_pci_renesas 32768 1 xhci_pci
nvme_core 282624 4 nvme
dw_mmc_starfive 20480 0
nvme_common 28672 1 nvme_core
pcs_xpcs 28672 1 stmmac
axp20x_i2c 12288 0
dw_mmc_pltfm 12288 1 dw_mmc_starfive
phylink 106496 2 stmmac,pcs_xpcs
axp20x 49152 1 axp20x_i2c
clk_starfive_jh7110_vout 16384 0
clk_starfive_jh7110_aon 12288 3
pinctrl_starfive_jh7110_aon 12288 0
clk_starfive_jh7110_isp 16384 0
dw_mmc 77824 1 dw_mmc_pltfm
spi_cadence_quadspi 49152 0
jh7110_trng 16384 0
phy_jh7110_usb 20480 1

If you need any more info just ask

[AllardKrings]

If I run:

docker run --cap-add SYS_ADMIN -it docker.io/carvicsforth/klipper-lb:v0.4.4 /bin/sh

and then issue “iptables”

I get:

/ # iptables
iptables v1.8.9 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.
/ #

So apparently the container default runs in nf-tables mode, the host runs in legacy mode. I am not a netwrok expert but maybe this info helps.

[AllardKrings]

Hello,

I did some more research. I found out that both hosts were running in LEGACY-mode. Switching to NFT-mode resulted in the loadbalancers running fine. Apparently from within the container the softlinks to iptables on the host work fine now.

[chazapis]

Thanks @AllardKrings for figuring this out. I'll update the documentation and link to the issue.