Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k3s fails and restarts, Fedora 32 #2105

Closed
krystalcode opened this issue Aug 7, 2020 · 2 comments
Closed

k3s fails and restarts, Fedora 32 #2105

krystalcode opened this issue Aug 7, 2020 · 2 comments

Comments

@krystalcode
Copy link

krystalcode commented Aug 7, 2020 •
edited by davidnuzik
Loading

Environmental Info:
K3s Version:
k3s version v1.18.6+k3s1 (6f56fa1)

Node(s) CPU architecture, OS, and Version:
Linux krystalcode 5.7.11-200.fc32.x86_64 #1 SMP Wed Jul 29 17:15:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single-machine install

Describe the bug:
Installed on a fresh Fedora 31 workstation installation with the get.k3s.io script (since then upgraded to Fedora 32 - no behavior change). I disabled swap and SELinux to make sure they're not the issue. k3s fails and restarts continuously.

I see the error when checking configuration: cgroup hierarchy: nonexistent?? (fail) but the referenced link https://github.com/tianon/cgroupfs-mount mentions that If you're using systemd as your init system, you do not need this. The other two errors seem to be due to module renaming.

Steps To Reproduce:

  • Installed K3s:
    With script: curl -sfL https://get.k3s.io | sh -
    Install package for SELinux support: rpm -i https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm
    Disabled swap: swapoff -a
    Set SELinux to permissive mode: setenforce 0

Expected behavior:
More details on the error.

Actual behavior:
k3s service fails and restarts continuously.

Additional context / logs:

$ sudo systemctl status k3s
k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2020-08-07 16:15:32 -05; 2s ago
       Docs: https://k3s.io
    Process: 26655 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 26656 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 26657 ExecStart=/usr/local/bin/k3s server (code=exited, status=1/FAILURE)
   Main PID: 26657 (code=exited, status=1/FAILURE)
        CPU: 6.069s

$ sudo systemctl status k3s
k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: activating (start) since Fri 2020-08-07 16:15:37 -05; 1s ago
       Docs: https://k3s.io
    Process: 26761 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 26762 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 26763 (k3s-server)
      Tasks: 58
     Memory: 395.0M
        CPU: 4.301s
     CGroup: /system.slice/k3s.service
             └─26763 /usr/local/bin/k3s server

Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.752728   26763 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.758787   26763 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.766117   26763 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.777112   26763 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.779155   26763 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.787220   26763 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.797774   26763 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: W0807 16:15:37.797785   26763 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: I0807 16:15:37.803025   26763 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestrict>
Aug 07 16:15:37 desktop.krystalcode k3s[26763]: I0807 16:15:37.803033   26763 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClai>

$ sudo k3s check-config
Verifying binaries in /var/lib/rancher/k3s/data/6ab65ee0083924a980f6161177c37163fa68681b710316bb3dc59f118ff27636/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.4 (legacy): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: module configs not found in modules.dep
info: reading kernel config from /boot/config-5.7.11-200.fc32.x86_64 ...

Generally Necessary:
- cgroup hierarchy: nonexistent?? (fail)
    (see https://github.com/tianon/cgroupfs-mount)
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: missing (fail)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: missing (fail)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 3 (fail)
@brandond
Copy link
Member

brandond commented Aug 8, 2020

You may need to use the kernel args from #2049 (comment) in order to switch back to legacy cgroups, until cgroupv2 support is available.

@krystalcode
Copy link
Author

krystalcode commented Aug 8, 2020 •
edited
Loading

Thanks, properly running and cgroup hierarchy: properly mounted [/sys/fs/cgroup] after the following: grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" and reboot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@krystalcode @brandond