k3s no space left on device #2682

kaiehrhardt · 2024-06-28T23:24:05Z

Jun 28 23:15:43 node-server-56d1 sh[1999]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jun 28 23:15:43 node-server-56d1 k3s[2006]: time="2024-06-28T23:15:43Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Jun 28 23:15:43 node-server-56d1 k3s[2006]: time="2024-06-28T23:15:43Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14"
Jun 28 23:15:44 node-server-56d1 k3s[2006]: time="2024-06-28T23:15:44Z" level=info msg="error extracting tarball into /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14-tmp after 20 files, 2 dirs, 778.839239ms: error writing to /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14-tmp/bin/k3s: write /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14-tmp/bin/k3s: no space left on device"
Jun 28 23:15:44 node-server-56d1 k3s[2006]: time="2024-06-28T23:15:44Z" level=fatal msg="extracting data: error writing to /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14-tmp/bin/k3s: write /var/lib/rancher/k3s/data/019fb8bac4d619f2f3682a8aef4f460c4759bd4d1510b12b05b498936e75da14-tmp/bin/k3s: no space left on device"
Jun 28 23:15:44 node-server-56d1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

Kairos version:

NAME="openSUSE Leap"
VERSION="15.6"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.6"
PRETTY_NAME="openSUSE Leap 15.6"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.6"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
KAIROS_IMAGE_LABEL="leap-15.6-standard-arm64-rpi4-v3.1.0-rc1-k3sv1.30.1-k3s1"
KAIROS_FLAVOR_RELEASE="leap-15.6"
KAIROS_RELEASE="v3.1.0-rc1"
KAIROS_SOFTWARE_VERSION="v1.30.1+k3s1"
KAIROS_MODEL="rpi4"
KAIROS_ID="kairos"
KAIROS_VERSION="v3.1.0-rc1-v1.30.1-k3s1"
KAIROS_ARTIFACT="kairos-opensuse-leap-15.6-standard-arm64-rpi4-v3.1.0-rc1-k3sv1.30.1+k3s1"
KAIROS_FLAVOR="opensuse"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_FAMILY="opensuse"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:leap-15.6-standard-arm64-rpi4-v3.1.0-rc1-k3sv1.30.1-k3s1"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_NAME="kairos-standard-opensuse-leap-15.6"
KAIROS_ID_LIKE="kairos-standard-opensuse-leap-15.6"
KAIROS_VERSION_ID="v3.1.0-rc1-v1.30.1-k3s1"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-leap-15.6 v3.1.0-rc1-v1.30.1-k3s1

CPU architecture, OS, and Version:

Linux node-server-56d1 6.4.0-150600.23.7-default #1 SMP PREEMPT_DYNAMIC Fri Jun 14 14:33:11 UTC 2024 (33f31da) aarch64 aarch64 aarch64 GNU/Linux

Logs
df -h

Filesystem                         Size  Used Avail Use% Mounted on
/dev/loop0                         2.0G  1.4G  455M  76% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   55M   55M     0 100% /usr/local
overlay                            2.0G   13M  2.0G   1% /var
overlay                            2.0G   13M  2.0G   1% /etc
overlay                            2.0G   13M  2.0G   1% /srv
devtmpfs                           4.0M  4.0K  4.0M   1% /dev
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              1.6G  8.9M  1.6G   1% /run
tmpfs                              2.0G   13M  2.0G   1% /run/overlay
tmpfs                              3.9G  8.0K  3.9G   1% /tmp
/dev/sda2                          5.9G  2.8G  2.9G  50% /run/initramfs/cos-state
tmpfs                              784M  4.0K  784M   1% /run/user/1000

lsblk

AME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0     2G  1 loop /
sda      8:0    0 238.5G  0 disk 
├─sda1   8:1    0    96M  0 part 
├─sda2   8:2    0   6.1G  0 part /run/initramfs/cos-state
├─sda3   8:3    0   4.1G  0 part 
├─sda4   8:4    0    64M  0 part /oem
└─sda5   8:5    0 228.2G  0 part /run/cilium
                                 /usr/local/.state/usr-share-pki-trust.bind/anchors
                                 /usr/share/pki/trust/anchors
                                 /usr/share/pki/trust
                                 /var/lib/wicked
                                 /var/lib/tailscale
                                 /var/lib/snapd
                                 /var/lib/rook
                                 /var/lib/rancher
                                 /var/lib/osd
                                 /var/lib/longhorn
                                 /var/lib/kubelet
                                 /var/lib/extensions
                                 /var/lib/dbus
                                 /var/lib/containerd
                                 /var/lib/cni
                                 /var/lib/calico
                                 /var/lib/ca-certificates
                                 /etc/zfs
                                 /etc/systemd
                                 /etc/sysconfig
                                 /etc/ssh
                                 /var/snap
                                 /etc/runlevels
                                 /etc/rancher
                                 /etc/pwx
                                 /etc/modprobe.d
                                 /var/log
                                 /usr/libexec
                                 /etc/kubernetes
                                 /etc/iscsi
                                 /etc/init.d
                                 /var/cores
                                 /etc/cni
                                 /root
                                 /opt
                                 /home
                                 /usr/local

cat /run/immucore/rootfs_stage.log

2024-06-24T12:00:15Z INF Running stage: rootfs.before

2024-06-24T12:00:15Z INF Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z ERR mkdir /etc/systemd/network/: file exists
2024-06-24T12:00:15Z ERR 1 error occurred:
	* mkdir /etc/systemd/network/: file exists


2024-06-24T12:00:15Z ERR Failed to connect to system bus: No such file or directory
: failed to run networkctl reload: exit status 1
2024-06-24T12:00:15Z ERR 1 error occurred:
	* failed to run networkctl reload: exit status 1


2024-06-24T12:00:15Z ERR no metadata/userdata found
2024-06-24T12:00:15Z INF Processing stage step 'Sentinel file for userdata'. ( commands: 0, files: 1, ... )
2024-06-24T12:00:15Z INF Done executing stage 'rootfs.before'

2024-06-24T12:00:15Z INF Running stage: rootfs

2024-06-24T12:00:15Z INF Processing stage step 'Layout configuration for active/passive mode'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/cos/recovery_mode" ] && grep -vq "rd.immucore.uki" /proc/cmdline: exit status 1)' stage name: Layout configuration for recovery mode
2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run grep -q "kairos.boot_live_mode" /proc/cmdline: exit status 1)' stage name: Layout configuration for booting local node from livecd
2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -e "/run/cos/uki_boot_mode" ] && [ ! -e "/run/cos/recovery_mode" ] && [ ! -e "/run/cos/autoreset_mode" ]: exit status 1)' stage name: Layout configuration for UKI boot
2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -e "/run/cos/uki_boot_mode" ] && ([ -e "/run/cos/recovery_mode" ] || [ -e "/run/cos/autoreset_mode" ]): exit status 1)' stage name: Layout configuration for recovery/autoreset mode on UKI
2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -e "/run/cos/uki_install_mode" ]: exit status 1)' stage name: Layout configuration for UKI installer
2024-06-24T12:00:15Z INF Done executing stage 'rootfs'

2024-06-24T12:00:15Z INF Running stage: rootfs.after

2024-06-24T12:00:15Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -r /run/cos/custom-layout.env ] && [ ! -f "/run/cos/recovery_mode" ] && [ ! -f /run/cos/live_mode ]: exit status 1)' stage name: add custom bind and ephemeral mounts to /run/cos/cos-layout.env
2024-06-24T12:00:15Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z WRN Not enough unpartitioned space in disk to operate
2024-06-24T12:00:15Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z WRN Not enough unpartitioned space in disk to operate
2024-06-24T12:00:15Z INF Done executing stage 'rootfs.after'

2024-06-24T12:00:15Z INF Running stage: rootfs.before

2024-06-24T12:00:15Z INF Done executing stage 'rootfs.before'

2024-06-24T12:00:15Z INF Running stage: rootfs

2024-06-24T12:00:15Z INF Done executing stage 'rootfs'

2024-06-24T12:00:15Z INF Running stage: rootfs.after

2024-06-24T12:00:15Z INF Done executing stage 'rootfs.after'

Any ideas?

Same as #1928?

after #1599 (comment)

edit

moved following context to #2689:

journalctl -xefu k3s

Jun 28 23:48:46 node-server-56d1 k3s[5691]: time="2024-06-28T23:48:46Z" level=fatal msg="failed to detect selinux: read /proc/thread-self/attr/current: invalid argument"
Jun 28 23:48:46 node-server-56d1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

I could open an additional issue if you like. -> done

The text was updated successfully, but these errors were encountered:

Itxaka · 2024-07-02T13:22:19Z

Yes, confirmed, your persistent is 55Mb only, but the partition is 228Gb. That means that the growpart worked but the resize2fs didnt and it left it in a broken state.

You can probably workaround this by running resize2fs /dev/sda5 from the recovery entry. That should expand the filesystem to the max size and reboot, no problems after that.

kaiehrhardt · 2024-07-02T13:59:34Z

You can probably workaround this by running resize2fs /dev/sda5 from the recovery entry. That should expand the filesystem to the max size and reboot, no problems after that.

I know and i already did it as mentioned in

after #1599 (comment)

But, i think an persistent solution via bootstrap process would be nice.

no problems after that.

Not rlly. Just the next issue as mentioned in

journalctl -xefu k3s

Jun 28 23:48:46 node-server-56d1 k3s[5691]: time="2024-06-28T23:48:46Z" level=fatal msg="failed to detect selinux: read /proc/thread-self/attr/current: invalid argument"
Jun 28 23:48:46 node-server-56d1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

But since it is a different problem, i could move it to a new issue. Any thoughts?

Itxaka · 2024-07-02T14:01:33Z

You can probably workaround this by running resize2fs /dev/sda5 from the recovery entry. That should expand the filesystem to the max size and reboot, no problems after that.

I know and i already did it as mentioned in

after #1599 (comment)

But, i think an persistent solution via bootstrap process would be nice.

no problems after that.

Not rlly. Just the next issue as mentioned in
journalctl -xefu k3s
Jun 28 23:48:46 node-server-56d1 k3s[5691]: time="2024-06-28T23:48:46Z" level=fatal msg="failed to detect selinux: read /proc/thread-self/attr/current: invalid argument"
Jun 28 23:48:46 node-server-56d1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
But since it is a different problem, i could move it to a new issue. Any thoughts?

Sounds good. selinux should work out of the box in leap IIRC, unless it now requires some packages like ubuntu does with the new 15.6 release, so it makes sense to have a card to track that. Thanks!

Itxaka · 2024-07-02T14:03:00Z

Just found that cryptsetup is only installled on amd64 in our dockerfiles, which causes dracut issues. just happen that the required binaries for the persistent expansion are just after the cryptsetup binary, so maybe dracut is reaching cryptsetup, hitting the error and not installing the needed binaries in the initramfs???

Testing this...

kaiehrhardt · 2024-07-04T21:54:56Z

Seems like the issue is still there.

NAME="openSUSE Leap"
VERSION="15.6"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.6"
PRETTY_NAME="openSUSE Leap 15.6"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.6"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_VERSION="v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_ID_LIKE="kairos-standard-opensuse-leap-15.6"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2-k3s1"
KAIROS_ARTIFACT="kairos-opensuse-leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2+k3s1"
KAIROS_FLAVOR_RELEASE="leap-15.6"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_FAMILY="opensuse"
KAIROS_SOFTWARE_VERSION="v1.30.2+k3s1"
KAIROS_IMAGE_LABEL="leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2-k3s1"
KAIROS_FLAVOR="opensuse"
KAIROS_MODEL="rpi4"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_ID="kairos"
KAIROS_NAME="kairos-standard-opensuse-leap-15.6"
KAIROS_VERSION_ID="v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-leap-15.6 v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_RELEASE="v3.1.0-rc2"

Filesystem                         Size  Used Avail Use% Mounted on
/dev/loop0                         2.0G  1.4G  453M  76% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   55M   16M   36M  31% /usr/local
overlay                            2.0G   13M  2.0G   1% /var
overlay                            2.0G   13M  2.0G   1% /etc
overlay                            2.0G   13M  2.0G   1% /srv
devtmpfs                           4.0M     0  4.0M   0% /dev
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              1.6G  8.9M  1.6G   1% /run
tmpfs                              2.0G   13M  2.0G   1% /run/overlay
tmpfs                              3.9G  4.0K  3.9G   1% /tmp
/dev/sda2                          5.9G  2.8G  2.9G  50% /run/initramfs/cos-state
tmpfs                              784M  4.0K  784M   1% /run/user/1000

2024-06-24T12:00:15Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z WRN Not enough unpartitioned space in disk to operate
2024-06-24T12:00:15Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )
2024-06-24T12:00:15Z WRN Not enough unpartitioned space in disk to operate
2024-06-24T12:00:15Z INF Done executing stage 'rootfs.after'

mudler · 2024-07-09T08:24:59Z

thanks for confirmation @kaiehrhardt

Itxaka · 2024-07-11T17:12:43Z

I dont get it, I just retested this and did not have any issues.

Using:

NAME="openSUSE Leap"
VERSION="15.6"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.6"
PRETTY_NAME="openSUSE Leap 15.6"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.6"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_VERSION="v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_ID_LIKE="kairos-standard-opensuse-leap-15.6"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2-k3s1"
KAIROS_ARTIFACT="kairos-opensuse-leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2+k3s1"
KAIROS_FLAVOR_RELEASE="leap-15.6"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_FAMILY="opensuse"
KAIROS_SOFTWARE_VERSION="v1.30.2+k3s1"
KAIROS_IMAGE_LABEL="leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2-k3s1"
KAIROS_FLAVOR="opensuse"
KAIROS_MODEL="rpi4"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_ID="kairos"
KAIROS_NAME="kairos-standard-opensuse-leap-15.6"
KAIROS_VERSION_ID="v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-leap-15.6 v3.1.0-rc2-v1.30.2-k3s1"
KAIROS_RELEASE="v3.1.0-rc2"

Directly from artifacts released, so:

$ docker run -ti --rm -v $PWD:/image quay.io/luet/base util unpack quay.io/kairos/opensuse:leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2-k3s1-img /image
$ xzcat build/kairos-opensuse-leap-15.6-standard-arm64-rpi4-v3.1.0-rc2-k3sv1.30.2+k3s1.img.xz | sudo dd of=/dev/sda oflag=sync status=progress bs=10MB

Using usb as boot media, after flashing plenty of space left in the device:

First boot, it expands and fills the partition with no issues (note, the grow partition step runs twice for some reason, but first pass runs with no issues and second it says that there is not enough space (because it already expanded):

2024-06-24T12:00:13Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )                                                                                                                                              
2024-06-24T12:00:15Z INF Extending last partition to max spacesing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )
2024-06-24T12:01:53Z INF Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... ) ( commands: 0, files: 0, ... )
2024-06-24T12:01:53Z WRN Not enough unpartitioned space in disk to operate

Result is a full grown persistent:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0     2G  1 loop /
sda      8:0    0 115.1G  0 disk 
├─sda1   8:1    0    96M  0 part 
├─sda2   8:2    0   6.1G  0 part /run/initramfs/cos-state
├─sda3   8:3    0   4.1G  0 part 
├─sda4   8:4    0    64M  0 part /oem
└─sda5   8:5    0 104.8G  0 part /run/cilium

Filesystem                         Size  Used Avail Use% Mounted on
/dev/loop0                         2.0G  1.4G  453M  76% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   99G   30M   93G   1% /usr/local
overlay                            2.0G   13M  2.0G   1% /var
overlay                            2.0G   13M  2.0G   1% /etc
overlay                            2.0G   13M  2.0G   1% /srv
devtmpfs                           4.0M     0  4.0M   0% /dev
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              1.6G  8.9M  1.6G   1% /run
tmpfs                              2.0G   13M  2.0G   1% /run/overlay
tmpfs                              3.9G  4.0K  3.9G   1% /tmp
/dev/sda2                          5.9G  2.8G  2.9G  50% /run/initramfs/cos-state
tmpfs                              784M  4.0K  784M   1% /run/user/1000

mudler · 2024-07-15T08:44:09Z

@mauromorales can you confirm that this works as expected with the latest version?

@kaiehrhardt can you also share the output of blkid and lsblk?

mauromorales · 2024-07-15T12:37:26Z

It seems to be working correctly with the latest release. I'm using an SD card to install, and I can successfully see the different pods running on k9s, and no fatal issues on k3s logs:

localhost:/home/kairos # cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.6"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.6"
PRETTY_NAME="openSUSE Leap 15.6"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.6"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
KAIROS_FLAVOR="opensuse"
KAIROS_FLAVOR_RELEASE="leap-15.6"
KAIROS_FAMILY="opensuse"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-leap-15.6 v3.1.0-v1.30.2-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:leap-15.6-standard-arm64-rpi4-v3.1.0-k3sv1.30.2-k3s1"
KAIROS_ARTIFACT="kairos-opensuse-leap-15.6-standard-arm64-rpi4-v3.1.0-k3sv1.30.2+k3s1"
KAIROS_TARGETARCH="arm64"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_NAME="kairos-standard-opensuse-leap-15.6"
KAIROS_ID_LIKE="kairos-standard-opensuse-leap-15.6"
KAIROS_VARIANT="standard"
KAIROS_MODEL="rpi4"
KAIROS_RELEASE="v3.1.0"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION="v1.30.2+k3s1"
KAIROS_ID="kairos"
KAIROS_IMAGE_LABEL="leap-15.6-standard-arm64-rpi4-v3.1.0-k3sv1.30.2-k3s1"
KAIROS_VERSION="v3.1.0-v1.30.2-k3s1"
KAIROS_VERSION_ID="v3.1.0-v1.30.2-k3s1"

localhost:/home/kairos # df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/loop0                         2.0G  1.4G  460M  76% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   19G  980M   17G   6% /usr/local
overlay                            2.0G   13M  2.0G   1% /var
overlay                            2.0G   13M  2.0G   1% /etc
overlay                            2.0G   13M  2.0G   1% /srv
devtmpfs                           4.0M     0  4.0M   0% /dev
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              1.6G  9.9M  1.6G   1% /run
tmpfs                              2.0G   13M  2.0G   1% /run/overlay
tmpfs                              3.9G  4.0K  3.9G   1% /tmp
/dev/mmcblk0p2                     5.9G  2.8G  2.9G  50% /run/initramfs/cos-state
tmpfs                              7.7G   12K  7.7G   1% /var/lib/kubelet/pods/4754bc9a-eee1-40d4-88a0-ac2153cb27bb/volumes/kubernetes.io~projected/kube-api-access-m5pkv
tmpfs                              7.7G   12K  7.7G   1% /var/lib/kubelet/pods/11090b1f-2a3d-4172-9c9f-aae7ab82e94c/volumes/kubernetes.io~projected/kube-api-access-krwdz
tmpfs                              170M   12K  170M   1% /var/lib/kubelet/pods/eb789508-487b-4362-b19f-a77d9566d788/volumes/kubernetes.io~projected/kube-api-access-qzwzf
shm                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/3f9c3c005931b3d4865c296007203a7228622b3902dfa2b194923e60d85293c5/shm
shm                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/5cf368a67de32fc549a58a995924dbcc3e0c153fc1b4f218d676ef08e19fd940/shm
shm                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/bc152e557fdb13ab816f8c7889407f60fa0f039313b645a3a45df8a13f17c19f/shm
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3f9c3c005931b3d4865c296007203a7228622b3902dfa2b194923e60d85293c5/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5cf368a67de32fc549a58a995924dbcc3e0c153fc1b4f218d676ef08e19fd940/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/bc152e557fdb13ab816f8c7889407f60fa0f039313b645a3a45df8a13f17c19f/rootfs
tmpfs                              784M  4.0K  784M   1% /run/user/1000
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/02c643df91855c6ee451a3760114983f6e1c86415de36571b6ecb5a677f27a9f/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/87f2744855f7dc122549f2a256aed90560a10c3364e1cd5cdc628f6522964e2a/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/191d686fda56a8ab6084a7fdcb7144eb7220e2bfaf0ff60a7986783cc4ceedc2/rootfs
tmpfs                              7.7G   12K  7.7G   1% /var/lib/kubelet/pods/447c340f-8c2e-4480-a93e-7f5310f28a57/volumes/kubernetes.io~projected/kube-api-access-z6btd
shm                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c98870240001f723a1f49f5550dbeb979f9f0032ebb8ffca82595a63920408a2/shm
shm                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/718ad1fc6ea898567bc37268ee7e376587e67991a2e26f4ee3e121678c87ae45/shm
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c98870240001f723a1f49f5550dbeb979f9f0032ebb8ffca82595a63920408a2/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/718ad1fc6ea898567bc37268ee7e376587e67991a2e26f4ee3e121678c87ae45/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e379edaa7efc84296f2dd7a8bd1e4e4fa0e44af473a9bb18d8f7976576aa0132/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7cc75efbd2408f1a25e8c9cc0a80bab9d077d87c4827ffe35b97025c5842b89f/rootfs
overlay                             19G  980M   17G   6% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4a092aeb8af45b2f72230c8ce97b8216bef51495833f1a4c653de1af5b1adbbc/rootfs

localhost:/home/kairos # lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0    2G  1 loop /
mmcblk0     179:0    0 29.7G  0 disk
|-mmcblk0p1 179:1    0   96M  0 part
|-mmcblk0p2 179:2    0  6.1G  0 part /run/initramfs/cos-state
|-mmcblk0p3 179:3    0  4.1G  0 part
|-mmcblk0p4 179:4    0   64M  0 part /oem
`-mmcblk0p5 179:5    0 19.4G  0 part /run/cilium
                                     /usr/local/.state/usr-share-pki-trust.bind/anchors
                                     /usr/share/pki/trust/anchors
                                     /usr/share/pki/trust
                                     /var/lib/wicked
                                     /var/lib/tailscale
                                     /var/lib/snapd
                                     /var/lib/rook
                                     /var/lib/rancher
                                     /var/lib/osd
                                     /var/lib/longhorn
                                     /var/lib/kubelet
                                     /var/lib/extensions
                                     /var/lib/dbus
                                     /var/lib/containerd
                                     /var/lib/cni
                                     /var/lib/calico
                                     /var/lib/ca-certificates
                                     /etc/zfs
                                     /etc/systemd
                                     /etc/sysconfig
                                     /etc/ssh
                                     /var/snap
                                     /etc/runlevels
                                     /etc/rancher
                                     /etc/pwx
                                     /etc/modprobe.d
                                     /var/log
                                     /usr/libexec
                                     /etc/kubernetes
                                     /etc/iscsi
                                     /etc/init.d
                                     /var/cores
                                     /etc/cni
                                     /root
                                     /opt
                                     /home
                                     /usr/local

localhost:/home/kairos # systemctl status k3s
* k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/k3s.service.d
             `-override.conf
     Active: active (running) since Mon 2024-07-15 12:31:41 UTC; 2min 14s ago
       Docs: https://k3s.io
    Process: 1653 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1687 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1688 (k3s-server)
      Tasks: 99
        CPU: 2min 13.745s
     CGroup: /system.slice/k3s.service
             |-1688 "/usr/bin/k3s server"
             |-1721 "containerd "
             |-2303 /var/lib/rancher/k3s/data/04062516f863dba6fbbbb251ae40e2cc82756b587e30fc88e9659564ec85a68f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3f9c3c005931b3d4865c296007203a7228622b3902dfa2b1>
             |-2382 /var/lib/rancher/k3s/data/04062516f863dba6fbbbb251ae40e2cc82756b587e30fc88e9659564ec85a68f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5cf368a67de32fc549a58a995924dbcc3e0c153fc1b4f218>
             |-2398 /var/lib/rancher/k3s/data/04062516f863dba6fbbbb251ae40e2cc82756b587e30fc88e9659564ec85a68f/bin/containerd-shim-runc-v2 -namespace k8s.io -id bc152e557fdb13ab816f8c7889407f60fa0f039313b645a3>
             |-3508 /var/lib/rancher/k3s/data/04062516f863dba6fbbbb251ae40e2cc82756b587e30fc88e9659564ec85a68f/bin/containerd-shim-runc-v2 -namespace k8s.io -id c98870240001f723a1f49f5550dbeb979f9f0032ebb8ffca>
             `-3597 /var/lib/rancher/k3s/data/04062516f863dba6fbbbb251ae40e2cc82756b587e30fc88e9659564ec85a68f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 718ad1fc6ea898567bc37268ee7e376587e67991a2e26f4e>

Jul 15 12:32:58 localhost k3s[1688]: I0715 12:32:58.356830    1688 shared_informer.go:313] Waiting for caches to sync for resource quota
Jul 15 12:32:58 localhost k3s[1688]: I0715 12:32:58.670518    1688 shared_informer.go:320] Caches are synced for resource quota
Jul 15 12:32:58 localhost k3s[1688]: I0715 12:32:58.849246    1688 shared_informer.go:313] Waiting for caches to sync for garbage collector
Jul 15 12:32:58 localhost k3s[1688]: I0715 12:32:58.849378    1688 shared_informer.go:320] Caches are synced for garbage collector

I will close the issue for now, but you are welcome to re-open @kaiehrhardt if there's something that I missed testing here

kaiehrhardt added bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed labels Jun 28, 2024

github-project-automation bot added this to 🧙Issue tracking board Jun 28, 2024

jimmykarily moved this to In Progress 🏃 in 🧙Issue tracking board Jul 1, 2024

jimmykarily removed the triage Add this label to issues that should be triaged and prioretized in the next planning call label Jul 1, 2024

jimmykarily moved this from In Progress 🏃 to Todo 🖊 in 🧙Issue tracking board Jul 1, 2024

kaiehrhardt mentioned this issue Jul 2, 2024

failed to detect selinux: read /proc/thread-self/attr/current: invalid argument #2689

Closed

Itxaka mentioned this issue Jul 3, 2024

🐛 Install cryptsetup for all arches in opensuse #2691

Merged

Itxaka closed this as completed in #2691 Jul 3, 2024

github-project-automation bot moved this from Todo 🖊 to Done ✅ in 🧙Issue tracking board Jul 3, 2024

Itxaka self-assigned this Jul 3, 2024

Itxaka removed the unconfirmed label Jul 3, 2024

mudler mentioned this issue Jul 9, 2024

Dracut immucore should fatal if binaries are missing #2692

Closed

mudler reopened this Jul 9, 2024

github-project-automation bot moved this from Done ✅ to Under review 🔍 in 🧙Issue tracking board Jul 9, 2024

mudler unassigned Itxaka Jul 9, 2024

mudler moved this from Under review 🔍 to Todo 🖊 in 🧙Issue tracking board Jul 9, 2024

mudler mentioned this issue Jul 9, 2024

📣 Kairos release v3.1.1 #2706

Closed

34 tasks

mauromorales self-assigned this Jul 15, 2024

mauromorales moved this from Todo 🖊 to In Progress 🏃 in 🧙Issue tracking board Jul 15, 2024

mauromorales closed this as completed Jul 15, 2024

github-project-automation bot moved this from In Progress 🏃 to Done ✅ in 🧙Issue tracking board Jul 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k3s no space left on device #2682

k3s no space left on device #2682

kaiehrhardt commented Jun 28, 2024 •

edited

Loading

Itxaka commented Jul 2, 2024

kaiehrhardt commented Jul 2, 2024

Itxaka commented Jul 2, 2024

Itxaka commented Jul 2, 2024

kaiehrhardt commented Jul 4, 2024

mudler commented Jul 9, 2024

Itxaka commented Jul 11, 2024

mudler commented Jul 15, 2024

mauromorales commented Jul 15, 2024

k3s no space left on device #2682

k3s no space left on device #2682

Comments

kaiehrhardt commented Jun 28, 2024 • edited Loading

Itxaka commented Jul 2, 2024

kaiehrhardt commented Jul 2, 2024

Itxaka commented Jul 2, 2024

Itxaka commented Jul 2, 2024

kaiehrhardt commented Jul 4, 2024

mudler commented Jul 9, 2024

Itxaka commented Jul 11, 2024

mudler commented Jul 15, 2024

mauromorales commented Jul 15, 2024

kaiehrhardt commented Jun 28, 2024 •

edited

Loading