[Question] Suitable choice for Raspberry Pi cluster?

[mortenlj]

I currently have a cluster of four Raspberry Pis (3Bs and 3B+es), running k3s on Rancher OS. My choice of Rancher OS was based on the thought that it allowed for simple OS updates while being specially designed for running docker containers. As it came time to update the OS, I realised that updates are not supported for RPI, and I have to flash the SD card again with the next version.

These RPIs are in a somewhat inaccessible location, with no screen or keyboard, so updates need to be possible to do remote, and it needs to be safe so that it either works, or fails back to previous state.

Since I want to run k3s, k3os seems like a good choice, but looking at the issues here, I see mentions of ARM not being officially supported, I see mentions of update not doing kernel updates, and getting it installed on an SD card seems not straight forward.

Is k3os a good choice for RPI, or should I look at something like Balena OS instead?

[mak3r]

BalenaOS is a docker based solution, not kubernetes. I'm not sure if that matters to your use case. k3os does setup containerd as PID 1 and it runs k3s (kubernetes) out of the box. There is currently work being done on the upgrade controller k3s-io/k3s#1498 which will allow you to do privileged operations on your k3s clusters remotely. There is a demo of the feature here https://youtu.be/UsPV8cZX8BY?t=872

[mortenlj]

I believe it would be possible to run k3s on Balena OS, much the same way that I've run it on Rancher OS (which is also docker based).

The reason I'm asking is because I see indications that ARM and/or Raspberry pi is just an afterthought, and not officially part of the product. It look that way, because important features seem to be lagging. I'm wondering if that's correct?

I'm about to invest time and energy into migrating to something new (since my current setup never landed the needed features), and I would like to avoid having to do it a second time a couple months down the road.

On paper, k3os looks like the better choice, but that is assuming Raspberry pi is a fully supported target with all the bells and whistles.

The upgrade controller is great, except the docs currently state that a vital feature is not supported on Raspberry pi. This question is basically asking if there are plans to fix that.

[dweomer]

If you need the fallback-to-last-known-good-kernel then balenaOS might be a better choice. That said, I would love to review a PR for a contribution that got this working!

[mortenlj]

Falling back would be a nice to have feature, but I'll be happy with an upgrade controller that makes a safe upgrade of the kernel (for some value of "safe"). What I want to avoid is having to manually update the kernel, or even worse, having to flash a new SD card to get a new kernel.

The problem with a PR, is that I haven't got a clue where to even start. 😄

[mortenlj]

I see that we kind of got sidetracked by the falling back to last known issue. My main concern, and the major point I was looking for a definitive answer for is this:

Is Raspberry Pi considered a fully supported target, or is it more of a best effort, side project that gets some love whenever there's nothing else to do?

[tomaszkiewicz]

Regarding using k3os on Raspberry I found https://github.com/sgielen/picl-k3os-image-generator to be quite easy to use to generate an image that you can dd to sd card and run.
I've tested it so far with a couple of RPI4 and it looks good :)