Elemental CAPI components shipping

[anmazzotti]

This issue serves to design and implement a way for the Elemental CAPI provider to ship the correct k8s components to hosts.

As a context, with CAPI we can have several bootstrap providers generating cloud-init configs to bootstrap nodes.

Note however that the bootstrap providers will not issue any instructions on how to download and install needed binaries, or which version of those binaries should be installed, or how to restart specific services after the cloud-init config is applied.

For example, when using the Kubeadm provider, the kubeadm binary and all other dependencies is expected to be already installed on the host machines.

This implies that if we want to support multiple providers, for example kubeadm, rke2, and k3s, we will need to have all these dependencies included in the Elemental image, or... we can find an alternative approach.

An alternative approach would be to publish bundles with the needed dependencies as an additional layer to the OS image.
The end user can potentially configure (and upgrade?) these bundles by defining them at the ElementalMachineTemplate level so that it should be propagated to the underlying ElementalMachines. For example:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ElementalMachineTemplate
metadata:
  name: elemental-cluster-md-0
  namespace: default
spec:
  template:
    spec:
      componentsBundle: registry.suse.com/rancher/elemental/bundles/rke2:v1.26.7+rke2r1

This will have the benefit that components can be customized and are also decoupled from the base Elemental image, so that it will be possible to update them independently.
The drawback is to maintain these bundles and make sure we are up to date with major versions of components and security vulnerabilities.

Open questions:

• How should bundling work? Should an installation script be included in the image? How should the agent use/unpack/install the bundle?
• How will be possible to create custom bundles to be easily used in the same way?
• How will bundles support different distributions? What if my base Elemental image is built on top of RHEL or Debian for example?
• How will bundle upgrade work within the context of CAPI upgrades?

[anmazzotti]

Linking parent issue #968

[davidcassany]

I have been wondering about some sort of bundles approach in a similar way of how we do it with the seedImage.

Imagine the upstream cluster serving the binaries to the downstream cluster, then the bundle could be essentially an image that exposes a tarball and some sort of free form environment variables to the downstream cluster. Such a tarball would have to match a simple contract such as having an install.sh script in its root so the agent only has to call ./install.sh with the provided environment variables.
Exposing such a tarball in upstream cluster could be done in a pod running a given image, such an image (call it bundle or whatever) could then include preloaded binaries with a simple script to place them where ever required or some more sophisticated script (in the spirit of Rancher provisioning) to actually fetch them dynamically from public locations and install them.

In fact, something like that could be even implemented and exercised within the current elemental-operator I believe. This could be a way to decouple the provisioning logic from the controller itself. In terms of elemental-operator that means providing the provisioning plan (or the meat of it ifyou want) from a pod, so we could actually switch to a different plan by simply providing another provisioning image. Current elemental-operator provisioning plan is hardcoded and it basically runs Rancher provisioning.

It might look like a convoluted approach and implies some controller logic but this also allows plenty of flexibility. We could sign and verify signature of supported bundle or provisioning images to ensure the supplay chain.

[anmazzotti]

This is most likely not needed.
So far both the k3s and rke2 providers do download the needed binaries themselves.

For rke2 containerd may be needed already installed, but it would be easy to just include it in the base Elemental image.

The new OS Plugin system can also be exploited now to install custom components, on top of the cloud-init config that we already support.

This looks plenty of options to me, so I am going to close this issue.