Elemental CAPI upgrades

[anmazzotti]

This issue is about streamlining a workflow for CAPI resources upgrades using the Elemental infrastructure provider.

This is about upgrading the Kubernetes version on the Elemental Hosts, not about upgrades of the Elemental OS itself. This can be managed by other means and it's not strictly related to CAPI.

Documentation: https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html

CAPI requires to create new MachineTemplates to trigger a rolling upgrade that will delete the old Machines and create new ones from the new template. From Elemental side this workflow heavily depends on a smooth Elemental Reset workflow.

Take in consideration that a Kubernetes version upgrade may require an update to the shipped components, for example a newer version of kubeadm, kubelet, or rke2, or k3s.

Considering the current bundle proposal (#990), we can expect the end user to create a ElementalMachineTemplate containing an updated (and matching the desired kubernetes version) bundle definition, so it shouldn't be a problem.

Some open questions I have are:

• Can we exploit the Reset functionality for upgrades, or do we need to create a new and "softer" workflow? The only drawback of Reset is that it may involve formatting partitions, or even shredding data. It may be a pretty long operation and therefore it may be overkill for a Kubernetes version upgrade.
• What exactly is the rollout workflow? I would expect the "old" ElementalMachine to be deleted first. Reconciling the deletion we may decide to trigger a Reset on the underlying ElementalHost. Once the ElementalMachine is actually deleted, I expect a new one to be created using the newly referenced ElementalMachineTemplate, then the assumption is that the previous host would finish reset, register a new ElementalHost and that would be bound to the new ElementalMachine as usual. Still the entire procedure should be tested and proven.

[anmazzotti]

Linking parent epic: #968

[anmazzotti]

Since the bundle proposal (#990) is off due to the k3s and rke2 providers installing their own binaries, this issue could be closed as well.
A new issue can be opened to support CAPI in place upgrades.