fix: Deterministic ordering of lifecycle hooks

[jimmidyson]

This commit introduces wrapper handlers to allow ordering
of lifecyce hooks to be called via a single registered hook.
This ensures that hooks are called in a defined order, fixing
issues that have arisen that require CNI hook to be applied
before metallb hook.

[dlipovetsky]

With one handler for a specific hook type running "sub-handlers" sequentially, we'll need to be careful to make sure the handler responds within the CAPI deadline of 30s.

We talked about running "sub-handlers" in parallel, and we can do that in follow-up PR.

[dkoshkin]

On main I see that timeout is 10seconds, am I looking at the correct place?

apiVersion: runtime.cluster.x-k8s.io/v1alpha1
kind: ExtensionConfig
metadata:
  annotations:
    runtime.cluster.x-k8s.io/inject-ca-from-secret: caren-system/cluster-api-runtime-extensions-nutanix-runtimehooks-tls
  creationTimestamp: "2024-08-08T22:21:32Z"
  generation: 2
  labels:
    cluster.x-k8s.io/provider: runtime-extension-caren
    clusterctl.cluster.x-k8s.io: ""
  name: cluster-api-runtime-extensions-nutanix
  resourceVersion: "4431"
  uid: 28e66cd5-b136-4f4b-b818-4c0037e6930f
status:
  - failurePolicy: Fail
    name: serviceloadbalancerhandler-acpi.cluster-api-runtime-extensions-nutanix
    requestHook:
      apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
      hook: AfterControlPlaneInitialized
    timeoutSeconds: 10
...

[dlipovetsky]

On main I see that timeout is 10seconds, am I looking at the correct place?

Good catch. Those 10s timeouts make sense for individual handlers, but we should increase it, perhaps to 30s, for the one aggregate handler.

[dlipovetsky]

On second thought, if the sub-handler fails, no other sub-handlers will run, so shortening the timeout isn't useful. But it will be necessary if/when we execute sub-handlers in parallel.

[dlipovetsky]

I'll reduce the MetalLB timeout to 10s, since the handler is already limited to 10s by CAPI's default. We'll want to raise the handler timeout to 30s, but in a way that fits with the library, and that'll need some thought.

[dkoshkin]

Verified in the logs that the handlers are called in the expected order and the HCPs are also created in the same order:
(Notice the metallb is last)

$ kubectl get hcp -w 
NAME                                             READY   REASON
cilium-cni-installation-dkoshkin-hook-ordering           
nutanix-ccm-dkoshkin-hook-ordering                       
node-feature-discovery-dkoshkin-hook-ordering                              
cluster-autoscaler-dkoshkin-hook-ordering                
nutanix-csi-dkoshkin-hook-ordering                       
snapshot-controller-dkoshkin-hook-ordering               
metallb-dkoshkin-hook-ordering