feat: Configure namespace sync in helm chart

[dlipovetsky]

What problem does this PR solve?:

• Allows user to configure target namespace label key using flag. Sets the default label key as an empty string. Labels cannot have an empty key, which means that, by default, no namespaces are target namespaces.
• Adds configuration fields to helm values, and passes them to manager as flags. Defines caren.nutanix.com/namespace-sync as the default target namespace label key.
• Documents configuration fields and their default values.

#725 fixes behavior when the source namespace and/or target namespace label key are empty strings.

Which issue(s) this PR fixes:
Fixes #

How Has This Been Tested?:

Special notes for your reviewer:

I think this behavior deserves to be documented, but right now the only place where we talk about configuration of CAREN itself is in the Chart README. Should I add something there, or under docs/?

[jimmidyson]

@dlipovetsky wdyt about adding helm configuration for enabling the sync controller (default enabled) with required sync source namespace flag? Don't start up if misconfigured.

Also default sync source namespace to pod (helm release) namespace?

[dlipovetsky]

@dlipovetsky wdyt about adding helm configuration for enabling the sync controller (default enabled) with required sync source namespace flag? Don't start up if misconfigured.

Also default sync source namespace to pod (helm release) namespace?

Both ideas sound good.

I'll leave the source namespace flag default value an empty string.

I will start the controller only if both the source namespace and target namespace label key flags have values.

[dlipovetsky]

(Rebased on main)

[dlipovetsky]

e2e tests failing, will rebase on main once #736 merges

[dlipovetsky]

The CAREN process fails to start. We don't get Pod logs for failing CI jobs, so I had to run the e2e test locally.

cluster-api-runtime-extensions-nutanix-6bc6c9dcbd-p6ghd E0624 16:36:10.605469       1 main.go:153] "Namespace Sync is enabled, but source namespace and/or target namespace label key are not configured." logger="main"
Stream closed EOF for caren-system/cluster-api-runtime-extensions-nutanix-6bc6c9dcbd-p6ghd (webhook)

The Deployment looks correct:

    spec:
      containers:
      - args:
        - --webhook-cert-dir=/runtimehooks-certs/
        - --defaults-namespace=$(POD_NAMESPACE)
        - --namespacesync-enabled=true
        - --namespacesync-source-namespace=caren-system
        - --namespacesync-target-namespace-label-key=caren.nutanix.com/namespace-sync

Will find the bug. Update: Found it by stepping through startup sequence. I made a copy-paste typo and bound the flag to the wrong variable. Fixed!

[dlipovetsky]

lint-test-helm fails:

Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Warning  Failed       3m27s (x4 over 4m59s)  kubelet            Failed to pull image "ghcr.io/nutanix-cloud-native/caren-helm-reg:v0.11.0-dev": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/nutanix-cloud-native/caren-helm-reg:v0.11.0-dev": failed to resolve reference "ghcr.io/nutanix-cloud-native/caren-helm-reg:v0.11.0-dev": ghcr.io/nutanix-cloud-native/caren-helm-reg:v0.11.0-dev: not found

Error from server (BadRequest): container "helm-repository" in pod "helm-repository-67575f489c-xxx5n" is waiting to start: trying and failing to pull image
Error printing details: failed waiting for process: exit status 1

I think this is because of https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pull/745/files#diff-42e26dc67aed8aa3edb2472b4403288c1699fb6dc47419b9a475f0f224fe4689R44-R46

[dlipovetsky]

Going to rebase on #746