UC Backups: "backup all namespaces" does not actually backup all namespaces

[judge-red]

What happened?

When creating a backup or backup schedule, it's possible to choose "all namespaces" (the default). However, KKP will then configure the backup to instead backup all namespaces that existed when the backup was created in the UI. In other words: the velero backup configuration will contain an explicit list of namespaces.

There are two problems with this:

1. Namespaces that are created later, will not be included in the backup.
2. Cluster scoped resources are not backed up. By default, Velero does include cluster scoped resources only if truly all namespaces (i.e. without filter) are backed up, and not include them if a list of namespaces is provided. (By default, because it's possible to overwrite this behavior to always/never include cluster scoped resources in Velero).

Expected behavior

If "backup all namespaces" is selected, all namespaces, including those created in the future (for a backup schedule) are backed up. And by implication, all cluster scoped resources are backed up as well.

How to reproduce the issue?

• Enable the UC Backup feature in the admin settings
• configure a backup location
• enable the UC backup for a UC
• create a backup or backup schedule where you enable the "backup all namespaces" checkbox
• download UC kubeconfig
• velero backup get <name> -o json | jq .spec.includedNamespaces

This should not be set all all (or possibly an empty list / nil would be acceptable).

After the backup was executed, you'll also find such log lines:

$ k -n velero logs -l app=velero --tail=1000 | grep cluster-scoped | head
Defaulted container "velero" out of: velero, velero-velero-plugin-for-aws (init)
time="2024-11-19T20:00:50Z" level=info msg="Skipping resource certificatesigningrequests.certificates.k8s.io, because it's cluster-scoped and only specific namespaces or namespace scope types are included in the backup." backup=velero/debug-test-schedule-20241119200049 logSource="pkg/util/collections/includes_excludes.go:155"
time="2024-11-19T20:00:50Z" level=info msg="Skipping resource ingressclasses.networking.k8s.io, because it's cluster-scoped and only specific namespaces or namespace scope types are included in the backup." backup=velero/debug-test-schedule-20241119200049 logSource="pkg/util/collections/includes_excludes.go:155"

How is your environment configured?

• KKP version: 2.26.0
• Shared or separate master/seed clusters?: shared

Provide your KKP manifest here (if applicable)

N/A

What cloud provider are you running on?

N/A

What operating system are you running in your user cluster?

N/A

Additional information

[csengerszabo]

/label sig/app-management
/label sig/api
/label sig/ui
/label customer-request

[rajaSahil]

Namespaces that are created later, will not be included in the backup.

Issue is with respect to api request payload, when we are doing an api call to create backup for all namespaces, we are sending specific namespaces present at that instant.

In the spec section, we have these namespaces included.

Included Namespaces:
    cloud-init-settings
    default
    kube-node-lease
    kube-public
    kube-system
    velero

A fix would be required in the api request payload for include namespaces:

includeNamespace: 
- *

[csengerszabo]

/assign @ahmadhamzh

[csengerszabo]

/transfer dashboard

[rajaSahil]

Cluster scoped resources are not backed up. By default, Velero does include cluster scoped resources only if truly all namespaces (i.e. without filter) are backed up, and not include them if a list of namespaces is provided. (By default, because it's possible to overwrite this behavior to always/never include cluster scoped resources in Velero).

To include cluster scoped resources, we need to pass this field in spec.

  includeClusterResources: true

Please take a look : https://velero.io/docs/main/api-types/backup/