Backup/Restore does not include the secrets used for clientSecretName and helmSecretName in fleet gitrepos

[Martin-Weiss]

SURE-3503
SURE-3531 - closed as duplicate

After a rancher restore in Rancher 2.5.9 the gitrepos defined in fleet do not work anymore as we have to use authentication against git and against the helm repo (harbor in our case) and the secrets defined in clientSecretName and helmSecretName of the gitrepos defined are not backed up / restored.

Could we get the backup operator fixed in a way that it automatically creates a backup of these secrets in the fleet-default / fleet-local namespaces, too? (maybe also in other fleet namespaces if additional ones have been created)

[SheilaghM]

This may be a Feature Request. We will discuss with PM and respond back.

[Martin-Weiss]

This may be a Feature Request. We will discuss with PM and respond back.

Thanks! Please keep in mind that loosing the secrets is somewhat similar to data-loss as customers might have created the secrets manually or in the fleet UI and it is not documented that these must be backed up separate/manually...

[MKlimuszka]

@nickwsuse , Eli believes this is already fixed.

[prachidamle]

Could we retest this usecase with latest Rancher version and latest Backup-restore operator to see if the issue is still seen? There is a high chance that latest versions might have this resolved already

[nickwsuse]

@eliyamlevy @prachidamle I'm not seeing the secrets I made before taking the backup after I deleted them and restored. The secrets I made are opaque, so not sure if there's an expected secret type for this to work, but I would assume it should keep all secrets in the namespaces regardless of type?

Steps:

1. Created a v2.7-head instance (Commit ID: 65089ee)
2. Created a downstream RKE1 and RKE2 cluster
3. Added an opaque secret with my S3 creds
4. Installed Rancher Backups v3.0.1-rc2 with my AWS S3 bucket info
5. Created a backup called "pre-fleet-secrets"
6. Created an opaque secret in the fleet-default namespace
7. Created an opaque secret in the fleet-local namespace
8. Created another backup called "post-fleet-secrets"
9. Deleted the secrets I made in both fleet namespaces
10. Created a restore using the "post-fleet-secrets" backup
11. Checked for the secrets I made that should have been restored

This test was a local restore, but I imagine we'd get the same results with a restore done in a migration as well.

[MKlimuszka]

@eliyamlevy , we checked with Ron and he said it would be ok to add this to the Q1 release.

[eliyamlevy]

Hello all,
The solution we came to was to create a resourceSelector which will look for any secret with a specific label to back it up. This selector will look in all namespaces and at every secret on the cluster. If the secret has the label resources.cattle.io/backup: true it will be backed up.

The reason for this approach is because these gitrepo secrets can be in any namespace and can have any name, which makes it difficult to identify them through the resource selectors unless they have some kind of standard identifier. This label will serve as that identifier to allow anyone to make sure any secret gets backed up.

The only caveat is to make sure to encrypt the backup if you are storing sensitive information, since the backup is in plain text unless encrypted. Encryption is already available through the Backup Restore Operator.

The way I tested this was to create many secrets with different names in different namespaces and added the resources.cattle.io/backup: true label to them. Then I took a backup, downloaded the backup tarball and checked if the secrets were there. Nick's steps are the perfect way to test as well, just have to add the label to the secret before backing up.

[nickwsuse]

Verified on v2.7-head ID: 6316baf and Rancher Backups v3.0.1-rc3

I repeated the steps I mentioned in my earlier comment, this time adding the resources.cattle.io/backup label as noted in Eli's comment, and the secrets that I added that label to came back when restored. The secrets that didn't have the new label were still gone as expected.

I did both an encrypted and unencrypted backup/restore just to be sure both still work.