Add velero annotations to all necessary deployments

[billimek]

See #15 for some background.

Deployments to annotate with velero restic backup stuff:

• home-assistant
• unifi
• plex
• node-red
• hass-mysql
• mc-minecraft
• mcsv-minecraft
• nzbget
• radarr
• rtorrent-flood
• sonarr
• grafana
• influxdb
• nextcloud
• loki
• alertmanager
• chronograf
• prometheus

[billimek]

• PR 15473 created to add annotation support for home-assistant chart
• PR 64 created to add annotation support for kube-plex chart

[echel0n]

One thing I've noticed is descheduler causes issues with Velero during backups if it re-schedules a pod during the backup phase, Velero offers a plugin system but its pre-backup and doesn't seem to do post-backup so adding a critical annotation to the pod doesn't seem plausable, got any thoughts on this ?

[billimek]

After lots of debugging, it becomes clear that is it not possible to leverage velero/restic to backup volumes from annotated pods which are running on non-amd64 (i.e. arm) nodes.

The reason for this is that the 'restic' daemonset needs to schedule pods on arm-based nodes in order for restic backup/restores to work properly.

There are currently no multiarch velero images available. This issue and this PR suggest that it may soon? be possible but not today.

This is sad :(

[billimek]

Annotations are all created where possible.

Also successfully tested velero backups against these annotations. Closing this issue.

[onedr0p]

@billimek should the podAnnotation label value for velero contain the name of the actual PVC instead of the helm variable?

For example radarr-config instead of config:

  values:
    image:
      repository: linuxserver/radarr
...
    podAnnotations:
      backup.velero.io/backup-volumes: radarr-config

[billimek]

I think it's supposed to be the name of the volume in the deployment/pod. In the case of radarr, this is config. This seems to correspond with the example in the velero restic documentation.

I see what you mean about using the name of the actual PVC and not the volume name. Indeed when I look in the velero restic backup logs, I see,

time="2020-01-18T06:00:32Z" level=info msg="Executing custom action" backup=velero/velero-daily-backup-20200118060005 group=v1 logSource="pkg/backup/item_backupper.go:330" name=radarr-bbb8f7dd7-ndfmr namespace=default resource=pods
time="2020-01-18T06:00:32Z" level=info msg="Executing podAction" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Adding pvc radarr-config to additionalItems" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Adding pvc nfs-media-downloads-pvc to additionalItems" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Adding pvc nfs-media-pvc to additionalItems" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Done executing podAction" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Backing up item" backup=velero/velero-daily-backup-20200118060005 group=v1 logSource="pkg/backup/item_backupper.go:169" name=radarr-config namespace=default resource=persistentvolumeclaims
time="2020-01-18T06:00:32Z" level=info msg="Executing custom action" backup=velero/velero-daily-backup-20200118060005 group=v1 logSource="pkg/backup/item_backupper.go:330" name=radarr-config namespace=default resource=persistentvolumeclaims
time="2020-01-18T06:00:32Z" level=info msg="Executing PVCAction" backup=velero/velero-daily-backup-20200118060005 cmd=/velero logSource="pkg/backup/backup_pv_action.go:49" pluginName=velero
time="2020-01-18T06:00:32Z" level=info msg="Backing up item" backup=velero/velero-daily-backup-20200118060005 group=v1 logSource="pkg/backup/item_backupper.go:169" name=pvc-a187d87b-d196-4cea-a6db-32feacc28f1c namespace= resource=persistentvolumes
time="2020-01-18T06:00:32Z" level=info msg="Executing takePVSnapshot" backup=velero/velero-daily-backup-20200118060005 group=v1 logSource="pkg/backup/item_backupper.go:395" name=pvc-a187d87b-d196-4cea-a6db-32feacc28f1c namespace= resource=persistentvolumes

Maybe it is 'looking up' the PVC name based on the deployment volume definition?

I certainly see the data being backed up when I look in the restic snapshot,

'2020-01-18T06:00:32Z':
total 12343
dr-xr-xr-x   7 root root       0 Jan 18 06:00 .
dr-xr-xr-x   1 root root       0 Jan 19 00:47 ..
drwxr-xr-x   3 1001 1001       0 Feb 11  2017 Backups
-rw-r--r--   1 1001 1001     423 Jan 18 05:13 config.xml
drwxr-xr-x   2 1001 1001       0 Jan 17 22:51 logs
-rw-r--r--   1 1001 1001 3969024 Jan 18 05:57 logs.db
drwx------   2 1001 1001       0 Oct 11 18:51 lost+found
drwxr-xr-x 503 1001 1001       0 Jan  4 23:47 MediaCover
-rw-r--r--   1 1001 1001 2859008 Jan 18 05:55 nzbdrone.db
-rw-r--r--   1 1001 1001 3332096 Nov  5  2018 nzbdrone.db.old
-rw-r--r--   1 1001 1001   32768 Jan 18 06:00 nzbdrone.db-shm
-rw-r--r--   1 1001 1001  173072 Jan 18 06:00 nzbdrone.db-wal
-rw-r--r--   1 1001 1001 2269184 Jan 20  2019 nzbdrone-new.db
-rw-r--r--   1 1001 1001       5 Jan 18 05:13 nzbdrone.pid
drwxr-xr-x   3 1001 1001       0 Feb  4  2017 xdg

[onedr0p]

You're right, it seems to be smarter than I am. I thought I read it was the PVC name 🤦‍♂

[onedr0p]

Also, backing up the nzbget config is something to add. It only stores the nzbget.conf