Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AutoRotate #212

Open
artem-bm opened this issue Sep 9, 2024 · 7 comments
Open

AutoRotate #212

artem-bm opened this issue Sep 9, 2024 · 7 comments
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@artem-bm
Copy link

artem-bm commented Sep 9, 2024

Hello,

The auto mode lacks the function of the maximum number of incremental backups.
In the number of days or copies created.

@artem-bm artem-bm added the enhancement New feature or request label Sep 9, 2024
@abbbi
Copy link
Owner

abbbi commented Sep 9, 2024

i dont think i will work on this, you can easily work around that by scripting via bash or alike.
See also:

https://github.com/abbbi/virtnbdbackup/blob/master/scripts/vmbackup.sh
or
https://github.com/slackdaystudio/vm-snapshot

@abbbi abbbi added the wontfix This will not be worked on label Sep 9, 2024
@sentry0
Copy link

sentry0 commented Oct 27, 2024

Hey, I'm the author of those scripts. I caution you against using the prune option in those scripts.

The prune option leaves the backups inconsistent because the script doesn't manage dirty bitmaps properly. I think it's fixable, it's just a pain to do.

I'm re-writing the whole thing to be a node script this weekend. I plan on making a more robust backup system that includes managing bitmaps. I may even look for a node file rotation system for better backup rotations.

@abbbi
Copy link
Owner

abbbi commented Oct 27, 2024

Should i remove the Script from the repo?

@sentry0
Copy link

sentry0 commented Oct 27, 2024

If you don't mind, you may want to pull it. The rotation looks broken, and I would not want people to rely on it.

Sorry for the trouble @abbbi My node script is working, I need to drop in the logic for rotations and cleaning up bitmaps next. If you would consider a PR I will send one your way.

abbbi added a commit that referenced this issue Oct 28, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@sentry0
Copy link

sentry0 commented Oct 29, 2024

Here is what I'm currently using to rotate my backups - https://github.com/slackdaystudio/vmsnap

This NodeJS application is identical to the original bash script but also scrubs bitmaps and checkpoints off VMs when rotating.

I'm in the middle of testing it now. My tests are coming back green, but I want to wait until the end of the month to see if the new checkpoint will be cut properly.

@rmk177
Copy link

rmk177 commented Feb 20, 2025

Let me reopen discussion. As far as I understand, the proper way is to merge backups is using some daemon on host, that monitor autorotate periods, but for now we have two standalone applications for backups and restore, that don't suits for this. Neverless we can use some commandline options that instructs on next backup make autorotate. For example, --groupBy monthly --rotate-depth 2 on next daily virtnbdbackup merge incremental backups for previous periods to next full backup and cleans not needed backups for specified periods. As far as I understand these operations makes image on restore for backup chain, so we can reuse them.

@sentry0
Copy link

sentry0 commented Feb 26, 2025

Let me reopen discussion. As far as I understand, the proper way is to merge backups is using some daemon on host, that monitor autorotate periods, but for now we have two standalone applications for backups and restore, that don't suits for this. Neverless we can use some commandline options that instructs on next backup make autorotate. For example, --groupBy monthly --rotate-depth 2 on next daily virtnbdbackup merge incremental backups for previous periods to next full backup and cleans not needed backups for specified periods. As far as I understand these operations makes image on restore for backup chain, so we can reuse them.

The vmsnap Node app can produce snapshots based on a configurable time period (using the --groupBy flag). This will lay out the backups in directories named after the period in a specific format. If you ran a new monthly snapshot today (2025-02-26) you would get a folder named DOMAIN_NAME/vmsnap-backup-monthly-2025-02 in your backup dir root.

This means at the start of every new period (month, quarter, bi-annual, year) vmsnap creates a new full backup and erases the old checkpoints from the domain. You can track down the correct period folder if you need to roll back to a point in time.

The side effect is that all periods start with a full backup and incremental backups afterward. This is not great for space if you run daily backups with --groupBy=month on large domains, as it produces one full backup every month.

I set my backups to be --groupBy=bi-annual. That will produce one full backup every 6 months and is suitable for storage for my needs. I also use the --prune flag to clean up old directories automatically around the 90-day mark of the current period.

This is the full command I run to backup my VMs:

vmsnap --backup --domains=DOMAINS --output=BACKUP_ROOT --groupBy=bi-annual --prune

This produces the following directory structure in my BACKUP_ROOT location:

/BACKUP_ROOT/
├── [   3]  Domain_1
│   └── [  14]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_2
│   └── [  35]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_3
│   └── [  76]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_4
│   └── [   9]  vmsnap-backup-bi-annually-2025-p1
└── [   3]  Domain_5
    └── [  32]  vmsnap-backup-bi-annually-2025-p1

Does this help explain the scheduling part, or have I added confusion?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

4 participants