Rclone only prune local

[kwiesmueller]

I wanted to check if there is a way to only prune local files?
I want to make sure only the last X backups are kept locally (to save disk space) and backups are kept unlimited on the remote server (as I can do storage management there in other ways).

I was surprised just now that the script pruned my remote backups.
Right now I'm struggling because somehow my local disk keeps getting full with backups even though I don't have players on the server. Pause is on, but it looks like the container runs a backup, tries upload, and prunes remotely. I had auth issues (because the token expired) which might have caused that.

[itzg]

The local tar file is only supposed to be retained long enough for the rclone copy operation:

docker-mc-backup/scripts/opt/backup-loop.sh

Lines 407 to 408 in 3c9c6f9

	command rclone copy "${outFile}" "${RCLONE_REMOTE}:${RCLONE_DEST_DIR}"
	rm "${outFile}"

It sounds like the entire backup lookup bailed out due to the auth issues and left behind the temp tar files. Can you provide any recent logs from one of those failures?

As for the pausing behavior, make sure you have pulled the latest versions of both the backup and minecraft server images. The backups should be skipping when the server is paused.

[kwiesmueller]

Yeah, the rclone failed because the token expired. A small issue with my setup.
I'm turning down the server on my K8s cluster if no one is using it for a while. But as a result the rclone doesn't run either so the token expires. So every time the backup was tried (or the container restarted) it created a file and then failed.
I'll have to think about a solution, like maybe sharing the PV and running the backup as a separate pod so it stays alive.

But either way it could make sense to decouple the delete/cleanup from the upload success. It could make sense to keep the file so backups don't get lost, but only if they get uploaded and deleted eventually.
Would it make sense to upload the entire local backup folder every time and clean it up. And then also make sure to prune the local files?

[itzg]

But either way it could make sense to decouple the delete/cleanup from the upload success. It could make sense to keep the file so backups don't get lost, but only if they get uploaded and deleted eventually.
Would it make sense to upload the entire local backup folder every time and clean it up. And then also make sure to prune the local files?

That's what I will want the code to be doing since it is apparently not at this time. That's why I was asking for logs to confirm the incorrect error handling. I'll blindly make the change and hope that it resolves future occurrences.

[kwiesmueller]

Sorry don't have the logs anymore since the container restarted. But they were rclone errors about the token being expired.
I'll save them next time.

[itzg]

That's ok. I'm actually more interested in the logs surrounding the failed call and more specifically if the container restarted when the rclone operation failed.

[itzg]

As for the original concern about the pruning of remote files, I suggest setting PRUNE_BACKUPS_DAYS to a very large value to effectively disable pruning or reduce the amount that is pruned. When operating normally, there is nothing to be pruned, so a distinct option for that is not applicable.

[itzg]

The latest image should now gracefully continue from a failed rclone copy operation and cleanup the temporary, local tar file. Can you try out the latest image and confirm it behaves as you expect? Meanwhile, I'm afraid you will need to manually cleanup the leftover tar files.

[kwiesmueller]

Will try asap