Job keeps failing with unable to save snapshot: snapshot is empty

[budimanjojo]

Describe the bug

After upgrading to v0.9.0, one of my backup job keeps failing and I don't know why.

Steps to reproduce

No idea yet.

Expected behavior

Job should run fine as the older version.

Actual results

Additional context

Here's the log of the failing pod:

Defaulted container "restic" out of: restic, provide-timezone (init)
Starting container
VolSync restic container version: v0.9.0+4e5e959
backup
restic 0.16.4 compiled with go1.21.8 on linux/amd64
Testing mandatory env variables
== Checking directory for content ===
== Initialize Dir =======
ID        Time                 Host        Tags        Paths
------------------------------------------------------------
1b04f8ab  2024-03-12 00:00:31  volsync                 /data
e58e09d2  2024-03-13 00:00:40  volsync                 /data
8befa5b1  2024-03-14 00:00:29  volsync                 /data
0c76ddb9  2024-03-15 00:00:42  volsync                 /data
13860d3c  2024-03-16 00:00:28  volsync                 /data
c2fb0527  2024-03-17 00:00:39  volsync                 /data
ec489a85  2024-03-18 00:00:40  volsync                 /data
a0b071d0  2024-03-19 00:00:43  volsync                 /data
6740b93d  2024-03-20 00:00:35  volsync                 /data
8a6ae86d  2024-03-21 00:00:32  volsync                 /data
------------------------------------------------------------
10 snapshots
=== Starting backup ===
/data /
using parent snapshot 8a6ae86d
Fatal: unable to save snapshot: snapshot is empty

I don't know what's causing the problem or how to fix it, but I'll be glad to provide more information if needed to help with this issue. Thanks before!

[jalim]

Update: so the snapshot was empty, it seems that when I was using 0.8.1 the empty snapshot didn't result in an error. So thinking this wasn't a bug after all in my case.

I’m seeing the same issue myself if any extra info needed. Only occurs on one of my backup jobs the remainder are all working as expected. Only thing different is that this is the newest backup job I’ve created, not sure what version of volsync it was originally created with but likely 0.8.1. Issue only arose after upgrading to 0.9.0.

[tesshuflower]

My guess is this is due to a fix I put in to ignore "lost+found" (it is new for v0.9.0). Do you have a lost+found dir in your empty source volume?

We do have a check to see if the source is empty and skip the backup, but it will think the dir is not empty if there's a lost+found. I think I will need to fix this empty dir check to also ignore lost+found.

[budimanjojo]

@tesshuflower yeah the directory is empty, which is weird because it shouldn't be empty. I don't know if it's the program using the PVC deleted the data or volsync. And I just notice that all the backups doesn't have any data in it so there goes my data :(

I'm gonna close this issue now as this is an intended behavior. But I still need to investigate how the data went missing. Thank you for the reply!

[budimanjojo]

@tesshuflower So, it was my BAD. I'm mounting the storage to a wrong location the whole time (budimanjojo/home-cluster@f02ee27).

But, there's a "lost+found" directory there. And, can you provide me on how to make the job not failing when the directory is empty? Like make it just skip without failing (maybe your "I think I will need to fix this empty dir check to also ignore lost+found" means this?)

[tesshuflower]

@budimanjojo I think the "lost+found" is causing an issue - normally our empty dir check would skip the backup if there was no data present in the volume. But now that we tell restic to ignore "lost+found", we still try to backup, only for restic to error out with no data to backup.

I will reopen this issue to implement a fix to the empty dir check so it ignores "lost+found".