snapshotter: use syncfs system call

[zhouhaibing089]

sync system call triggers a full page cache sync which may not always work, especially in kubernetes environment where it is very easy to be interfered by others. I have seen several cases where a broken nfs mount is blocking kaniko from doing its job.

With syncfs, it only writes cache back to disk for the current filesystem that is used by kaniko which is supposed to be more reliable.

Here is the details that I captured recently. From kaniko executor logs:

/workspace # executor --no-push
INFO[0000] Retrieving image manifest ubuntu:22.04
INFO[0000] Retrieving image ubuntu:22.04 from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest ubuntu:22.04
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'ubuntu:22.04' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN apt-get update requires it.
INFO[0003] RUN apt-get update
INFO[0003] Initializing snapshotter ...
INFO[0003] Taking snapshot of full filesystem...

It hangs there forever in Taking snapshot of full filesystem - see #1333 as well.

I looked the stack of kaniko process and see this:

root@<my-host-name>:/proc/180716/task# cat 182585/stack
[<0>] sync_inodes_sb+0x10b/0x2b0
[<0>] sync_inodes_one_sb+0x15/0x20
[<0>] iterate_supers+0xa0/0x100
[<0>] ksys_sync+0x42/0xb0
[<0>] __do_sys_sync+0xe/0x20
[<0>] do_syscall_64+0x59/0xc0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

where 180716 is the pid of kaniko executor, and one of its subtask is waiting in ksys_sync forever.

At that point of time, running sync (command) on the host was also stuck forever. We further checked and then identified the system call was waiting for one of nfs mount to flush its superblock (but due to connectivity issues, it can't complete).

root@<my-host-name>:/# dmesg -T | grep nfs
[Thu Oct 26 04:05:18 2023] FS-Cache: Netfs 'nfs' registered for caching
[Thu Oct 26 04:12:00 2023]  ? nfs_ctx_key_to_expire+0x69/0xf0 [nfs]
[Thu Oct 26 04:12:00 2023]  nfs_wb_all+0x28/0xf0 [nfs]
[Thu Oct 26 04:12:00 2023]  nfs4_file_flush+0x73/0xb0 [nfsv4]
[Thu Oct 26 04:12:28 2023] nfs: server <nfs-server-host-name> not responding, still trying
[Thu Oct 26 04:12:43 2023] nfs: server <nfs-server-host-name> not responding, still trying
[Thu Oct 26 04:14:00 2023]  ? nfs_ctx_key_to_expire+0x69/0xf0 [nfs]
[Thu Oct 26 04:14:00 2023]  nfs_wb_all+0x28/0xf0 [nfs]
[Thu Oct 26 04:14:00 2023]  nfs4_file_flush+0x73/0xb0 [nfsv4]

While this is technically not related to kaniko, but by switching to syncfs with fd from current directory, it is going to only sync the current file system which holds the provided fd, and this is not affected by external filesystems.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes unit tests
• Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

• The code flow looks good.
• Unit tests and or integration tests added.

[aaron-prindle]

Thanks for the PR here @zhouhaibing089, appreciate the fix! Left one nit comment, once that is addressed we can get this merged

[aaron-prindle]

LGTM! Thanks @zhouhaibing089!

[rayunstop]

great job, help a lot. Thanks.