WARNING: failed to getxattr

[nijhawank]

Running "rmlint --xattr" is throwing these warnings...
WARNING: failed to getxattr for blah blah blah: Attribute not found

I assume these warnings are normal when running for the first time as related extended attributes are not there yet but I expect these warnings to stop on subsequent executions on the same set of files but they do not.
I even checked the files with "xattr -l " and it correctly shows the rmlint user.rmlint.blake2b.cksum and user.rmlint.blake2b.mtime extended attributes which means that rmlint was able to successfully write the extended attributes for the files but then why rmlint is throwing these warnings?

This is on macOS Mojave with APFS filesystem. rmlint was installed using brew.

[sahib]

e84681f should fix this. It seems macOS uses a different error code than ENODATA to say "this attribute does not exist".

[ylluminate]

I'm not seeing speed improvements when using --xattr with macOS Catalina after having done an initial scan. Furthermore, when I run xattr -l on a file, I don't see any attributes listed. This is on an APFS volume. rmlint version:

version 2.10.1 compiled: Nov 12 2020 at [20:14:03] "Ludicrous Lemur" (rev unknown)
compiled with: -mounts -nonstripped -fiemap +sha512 +bigfiles +intl +replay +xattr -btrfs-support

Is there any way to further troubleshoot this and verify that the extended attributes are working?

[SeeSpotRun]

So the warnings have stopped, but now the xattr's are no longer being written to the files?

[ylluminate]

@SeeSpotRun this is what I'm seeing, yes.

[SeeSpotRun]

Strange, because e84681f doesn't actually change the "payload" line which is the rm_sys_setxattr() line here:

https://github.com/sahib/rmlint/blame/develop/lib/xattr.c#L185

Has something else changed? Can you manage to write xattrs for any files?

[ylluminate]

Hmm, do you have a specific test scenario I can execute to give you the info needed to isolate this?

[SeeSpotRun]

The following should be enough to test:

$ mkdir testdir
$ echo xx > testdir/a
$ echo xx > testdir/b
$ ./rmlint --xattr testdir # assuming you are running from the dir that you compiled rmlint in
$ xattr -l testdir/*

If you have different filesystem types maybe try it on each of them.

[ylluminate]

Alright, this worked perfectly (APFS volume). I found that the attributes WERE indeed stored correctly at least in this test run:

$ cd /path/to/a
$ mkdir testdir
$ echo xx > testdir/a
$ echo xx > testdir/b
$ ls testdir/
 a   b
$ rmlint --xattr testdir
WARNING: failed to getxattr for /path/to/a/testdir/a: Attribute not found
WARNING: failed to getxattr for /path/to/a/testdir/b: Attribute not found

# Duplicate(s):
    ls '/path/to/a/testdir/a'
    rm '/path/to/a/testdir/b'

==> Note: Please use the saved script below for removal, not the above output.
==> In total 2 files, whereof 1 are duplicates in 1 groups.
==> This equals 3 B of duplicates which could be removed.
==> Scanning took in total 0.245s.

Wrote a json file to: /path/to/a/rmlint.json
Wrote a sh file to: /path/to/a/rmlint.sh
$ xattr -l testdir/*
testdir/a: user.rmlint.blake2b.cksum:
00000000  34 33 36 36 35 36 33 62 39 31 30 65 33 34 38 36  |4366563b910e3486|
00000010  36 33 35 33 61 31 62 66 63 62 64 30 36 32 61 33  |6353a1bfcbd062a3|
00000020  32 35 64 39 32 62 34 66 39 36 31 62 30 38 36 65  |25d92b4f961b086e|
00000030  31 37 34 61 31 37 33 32 33 35 35 31 30 63 30 63  |174a173235510c0c|
00000040  34 31 63 37 34 31 31 33 35 61 66 36 62 63 32 38  |41c741135af6bc28|
00000050  35 31 33 30 34 38 61 39 62 35 32 61 33 30 38 38  |513048a9b52a3088|
00000060  63 63 62 39 39 36 35 31 30 30 36 66 36 66 30 37  |ccb99651006f6f07|
00000070  31 65 66 36 35 62 63 64 33 64 31 33 66 37 32 39  |1ef65bcd3d13f729|
00000080  00                                               |.|
00000081
testdir/a: user.rmlint.blake2b.mtime: 1615841589.4580715
testdir/b: user.rmlint.blake2b.cksum:
00000000  34 33 36 36 35 36 33 62 39 31 30 65 33 34 38 36  |4366563b910e3486|
00000010  36 33 35 33 61 31 62 66 63 62 64 30 36 32 61 33  |6353a1bfcbd062a3|
00000020  32 35 64 39 32 62 34 66 39 36 31 62 30 38 36 65  |25d92b4f961b086e|
00000030  31 37 34 61 31 37 33 32 33 35 35 31 30 63 30 63  |174a173235510c0c|
00000040  34 31 63 37 34 31 31 33 35 61 66 36 62 63 32 38  |41c741135af6bc28|
00000050  35 31 33 30 34 38 61 39 62 35 32 61 33 30 38 38  |513048a9b52a3088|
00000060  63 63 62 39 39 36 35 31 30 30 36 66 36 66 30 37  |ccb99651006f6f07|
00000070  31 65 66 36 35 62 63 64 33 64 31 33 66 37 32 39  |1ef65bcd3d13f729|
00000080  00                                               |.|
00000081
testdir/b: user.rmlint.blake2b.mtime: 1615841595.0322227

So I looked again at the previous run file attributes. When I first tested, xattr -l did NOT yield any results (time before last), but this time I am indeed finding these same named attributes present for other files in these folders.

I'm going to do ANOTHER run now and see if the time improves, but I fear that it won't...

Is there another test (or argument I'm missing) I can perform during the run that will verify as to whether the extended attributes are not only present, but also being COMPARED/USED vs generating new ones DURING rmlint's actual operation?

[SeeSpotRun]

Is there another test (or argument I'm missing) I can perform during the run that will verify as to whether the extended attributes are not only present, but also being COMPARED/USED vs generating new ones DURING rmlint's actual operation?

Not at present.

I suspect what you are seeing is same as #462. Non-duplicate files are generally not fully hashed because their uniqueness is determined either by their unique file length, or because they differ from other same-length files somewhere in the first few bytes or kilobytes.

I'm working on an option for #462 and this may also resolve your problem.

[ylluminate]

Hmm, perhaps you're right. I'm extremely discombobulated right now because on this last run it flew through the process and comparison of around 6TB of files within just about half an hour. For some reason the first three runs did NOT do this and ran multiple days cumulatively (approximately 20 hours a run), but this time it clearly used the extended attribute data...

By the way, are you aware of whether or not extended attributes work properly with ZFS on macOS yet? I assume that it does since they seem to work fine otherwise, but I thought I'd pose the question before digging into a protracted test.

[SeeSpotRun]

are you aware of whether or not extended attributes work properly with ZFS on macOS yet?

No idea

[ylluminate]

Can you tell me what I can run outside of rmlint to generate these extended attributes independently of rmlint such that rmlint will recognize them when it is run at a later time? For example, I'm in the middle of copying several TB of data and would like to store the checksums while the copy is still proceeding so as to speed up the actual rmlint process once I start it in a couple days...

[SeeSpotRun]

You can just do:

$ rmlint -o null --hash-uniques --xattr <source_file>
$ cp <source_file> <dest>