[OS X] [APFS] Make sure our file system unicode normalization will be correct

[guruz]

https://news.ycombinator.com/item?id=13953800 -> http://mjtsai.com/blog/2017/03/24/apfss-bag-of-bytes-filenames/

https://eclecticlight.co/2017/04/07/apfs-and-macos-10-13-many-apps-and-tools-will-need-to-be-revised/
https://eclecticlight.co/2017/04/06/apfs-is-currently-unusable-with-most-non-english-languages/

[ogoffart]

For writing file, we only rely on Qt, in the propagator, which should normalize filenames, i believe.
When reading file in the update phase, on mac, we normalize their name already (with iconv)

In other words, I believe we should be fine. But it's worth testing.

[guruz]

Moving to 2.4, we should probably go with the approach of using Qt for directory iteration and stat()ing as mentioned in #5661 (comment)

[guruz]

@jturcotte Is your macOS version able to create disk images with APFS?

[guruz]

From https://mjtsai.com/blog/2017/06/27/apfs-native-normalization/

The default for macOS 10.13 will be case-insensitive APFS. It is normalization-preserving (unlike HFS+) but not normalization-sensitive. I expect this to be highly compatible with existing Mac apps. The main difference is that when you read filenames they are no longer necessarily in Form D, but you shouldn’t have been relying on that, anyway.

[guruz]

TODO:

• Test that 2.3.3 is OK
• Test that 2.4.0 is OK after having csync: Use Qt for encoding conversion instead of iconv #5875

[ogoffart]

It is normalization-preserving (unlike HFS+) but not normalization-sensitive.

Then there should not be any problem. We already normalize anyway on mac.
The issues might be that we always write in the Form D, even if it was on the disk with another form. Since it is "not normalization-sensitive", that will not be a problem for owncloud itself. However, if there are application that rely on a filename NOT to be in the D form, this might be a problem when we update a file from the server. (Such application would be currently broken with HFS+, so that's only future applications)

So I bet there won't be issues.

[jturcotte]

@guruz Doesn't look like I can for a .dmg, only Mac OS Journaled, ExFAT and FAT.

[guruz]

Test that 2.4.0 is OK after having #5875

On my Sierra Mac 10.12.5 it does NOT work. A file Amélie (filename created on shell using touch copy pasted from https://en.wikipedia.org/wiki/Unicode_equivalence#Example ) is silently ignored..

Well not silentely, it will print 07-14 13:57:34:401 [ warning sync.csync ]: Local stat failed, errno 2

As mentioned above #5650 (comment) this might be better with 10.13..

(BTW, the same file works on a HFS+ sync dir)

[guruz]

@SamuAlfageme Do you have a Mac with High Sierra?

[guruz]

Same behaviour with 2.3.3 :[

The file gets silently ignored on an APFS volume but gets synced on a HFS+ volume (probably because there macOS normalizes..)

[guruz]

@michaelstingl This is something we need to investigate further and discuss if it is a blocker.

[guruz]

(I'm using 10.12.5..)

https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html

Runtime normalization will also be available in iOS 10.3.3 and macOS Sierra 10.12.6. Being normalization-insensitive ensures that normalization variants of a filename cannot be created in the same directory, and that a filename can be found with any of its normalization variants. This means that developers don’t need to do any additional work to ensure correct normalization behavior in these versions of macOS and iOS.

[michaelstingl]

Yes, further investigation required. I've heard Apples API's/interfaces aren't final, could be easier when final release comes closer.

@guruz @SamuAlfageme What are the steps to verify/reproduce this?

[guruz]

@michaelstingl Ideally: To have a Mac with High Sierra (10.13) beta
(Maybe there is other ways, still investigating)

[michaelstingl]

I will prepare a VM with High Sierra.

[guruz]

Thanks a lot. Just be aware that according to what I read they differ in behavior between fresh install of the OS and update of OS (update converts the fs) Although maybe that was an iOS only thing

[michaelstingl]

My idea was to clone my existing Sierra VM, then update to High Sierra. I will keep an Snapshot, so I can revert whenever it's needed. I can also try a High Sierra clean install in another VM.

[michaelstingl]

First step in the Updater asks if I want to upgrade my system volume to APFS:

Checkbox is disabled by default.

[michaelstingl]

It seems migrating the volume which is in a VM image is very difficult. I wait all day and remaining time became bigger and bigger. (up to 10h 55m) Now, it starts to decrease. Let's see…

[guruz]

tested on High Sierra APFS VM with APFS, not case sensitive, not encrypted.

2.3.2/2.3.3 properly syncs the filename Amélie pasted from wikipedia.
master (with #5875) properly syncs the filename Amélie pasted from wikipedia.

master (with #5875) also properly syncs an emoji filename. (2.3 doesn't, it's a new feature of master)

The OS didn't let me create filenames with invalid utf8 (i think)

So I conclude APFS is fine with High Sierra, bad with Sierra which has APFS only for testing.
I'll close this issue on monday or so after we thought a bit more about this. 👍