Windows paths allow unpaired surrogates

[daurnimator]

Windows allows unpaired surrogates in paths; which are not valid UTF-16LE.

Originally posted by @daurnimator in #2527

[emekoi]

see here.

[tgschultz]

The only way I can think to deal with this universally would be to consider all file paths to be raw bytes and ignore encoding. I'm not sure that's viable in a cross-platform api.

[shawnl]

@tgschultz these can be encoded with WTF-8, if someone cares enough to do the work. I haven't even gotten my UTF-8 improvements merged yet. The problem is that then a whole bunch of APIs have to accept WTF-8 instead of UTF-8, and have flags to tell them apart, so its a huge maintenance burden to support junk. As Windows already has a bunch of nutty path-name restrictions, I think erroring out on non-valid unicode is fine.

[mikdusan]

came across this article might be relevant: https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html

[tgschultz]

Actually yeah, I think you're right. Erroring out on the edge case of unpaired surrogates is probably better, and then the programmer can use the OS API directly if they really need to support it.

Does anyone have information about how often such cases occur in reality? My first guess would be in Asian locales where 2-byte character representations were used pre-UCS2.

Another alternative would be having a real string type in std in the form of a tagged byte buffer with the encoding specified. Of course, that also means rewriting a whole lot of APIs.

[daurnimator]

Actually yeah, I think you're right. Erroring out on the edge case of unpaired surrogates is probably better, and then the programmer can use the OS API directly if they really need to support it.

I think erroring out is the wrong option, converting to/from WTF-8 is the well-established solution in other languages.

Does anyone have information about how often such cases occur in reality?

They get onto the file system often enough if someone is truncating strings in Java or Javascript (languages where strings are UCS-2) without realising they could be astral plane characters (e.g. emojis).
I would not expect e.g. directory iteration in zig to suddenly fail in the presence of such characters. Or that a path returned from a directory iteration is unable to be passed to delete.

[jdalton]

Reference for anyone wanting to tackle a support PR bun handles it here:

https://github.com/oven-sh/bun/blob/f77b217abf2f33fdea4a50298d07cd601a655b0d/src/string_immutable.zig#L1835-L1848