[msla]

> On UNIX, paths are UTF-8 by convention, but not forced to be valid.

On UNIX, paths are a sequence of bytes, with two bytes being sacred to the kernel (0x2F, used to separate path elements, and 0x00, used to terminate paths) and no other bytes being interpreted in any way. Any character encoding which respects the sacred bytes by not using them to encode any other characters is therefore usable to make UNIX paths; in fact, a UNIX path can contain multiple encodings, as long as they're all suitably respectful.

That requirement for respect means that UTF-16 and UCS-2 and UCS-4 are not suitable. UTF-7 is, however, as is UTF-8, and all of the ISO/IEC 8859 encodings are as well, not to mention a whole raft of non-standard "extended ASCII" character sets. In theory, UTF-16 in some suitably respectful encoding would work, too, but gouge my eyes out with a goddamned spoon.

[deathanatos]

They know all that. The operative words there are "by convention", not "by requirement".

I.e., their comment is an RFC "OUGHT TO", not an RFC "MUST".

[msla]

My point is that assuming UNIX paths are UTF-8 by default is fragile, and that assuming UNIX paths have any consistent character encoding is also somewhat fragile. You can check to see if the path is UTF-8, but, otherwise, mitts off unless the user explicitly tells you an encoding.

[deathanatos]

> assuming UNIX paths are UTF-8 by default is fragile, and that assuming UNIX paths have any consistent character encoding is also somewhat fragile.

Again, we're both aware that it's not guaranteed. But the convention these days is nonetheless UTF-8.

> mitts off unless the user explicitly tells you an encoding.

This doesn't adequately solve the problem, though. Typed languages have to emit some type of value: having that be the "string" type of the language is useful, as you can do things like printf a message that includes the filename. (Or display an "Open file…" dialog. Or…)

There are middle-grounds, such as byte smuggling and escaping. But if you take the stance that filenames are arbitrary bags of bytes (which is actually a subset: the reality is even worse) — then anything that returns a filename is stuck: it can't return a string ("mitts off"). You can take Rust's approach with Path (a type specific to Paths) but people wail about that all the time too ("why are there so many types?"), and you can't print it because it's not text!

"Error out if the file name isn't conventional" is a pragmatic tradeoff: bad file names will cause errors, but it makes basically all other operations much more tractable. It's not worth supporting insane file names.

There are workarounds, of course (such as just replacement-charactering "�" anything that can't be understood), and finding a format that can encode Paths when transmitting them, but these all take more time and effort. Allowing non-text files introduces unnecessary complexity and bugs into every single program that needs to deal with the file system.