In 2009, David A. Wheeler wrote a comprehensive article covering problems with Unix/Linux/POSIX filenames¹. Given that the OS naïvely treats filenames as a simple stream of bytes, he advocated that developers use UTF-8 for encoding filenames. He mentioned the issue of multiple normalisation systems being used to encode characters that have more than one Unicode representation but glossed over it because such problems are “overshadowed by the terrible awful even worse problems caused by filenames all being in random unguessable charsets”.
I’m guessing that, by now, most developers on Unix-like systems would be using UTF-8 for filenames – though a decade after these articles were published, there still doesn’t seem to be any good/universal solution to the problem of characters with multiple Unicode representations.
You should normalize names on write, on read is very hard to fix. You can have a perfectly valid, denormalized strings representing codepoints with different normalizations.
So if you have four possible normalizations: NFD, NFC, NFKD, NFKC and your string has N ambiguous codepoints, the number of possible strings you need to try is N^4.
NFD was used by HFS+, but got abandandoned by the current insecure APFS. (which uses unidentifiable names). NFD is faster to produce, but NFC is complete and needs less space. with NFD you can still have reordered sequence variants, and thus nonidentifiable names.
and the NFKD, NFKC hacks should only be used with python internally, because they didn't understood Unicode. or just read the TR's without understanding it.
it will need several more decades until filesystems will find out about their wrong decisions. maybe I'll bug them with CVE's some day.
Indeed. The unix world would be a much happier place if the creat system call normalized the strings it receives to replace literal spaces with non-breaking spaces, and similar stuff. Regular users wouldn't notice, and it would simplify tons of shell scripts.
> if the creat system call normalized the strings it receives
Nowadays, there’s an understanding to assume those bytes are strings encoded in some ISO-8859 variant or UTF-8, but technically, the creat system call doesn’t receive strings; it receives byte arrays.
Historically, that was the (somewhat) right decision because it meant file systems didn’t need to know much about character encodings (they only needed to know the byte value of ‘/‘ and that zero is the name terminator), giving you a nice separation of concerns.
With Unicode, if you want to normalize names on write, or even only reject incorrectly normalized names, or have case-insensitive file names, your file system code needs to know a lot of Unicode. That can be problematic on small embedded systems.
And have tool 1 happily create files that, according to tool 2, compiled with a different flag setting, cannot exist or create multiple files in a directory that, according to tool 2 have the same names?
I know two sort-of examples of this. Firstly, there’s MS-DOS long file names. That’s a hack (in the positive sense of the word) that gives code that doesn’t know about long file names the 8.3 file names that it expects.
It works, but code that isn’t aware of long file names will only write 8.3 ones, so even a simple file copy using an old copy tool will drop long file names.
Secondly, there’s macOS. It has a minor thing with directory separators. The Unix layer thinks ‘/‘ is the directory separator, old Mac code thinks it’s ‘:’. That works relatively well, mostly because old style Mac code doesn’t use file paths in the UI.
And of course, every desktop OS has to deal with it when it mounts various drives that handle file names differently. HFS+ used NFD normalization, most other file systems that normalize names use NFC, disks may be case preserving or not, case insensitive or not, normalization insensitive or not, etc.
On Unix-like OSes a path /a/b/c/d/e/f can walk six different file systems, each with different rules (even without soft or hard links). It wouldn’t surprise me to find bugs in programs there.
Side note - after all these years I still don't feel comfortable with using special characters (like ą, ż, ź) and spaces in filenames in Windows. DOS times sit deeply in my soul and It just doesn't feel right.
This is amusing but my filesystem has no spaces in it. It's still a pain in the ass from the command line to handle spaces and instead of `-print0` and all that shit I just ban spaces from my filesystem. `-` and `_` are sufficient as spacers.
I know zsh handles auto-complete well but I can `y-a-w` out of my `nvim` `:terminal` a lot easier.
I like this approach. How do you enforce your ban? Is there a mount(8) option for that? Or do you simply avoid these filenames "by hand"? What happens if you untar an archive and it contains filenames with spaces?
Haha, I do not enforce the ban. You are correct. If I untar'd something I'd be sitting there with a space there and I'd look like a fool. In practice, most of these files are in my `~/Downloads` and I don't really keep them around there very much.
Do your files really not have file extensions or is it just that they are hidden by default in the file browser?
In Mac OS extensions are important but they are hidden by default. I always force them to be shown when I setup a new Mac. Hiding them obscures important information. Hiding them offers nearly no benefit.
I use Linux exclusively, where file extensions are entirely optional. While, yes, they are shown in the file browser, that's a tool of last resort for me: my display is typically a grid of terminals and I do most of my work through them. Text files, such as executable scripts with shebangs, do not get file extensions. Plain text files, such as when I'm taking notes about something, do not get file extensions. Stuff like images, music, etc get file extensions.
Windows user here. I feel that the computer should work for the human, not the other way around. I use whatever unicode characters are appropriate for the file, except the minimum set of disallowed characters: colon, slash, asterisk...
Any tool that can't handle that is faulty, and I'll either stop using it or make a temporary symlink to get the job done. But most tools are fine with unicode these days. Of course I speak from the perspective of a personal computer situation with no deadlines or business requirements or software limitations.
I still use the old cmd.exe with the Terminal font. It doesn't even render most unicode properly, they come out as ?. I deal with it because I like cmd.exe and I like Terminal and I'd rather see ? than butcher the file's name by romanizing it.
I’m guessing that, by now, most developers on Unix-like systems would be using UTF-8 for filenames – though a decade after these articles were published, there still doesn’t seem to be any good/universal solution to the problem of characters with multiple Unicode representations.
¹ https://dwheeler.com/essays/fixing-unix-linux-filenames.html