I'm curious how much of this behaviour is still intentional design vs. just inertia. Are the modern filesystems still constrained by these older choices, or is it mostly for compatibility?
The filesystems of macOS are particularly opinionated, much more than most Unices which tend toward "anything is allowed [and usually preserved] except \0 and /".
macOS supports case-insensitivity[0] and performs unicode normalization[1] on filenames, and decomposes name data to an extent that the question "what does the fs see" is a bit moot.
With that said, the internal storage of filenames in APFS are a nul-terminated UTF-8 string[2], with (i'm pretty sure) colons as colons, which the Finder displays as slashes.
[0] if you make a file named "Makefile" then touch a file named "makefile", it'll touch the first file, instead of making a second file.
[1] if you make a file named "schön" (s-c-h-combining¨-o-n) and then search for (s-c-h-ö-n), you can find it, or vice versa. The particular normalization/canonicalization used is NFD.
Well, strictly speaking Linux does too, since it supports mounting local or remote filesystems with this feature
For a long time, the real distinction was that “native” Linux filesystems didn’t support it, but “foreign” ones did. However, nowadays even some of the “native” filesystems have optional support for case-insensitivity (e.g. casefold feature on ext4)
The real difference now: on macOS, it is normal to have this feature turned on, exceptional to have it disabled; on Linux, it is the other way around
Combining marks come after the character they modify, btw. (Presumably thanks to support from things like harfbuzz, modern systems will happily put two dots above an h.)
You can imagine many internal APIs as taking "look up this <name> from directory <handle>". Surprisingly often the only practical limit is that the name does not contain NULL bytes.
Path separators, whether to accept directory entries with path separator in them, etc. are usually handled layer above