[bflesch]

I agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values.

IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string.

For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.

[Georgelemental]

Homoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode).

That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.

[bflesch]

Of course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.

[tosti]

The NT kernel predates UTF8 by at least 3 years. It originally used UCS2, which covers the basic multilingual plane of the first unicode standard.