My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to vulnerabilities in the past.
I disagree. UTF-8 is the right default. Pure ASCII strings are rarely needed in modern software. Unless you know that you need ASCII, your strings should support unicode.
I agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values.
IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string.
For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.
Homoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode).
That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.
Of course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.