|
|
|
|
|
|
by svieira
3827 days ago
|
|
|
Python is UTF-8 by default and only upgrades to UTF-16 / 32 when it would make sense to do so given the characters in the string. > UTF-32 is wrong, [because] it's wasteful and still doesn't accomplish what people naively expect due to grapheme clusters potentially taking more than one UTF-32 code unit. Out of curiosity, is there a correct way to encode unicode that doesn't involve this level of surprise? I thought that this was still an unsolved problem at this point. |
|
|
Then you offer a data structure that lets you perform O(1) or O(logn) operations on sequences of single-character strings.
If it's read-only you could make it just be an index, blah blah the details don't matter a lot, the point is you can make something that's both correct to grapheme clusters and probably more space-efficient than UTF-32 despite the extra data.
And then the encoding inside the character strings isn't particularly important, but might as well use UTF-8.
-
Either that or make yourself a hilariously inefficient format based on:
UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in Stream-Safe Text Format if it would not contain any sequences of non-starters longer than 30 characters in length when normalized to NFKD.
Who's with me on 128-byte characters.