[est]

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes.

In python:

    len(u'汉字') == 2
    len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options
    len(u'汉字'.encode('utf8')) == 6

[csense]

Issues like this are why I hate internationalization.

If it was simple as making everything Unicode and it Just Working, it would be possible. But the number of difficulties and problems I've seen have made me decide -- and tell everyone I know -- to avoid dealing with internationalization if you value your sanity.

Issues discussed here:

* Different incompatible variable-length encodings

* Broken implementations

* Character count != length

Issues discussed elsewhere:

* It's the major showstopper keeping people away from Python 3

* Right-to-left vs. left-to-right [1]

* BOM at the beginning of the stream

Conceptual issues -- questions I honestly don't know the answer to when it comes to internationalization. I don't even know where to look to find answers to these:

* If I split() a string, does each piece get its own BOM?

* If I copy-paste text characters from A into B, what encoding does B save as? If B isn't a text editor, what happens?

* If a chr(0x20) is part of a multi-byte escape sequence, does it count as a space when I use .split()?

* When it encounters right-to-left, does the renderer have to scan the entire string to figure out how far to the right it goes? Wouldn't this mean someone could create a malicious length-n string that took O(n^2) time to process?

* What happens if I try to print() a very long line -- more than a screenful -- with a right-to-left escape in a terminal?

* If I have a custom stream object, and I write characters to it, how does it "know" when to write the BOM?

* Do operators like [] operate on characters, bytes, 16-bit words, or something else?

* Does getting the length of a string really require a custom for loop with a complicated bit-twiddling expression?

* Is it possible for a zero byte to be part of a multibyte sequence representing a character? How does this work with C API's that expect zero-terminated strings?

* If I split() a string to extract words, how do the substrings know the BOM, right-to-left, and other states that apply to themselves? What if those strings are concatenated with other strings that have different values for those states?

* What exactly does "generating locales" do on Debian/Ubuntu and why aren't those files shipped with all the other binary parts of the distribution? All I know about locale generation is that it's some magic incantation you need to speak to keep apt-get from screaming bloody murder every time you run it on a newly debootstrapped chroot.

* Is there a MIME type for each historical, current, and future encoding? How do Web things know which encoding a document's in?

* How do other tools know what encoding a document uses? Is this something the user has to manually tell the tool -- should I be saying nano thing.txt --encoding=utf8? If the information about the format isn't stored anywhere, do you just guess until you get something that seems not to cause problems?

* If you're using UTF-16, what endianness is used? Is it the same as the machine endianness, or fixed? What operations cause endian conversion?

* Should my C programs handle the possibility that sizeof(char) != 1? Or at least check for this case and spit out a warning or error?

* What automated tools exist to remove BOM's or change accented characters into regular ones, if other automated tools don't accept Unicode? Once upon a time, I could not get javac to recognize source files I'd downloaded which had the author's name, which included an 'o' with two dots over it, in comments. That was the only non-ASCII character in the files, and I ended up removing them; syncing local patches with upstream would have been a nightmare. Do people in different countries run incompatible versions of programming languages that won't accept source files that are byte-for-byte identical? It sounds ridiculous, but this experience suggests it may be the case.

[1] http://xkcd.com/1137/

[notJim]

I have attempted to answer your questions, but as a general comment, most of them would be answered by having a basic understanding of unicode and popular encodings for it. You're a programmer, understanding this is part of your job. (If you aren't a programmer,then my my do you sure ask a lot of technical questions.) You could read this: http://www.joelonsoftware.com/articles/Unicode.html. Then read a bit more. Then forget everything. Then read a bunch more and then give a small talk on unicode. Then suddenly it's 2AM and you're answering asinine questions about unicode on HN. What I'm saying is, be careful.

One last general comment I have is that a lot of your questions relate to things that you shouldn't necessarily need to understand the exact details of to do a lot of things. Instead, you should use an off-the-shelf, battle-tested unicode library. As far as I know, they exist for every platform by now. Of course this doesn't free you from knowing stuff, but it means that instead of knowing exactly the range of the surrogate pairs, all you need is a mental model of what's going on. When you're surprised, you can begin to fill in the gaps.

1. Use UTF-8 if you're using byte strings, or your platforms unicode string type. If the latter, it will have its own internal encoding, and you'll work at the character level. In either case, as soon as a string comes in from elsewhere (network, disk, OS), sanitize it to a known state, i.e., a UTF-8-encoded byte string, or your platform's unicode string type. In case you can't do that, reject the string, log it, and figure out what's going on.

2. Use a non-broken UTF-8 implementation.

3. Yeah. Your UTF-8 implementation is handling this for you now.

4. Not familiar with this/don't use Python enough.

5. Haven't dealt with this, but it's definitely got some complications to it. I would guess more for layout than for programming, but I can't be sure.

6. The BOM tells your decoder to that the forthcoming stream is little endian or big endian. It itself is not actually part of the string. Admittedly, a lot of programs have trouble with BOMS still, which is why you're using UTF-8 (without BOMs, because you don't need it.)

7. No, when you .split, the BOM is no longer part of the string. You don't have a BOM again until you transmit the string or write it to disk, as it's not needed (your implementation uses whatever byte ordering it likes internally, or the one you specified if using a byte string.)

8. The string is probably transmitted in whatever the internal encoding of your OS is. That means UTF-16 on Windows and UTF-8 on Linux, AFAIK. If you're writing a desktop app, your paste event should treat this string as a pile of unsafe garbage until you've put it into a known state (i.e., a unicode object or byte string in a known encoding.) When you save it, it saves in whatever encoding you specify. You should specify UTF-8.

9. I'm not sure exactly what we're talking about here. The byte 0x20 is a valid UTF-8 character (the space), or part of a 2-byte or 4-byte character in UTF-16. However, as long as you're working with a unicode string type, your .split function operators on the logical character level, not the byte level. If you're using a byte string (e.g., python's string type), then yes, the byte 0x20 is a space, because your split method assumes ASCII. If you try to append a byte string containing 0x20 to a unicode string, you should get an exception, unless you also specify an encoder which takes your byte string and turns it into a unicode string. Your unicode string implementation may have a default encoder, in which case the byte string would be interpreted as that encoding, and an exception would only be thrown if it's invalid (which means if the default encoding was UTF-16, this would throw an exception, because 0x20 is not a valid UTF-16 character.) This answer is long, and HN's formatting options are lacking, so let me know, and I'll try to be clearer.

10. Again I haven't yet dealt with RTL, but the characters are in the same order internally regardless of whether they're to be displayed RTL or LTR. It's a sequence of bytes or characters, the encoder and decoder do not care what those characters actually are. So if I have "[RTL Char]ABC ", that's the order it will be in memory, even though it will display as " CBA." In UTF-8, this string would be 7 bytes long, in UTF-16, 10. In both cases, the character length of the string is 5.

11. I'm not sure why this would be a problem, provided your terminal can handle unicode, which most can in my experience (there's some fiddling you have to do in windows.) It should wrap or break the line the same as with RTL. I believe the unicode standard includes rules for how to break text, but not positive.

12. I'm not really sure what you mean. Your object will write whatever bytes it writes. If you're using a UTF-16 encoder, usually you can specify the Endianness and whether to write a BOM or not.

13. If you're using a unicode type in a language like python, [] operates on logical characters. If you're using a byte string type (python's string, a PHP or C string, for ex), [] operates on bytes.

14. If you're using a unicode type, split returns unicode objects, which have their own internal encoding. Again, right-to-left characters look exactly like any other character to the encoder and decoder. If you're using byte strings, you need to use a unicode-aware split function, and tell it what encoding the string is in. It will return to you strings in the same encoding (and endianness.)

16. Not familiar with this.

17. MIME types are separate from encoding. I can have an HTML page that is in UTF-8 or UTF-16, both have the MIME type txt/html, same with txt/plain and so on. MIME and encoding operate at different levels. Web things knowing the encoding is actually fairly complicated. The correct thing to do is to send an HTTP header that specifies the encoding, and add a <meta charset="bla"> attribute in the HEAD of the page. If you don't do this, I think browsers implement various heuristics to attempt to detect the encoding. Having a type for every future encoding is an unreasonable demand, because the future is notoriously difficult to predict. If you have a crystal ball, I'm sure the standards committees would love to hear from you.

18. Also somewhat complicated! I think there are tools which can guess for you, using similar heuristics that I mentioned that browsers use. You should specify the encoding using your text editor, as you showed. It is not too much to ask to tell people to set their text editors up correctly. Your projects should have a convention that all files use, just like with code formatting. If someone tries to check in something that's not valid UTF-8, you can have a hook that rejects the commit if you want. Then they can fix it. The format is not stored anywhere, which is why you should have the convention and yell at people who mess up (not literally, be nice.) If you don't know what a file is, you can use aforementioned tools, or try a few encodings and see what works. Yes, it's a hassle, which is why you should set a convention.

19. You can specify LE/BE when you say what encoding something is. As in, you say, hey encode this here text as UTF-16 LE, and it says right-o, here we go!

20. A C char is not aware of the encoding, so that wouldn't have any effect! Some other guy said that a char is always a byte, so answer: no.

21. These are two wildly different things. A BOM is always FFEF or FEFF, depending on endianness, so you can look for those, and chop off the two bytes, I guess. For dealing with accents, look into unicode normal forms, they define a specific way to compose and decompose accents. I'm not sure about your javac woes, there ought to be some way to tell javac what encoding to expect, like you can do with python. It may be the case that javac guesses the encoding based on your locale, and his was different than yours.

[pjscott]

I'll take a few of those questions.

> If I split() a string, does each piece get its own BOM?

Conceptually, each piece is a sequence of code points. The BOM stuff only comes into play when you turn it into an external encoding. And frankly, I would much rather use UTF-8, explicitly specify that the encoding is UTF-8, and not have to worry about adding a BOM.

> If a chr(0x20) is part of a multi-byte escape sequence, does it count as a space when I use .split()?

In valid UTF-8, all bytes in multibyte characters will have the high bit set. A space can only be represented as the 0x20 byte, and an 0x20 byte can only be a space. If you've got malformed input, then that's a whole other can of worms.

> Is it possible for a zero byte to be part of a multibyte sequence representing a character? How does this work with C API's that expect zero-terminated strings?

In UTF-8, the answer is no. In other multibyte encodings (e.g. UTF-16), you should not expect to be able to treat it at all like ASCII.

> If you're using UTF-16, what endianness is used? Is it the same as the machine endianness, or fixed? What operations cause endian conversion?

When reading external text, you can detect this from the BOM -- byte order, after all, is why you have a byte order marker. When converting from your internal format to UTF-16, you pick whatever is most convenient.

> Should my C programs handle the possibility that sizeof(char) != 1? Or at least check for this case and spit out a warning or error?

I don't know any popular non-embedded platform on which sizeof(char) != 1. That said, it can't hurt to get it Right.

> What automated tools exist to remove BOM's or change accented characters into regular ones, if other automated tools don't accept Unicode?

In Python, there's a library called "unidecode" which does a pretty good job of punching Unicode text until it turns into ASCII.

[xentronium]

> Should my C programs handle the possibility that sizeof(char) != 1? Or at least check for this case and spit out a warning or error?

sizeof(char) == 1 by definition.

[est]

> Issues like this are why I hate internationalization.

Same here.

The one to blame should be Unicode. It's actually near impractical MULTIcode defined by a committee.

[notJim]

I don't really think you can argue that UCS2 is better than UTF-8, because UCS2 is simply broken. It's not a reliable way to encode unicode characters. It's like arguing that my Ferrari which is currently on fire is a better car than your Lamborghini (which is not.) I mean, they do each have their merits, but a flaming car is not useful to anyone.

I think this is probably semantics, but I just wanted to point that out in case anyone is confused, which would be understandable because this shit is whack.

[dietrichepp]

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing?

UTF-16 solves problems that don't exist.

(Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

[pjscott]

Be careful not to confuse UTF-16 and UCS-2. UTF-16 is a variable-width encoding, so using it doesn't actually make counting anything easier. UCS-2 is fixed-width, and evil. With UCS-2 you can easily count the number of code points, as long as they fall in the BMP (!), but this is not the same as counting graphemes.

[SeoxyS]

It's not a common use case, but I've had to do it. Luckily, it's fairly easy:

    int count_multibytes_encountered(char *text, unsigned long len) {
        int count = 0;
        for (int i = 0; i < len; i++) {
            if ((*(text+i) & 0b10000000) == 0b10000000 &&       // check if it's a multi-char byte
                (~*(text+i) & 0b01000000) == 0b01000000) {      // and check that it's not a leading byte
                count++;
            }
        }
        return count;
    }

[Daniel_Newby]

> Why do you want to count Unicode characters?

Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information.

> Why do you care if it is fast to do so?

Efficient full text search that can ignore decorative combining characters.

[meatmanek]

> Text editing and rendering

Unless you're working entirely in fixed point characters (and you probably aren't, given that even fixed-width systems like terminal emulators use double-wide glyphs sometimes), you need to know the value of each character to know its width. That involves the same linear scan over the string that is required to calculate the number of glyphs in a variable-width encoding.

[andolanra]

If you implement naïve Aho-Corasick text search over one-byte characters, it works without modification on UTF-8 text. It does not ignore combining characters, but UCS-2 also features combining characters (c.f. other comments in this same thread), so no matter what encoding you use, you must first normalize the Unicode text and the search string before you compare for equivalence (or compatibility, which is a looser notion than equality for Unicode code point sequences.)

[masklinn]

> Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information.

Except these parts of the system have to work on unicode glyphs (rendered characters) which will span multiple codepoints anyway, so counting codepoints remains pointless. The only use it has is knowing how much codepoints you have. Yay.

[pjscott]

How does fast character counting help with full-text search?

[Daniel_Newby]

The best search algorithms can skip ahead upon a mismatch. A variable-length encoding requires branch instructions in the inner loop, leading to pipeline flushes and potentially dramatic slow down.

[dietrichepp]

This is incorrect. Searching text with a variable-length encoding does not require extra branch instructions. If you're searching through UTF-8 text, you can just pretend it's a bunch of bytes and search through that.

This isn't counting problems with normalization, of course. You will have to put your needle and haystack both into the same normalization form before searching. But you had to do that anyway.

[est]

> Why do you want to count Unicode characters?

Because, there are other countries which use more than English language?

I fucking hate you ascii-centric ignorant morons sometimes, you know, for example

- display welcome message character by character fro left to right

- Extract the first character because it's always the surname

- catch two non-ascii keyword and find its index in a string

In the first example, should I just put byte by byte, which displays as garbage, and suddenly three bytes become a recognizable character?

[SnowLprd]

Could you not have communicated your examples without the hostility?

[dietrichepp]

I think you misunderstand. I wasn't asking why Unicode characters should be counted instead of bytes or ASCII characters, I was asking why you would even want to count characters at all.

> I fucking hate you ascii-centric ignorant morons

Nice.

> You ignorant, arrogant fuck.

This is why I quit posting under an alias, so I wouldn't be tempted to say such things.

> display welcome message character by character fro left to right

UTF-16/UCS-4/UCS-2 doesn't solve anything here. Counting characters doesn't help. For example, imagine if you try to print Korean character-by-character. You might get some garbage like this:

    ᄋ
    아
    안
    안ᄂ
    안녀
    안녕
    안녕ᄒ
    안녕하
    안녕하ᄉ
    안녕하세
    안녕하세ᄋ
    안녕하세요

Fixed width encodings do not solve this problem, and UTF-8 does not make this problem more difficult. I am honestly curious why you would need to count characters -- at all -- except for posting to Twitter.

Splitting on characters is garbage. (This example was done in Python 3, so everything is properly encoded, and there is no need to use the 'u' prefix. The 'u' prefix is a nop in Python 3. It is only there for Python 2.x compatibility.)

    >>> x
    '안녕하세요'
    >>> x[2:4]
    'ᆫᄂ'

I tried in the Google Chrome console, too:

    > '안녕하세요'.substr(2,2)
    "하세"
    > '안녕하세요'.substr(2,2)
    "ᆫᄂ"

I'm not even leaving the BMP and it's broken! You seem to be blaming encoding issues but I don't have any issues with encoding. It doesn't matter if Chrome uses UCS-2 or Python uses UCS-4 or UCS-2, what's happening here is entirely expected, and it has everything to do with Jamo and nothing to do with encodings.

    >>> a = '안녕하세요'
    >>> b = '안녕하세요'
    # They only look the same
    >>> len(a)
    5
    >>> len(b)
    12
    >>> def p(x):
    ...     return ' '.join(
                'U+{:04X}'.format(ord(c)) for c in x)
    
    >>> print(' '.join('U+{:04X}'.format(ord(c))
              for c in b))
    >>> print(p(a))
    U+C548 U+B155 U+D558 U+C138 U+C694
    >>> print(p(b))
    U+110B U+1161 U+11AB U+1102 U+1167 U+11BC U+1112 U+1161 U+1109 U+1166 U+110B U+116D

See? Expected, broken behavior you get when splitting on character boundaries.

If you think you can split on character boundaries, you are living in an ASCII world. Unicode does not work that way. Don't think that normalization will solve anything either. (Okay, normalization solves some problems. But it is not a panacea. Some languages have grapheme clusters that cannot be precomposed.)

Fixed-width may be faster for splitting on character boundaries, but splitting on character boundaries only works in the ASCII world.

[est]

> Counting characters doesn't help.

Why? If you can count characters (code points) then it's natural that you can split or substring by characters.

Try this in javascript:

    '안녕하세요'.substr(2,2)

Internally Fixed length encoding is much faster than variable-length encoding.

> Unicode does not work that way.

It DOES.

> Splitting on characters is garbage.

You messed up Unicode in Python in so many levels. Those characters you seen in Python console is, actually not Unicode. These are just bytes in sys stdout that happens be to correctly decoded and properly displayed. You should always use the u'' for any kind of characters. '안녕하세요' is WRONG and may lead to unspecified behaviors, it depends on your source code file encoding, intepreter encoding and sys default encoding, if you display them in console it depends on the console encoding, if it's GUI or HTML widget it depends on the GUI widget or content-type encoding.

> I'm not even leaving the BMP and it's broken!

Your unicode-fu is broken. Looks like your example provided identical Korean strings, which might be ICU module in Chrome auto normalized for you.

> You can't split decomposed Korean on character boundaries.

In a broken unicode implementation, like Chrome browser v8 js engine.

> I happen to be using Python 3. It is internally using UCS-4.

For the love of BDFL read this

http://www.python.org/dev/peps/pep-0414/

http://docs.python.org/3/whatsnew/3.3.html

[codeka]

I'm sorry but you're wrong. I suggest you inform yourself better of the subject you're talking about before you call people "ignorant morons" next time.

dietrichepp is talking about Normalized Form D, which is a valid form of Unicode and cannot be counted using codepoints like you're doing.

Maybe you can try:

'𠀋'.substr(0,1)

[est]

yeah sure why not.

    >>> u'𡘓'[0:1]
    u'\U00021613'

    >>> u'Hi, Mr𡘓'[-1]
    u'\U00021613

    >>> u'𠀋'[0:1]
    u'\U0002000b'

Javascript won't work because UCS2 in js engine, duh.

Actually Javascript is messed up with Unicode string and binary strings, that's why Nodejs invented Buffer

http://nodejs.org/api/buffer.html

[cmccabe]

Code points aren't letters.

Consider the following sequence of code points: U+0041 U+0308 [edit: corrected sequence]

That equals this european letter: Ä

Two code points, one letter. MAGIC! You can also get the same-looking letter with a single code point using U+00C4 (unicode likes redundancy).

Not all languages have letters. Not all languages that have letters represent each one with a single code point. Please think twice before calling people "morons."

[est]

> Two code points, one letter.

Yes I under stand there are million ways to display the same shape using various unicode. But how does that make code point counting impossible?

AND if you explictly using COMBINING DIAERESIS instead of single U+00C4, counting diaeresis separately is wrong somehow?

Why don't we make a law stating that both ae and æ is single letter?

[est]

> See? Expected, broken behavior you get when splitting on character boundaries.

Yeah, like your Jamo trick is complex for a native CJK speaker.

Thought Jamo is hard? Check out Ideographic Description Sequence. We have like millions of 偏旁部首笔画 that you can freestyle combine with.

And the fun is the relative length of glypes, 土 and 士 is different, only because one line is longer that the other. How would you distinguish that?

But you know what your problem is?

It's like arguing with you that you think ส็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็็ is only one character.

IMPOSSIBU?!!!???

And because U+202e exists on the lolternet so we deprive your ability to count 99% normal CJK characters???!??!111!

Combination characters is normalized to single character in most cases, and should be countable and indexable separately.

If you type combination characters EXPLICITLY, they will be counted with each combination, naturally, what's wrong with that?

Or else why don't we abandon Unicode, every country deal with their own weird glype composition shit?

[fusiongyro]

By being unnecessarily insulting you're degrading your own position, and your argument is collapsing under the weight of your anger and sarcasm. I see Dietrich and Colin working with definitions of code point, character and glyph that illuminates why counting one way will lead to problems when you slip into thinking you're counting the other way. Then in your much-too-fired-up responses you conflate them again, and muddy us back to square one.

It seems to me you're deriding us for being native speakers of languages with alphabets, and also deriding us for wanting APIs that prevent developers from alphabet-language backgrounds from making the mistakes our assumptions would incline us towards. You're going to have to decide if you're angry because you like the "simplicity" of UTF-16, because we don't speak a CJK language as well as you do (maybe Dietrich or Colin does; I have no idea) or because you're just angry and this is where you've come to blow off steam. If it's the third, I hope you'll try Reddit first next time, since this kind of behavior seems to be a lot more acceptable there than here.

[est]

For fuck's sake I am not defending UTF16's simplicity, I am defending that:

fixed width can count code points (I worded it as "character") faster than variable-length

Then his dietrichepp tries to educate me two code points combined should be treated equaly with another single code point, WTF y u no normalization?

Downvote me as you like, but you can't change the fact that UCS4 is used internally in Unicode systems.

Any reason other than for faster code point counting?

-----------------

dietrichepp also offended me that unicode characters should not count or offset. QTF:

> Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing?

[speleding]

Counting code points in UTF8 can actually be faster than UCS2 in case you do not know the length in advance such as null-terminated strings. The UTF8 encoding is cleverly defined such that to find the number of code points you just need to check the number of bytes that have the top bit cleared. Since UTF8 strings are generally shorter than UCS2 it can be faster in some cases. Either way this is not a serious enough concern to use one encoding over the other.

[threedaymonk]

"Character". You keep saying that word …

    >>> len(u'épicé')
    5
    >>> len(u'épicé')
    7