| I think you misunderstand. I wasn't asking why Unicode characters should be counted instead of bytes or ASCII characters, I was asking why you would even want to count characters at all. > I fucking hate you ascii-centric ignorant morons Nice. > You ignorant, arrogant fuck. This is why I quit posting under an alias, so I wouldn't be tempted to say such things. > display welcome message character by character fro left to right UTF-16/UCS-4/UCS-2 doesn't solve anything here. Counting characters doesn't help. For example, imagine if you try to print Korean character-by-character. You might get some garbage like this: ᄋ
아
안
안ᄂ
안녀
안녕
안녕ᄒ
안녕하
안녕하ᄉ
안녕하세
안녕하세ᄋ
안녕하세요
Fixed width encodings do not solve this problem, and UTF-8 does not make this problem more difficult. I am honestly curious why you would need to count characters -- at all -- except for posting to Twitter.Splitting on characters is garbage. (This example was done in Python 3, so everything is properly encoded, and there is no need to use the 'u' prefix. The 'u' prefix is a nop in Python 3. It is only there for Python 2.x compatibility.) >>> x
'안녕하세요'
>>> x[2:4]
'ᆫᄂ'
I tried in the Google Chrome console, too: > '안녕하세요'.substr(2,2)
"하세"
> '안녕하세요'.substr(2,2)
"ᆫᄂ"
I'm not even leaving the BMP and it's broken! You seem to be blaming encoding issues but I don't have any issues with encoding. It doesn't matter if Chrome uses UCS-2 or Python uses UCS-4 or UCS-2, what's happening here is entirely expected, and it has everything to do with Jamo and nothing to do with encodings. >>> a = '안녕하세요'
>>> b = '안녕하세요'
# They only look the same
>>> len(a)
5
>>> len(b)
12
>>> def p(x):
... return ' '.join(
'U+{:04X}'.format(ord(c)) for c in x)
>>> print(' '.join('U+{:04X}'.format(ord(c))
for c in b))
>>> print(p(a))
U+C548 U+B155 U+D558 U+C138 U+C694
>>> print(p(b))
U+110B U+1161 U+11AB U+1102 U+1167 U+11BC U+1112 U+1161 U+1109 U+1166 U+110B U+116D
See? Expected, broken behavior you get when splitting on character boundaries.If you think you can split on character boundaries, you are living in an ASCII world. Unicode does not work that way. Don't think that normalization will solve anything either. (Okay, normalization solves some problems. But it is not a panacea. Some languages have grapheme clusters that cannot be precomposed.) Fixed-width may be faster for splitting on character boundaries, but splitting on character boundaries only works in the ASCII world. |
Why? If you can count characters (code points) then it's natural that you can split or substring by characters.
Try this in javascript:
Internally Fixed length encoding is much faster than variable-length encoding.> Unicode does not work that way.
It DOES.
> Splitting on characters is garbage.
You messed up Unicode in Python in so many levels. Those characters you seen in Python console is, actually not Unicode. These are just bytes in sys stdout that happens be to correctly decoded and properly displayed. You should always use the u'' for any kind of characters. '안녕하세요' is WRONG and may lead to unspecified behaviors, it depends on your source code file encoding, intepreter encoding and sys default encoding, if you display them in console it depends on the console encoding, if it's GUI or HTML widget it depends on the GUI widget or content-type encoding.
> I'm not even leaving the BMP and it's broken!
Your unicode-fu is broken. Looks like your example provided identical Korean strings, which might be ICU module in Chrome auto normalized for you.
> You can't split decomposed Korean on character boundaries.
In a broken unicode implementation, like Chrome browser v8 js engine.
> I happen to be using Python 3. It is internally using UCS-4.
For the love of BDFL read this
http://www.python.org/dev/peps/pep-0414/
http://docs.python.org/3/whatsnew/3.3.html