Hacker News new | ask | show | jobs
by runeks 3297 days ago
I'm not familiar with UTF-8, which -- I believe -- is what Data.Text is used to represent. Do all UTF-8 strings have a well-defined length?

A list of ASCI chars obviously does (the length of the list), but I'm not sure about Data.Text.Text.
3 comments
rnhmjoj 3297 days ago
Yes, I'm sure the length of a UTF-8 string is something tricky but Data.Text has a length function nonetheless. The APIs are very similar if not identical. You can see it here:

https://hackage.haskell.org/package/text-1.2.2.2/docs/Data-T...

https://hackage.haskell.org/package/bytestring-0.10.8.1/docs...

link
HelloNurse 3297 days ago
Of course, a real string type would store length along with the characters, making the length function trivial and fast.
link
leshow 3296 days ago
It's hard to say what length should be because it could mean different things. How many u8 values there are? How many graphemes are present?

If it's just counting how many u8's in a slice, that's trivial. But storing the total number of variable-length graphemes isn't. I assume that's why Data.Text's length function is O(n), and other languages that have UTF-16 or UTF-8 strings also have length functions that are linear.

link
HelloNurse 3296 days ago
Graphemes? Strings can be expected to be transformed so that they contain the "right" type of characters (in particular, plain letters followed by the respective combining diacritical marks vs a smaller number of precomposed accented letters) before asking about their length, which is how many characters they contain. If you want to count "graphemes" you can have a different function and possibly a separate cached result.

This in theory. It is readily apparent from the Data.Text page at https://hackage.haskell.org/package/text-1.2.2.2/docs/Data-T... that the authors care about performance only selectively ("fusion" of buffer allocations is fun, relatively messy data structures to cache important data such as string length are not fun), and that they don't care enough about Unicode to add serious string-level abstractions over the Unicode tables exposed in the Char type.

link
e12e 3296 days ago
Well, how long is the sequence "a<Backspace>". It's two characters and one, two or zero letters?
link
rnhmjoj 3296 days ago
This has been covered a few times before. See this for example: https://mathiasbynens.be/notes/javascript-unicode

Data.Text seems to recognize symbols made of several codepoints (like emoji) as one but still counts diacritics and combining characters as different symbols.

link
leshow 3296 days ago
Data.Text is represented as a UTF-16 vector
link