Hacker News new | ask | show | jobs
by ilovetux 298 days ago
Disclaimer, I just pulled this quote from Google ai which probably took it from somewhere else, but I just wanted to provide a little context. ASCII encoded text is also valid utf8.

> The first 128 characters of Unicode, which are the same as the ASCII character set (characters 0-127), are encoded in UTF-8 using a single byte with the exact same binary value as their ASCII representation. This means that any file containing only ASCII characters is also a valid UTF-8 file
2 comments
staplung 298 days ago
Yes, but the box drawing characters in "ASCII" are all above 127 so they don't encode the same way. So that last AI generated sentence is basically false (or really misleading): ASCII files that consist only of characters in the lower 127 will also be valid UTF-8. But ASCII files that use characters above 127 will not be valid UTF-8.

Now, technically, ASCII only concerns the lower 127 characters. There's no single standard definition as to what the upper half of the byte space represents in ASCII itself so technically it's true that all valid ASCII files are valid UTF-8. By the same logic however, the box drawing characters are not ASCII. They're actually part of something called code page 437, which maps those bit patterns to box drawing characters. With other code pages they map to something else, often non-Latin characters or ones with accents.

So, the name ASCII flow is misleading and the the output options are too. ;-) Basically, if the high bit is set in UTF-8 it indicates that more than one byte is needed to represent the code point.

link
ilovetux 298 days ago
Granted, all of that is true, but GP specifically differentiated between ASCII and ASCII Extended, then GP went on to say that after choosing the ASCII option and pasting the text in a text editor on Mac it was reported as UTF-8, which I was pointing out would be true because if the ASCII option is chosen as opposed to the ASCII Extended option then what he ends up with (ASCII) is valid UTF-8 as reported by the text editor.
link
em3rgent0rdr 298 days ago
Indeed, UTF-8 "was designed for backward compatibility with ASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file."

https://en.wikipedia.org/wiki/UTF-8

link