This is strange to me. This is clearly meant, in unicode, to be 'G' that we all know and love. It has uselessly expanded "the alphabet" (to be western-centric) in a confusable way.
Unicode maybe should have been three dimensional, with "concept of G" in the 2D space, and "ways of representing G" behind G, along the third axis. All ways of representing G, whether little capital, capital, lower case, would or at least could equate to conceptual G in the 2D space.
> Unicode maybe should have been three dimensional, with "concept of G" in the 2D space, and "ways of representing G" behind G, along the third axis. All ways of representing G, whether little capital, capital, lower case, would or at least could equate to conceptual G in the 2D space.
It brings up interesting, long-standing problems. Which of these count as the same letters?
* Letters in two languages with the same appearance and making the same phonetic sound
* Letters in two languages with the same appearance but making slightly different phonetic sounds. E.g., R in English and French
* Letters in in two languages that are otherwise the same, but one has an accent. Is the accent part of the letter? Separate? Are they really the same letter?
* Letters in two languages with the same appearance but making completely different phonetic sounds.
* Similar (by any property) letters in two related languages; e.g., both Indo-European
* Similar (by any property) letters in two unrelated languages; e.g., French and Vietnamese.
* Letters with the same phonetic sound but different appearances.
* Letters with the same appearance, one is phonetic and one an ideograph
* Letters that are otherwise identical, but alphabetize differently in their respective languages
* EDIT: Forgot a key one; Letters that are otherwise identical, but follow different rules of how they combine with the letters around them (a common issue, though not familiar to English speakers).
* Letters that are in all ways identical but belong in different languages. In which languages code group does the letter belong? One? Both? What if the subset of Unicode supported by an application includes one language but not the other?
> Letters in in two languages that are otherwise the same, but one has an accent. Is the accent part of the letter? Separate? Are they really the same letter?
It gets worse than this. Example: the letters Ä and Ö exist in both Swedish and German (as an example).
In German they are actually counted as the letters A and O with diaereses above them, and they alphabetize together with other instances of the letters A and O, because that's what they are.
In Swedish those are their own letters, which are completely separate from the letters A and O. They get their own place in the alphabet (second-to last and last, respectively), and replacing them with AE and OE is technically not acceptable in Swedish like it is in German (though it's often done anyway, e.g. on airline tickets).
And in Unicode they are represented by the same code-point even though in one language it is a letter, and in the other language it's only a variation on another letter. What a mess.
That character is from the phonetic alphabet so it's not the "concept of G", it's the concept of a "voiced uvular stop", which happens to looks visually like G. So what Unicode is doing is separating two conceptually different ideas, exactly as intended.
The cases where Unicode has taken similar looking characters and combined them into one have not been successful, Han Unification for example was widely viewed as a misstep and has caused many problems, such as making it impossible to embed certain Japanese characters in Chinese text without higher-level markup.
As mentioned by others on this thread, the real issue is not with Unicode per se, but rather with the ways that web browsers handle it (or fail to handle it, as the case may be).
I think it is very much an issue in Unicode that they did not define the NFKD of ɢ to be G. As far as I can tell, the rationale is that ɢ is semantically different because it is used in IPA. I find that pretty weak, considering the ubiquity of smallcaps. Asking browsers to diverge (as far as equivalence goes) from Unicode standards sounds a lot like a failure of Unicode.
That's a really interesting proposal, but I'm afraid it would be difficult to implement in practice. If this third dimension were actually encoded into the number that represents each character, you'd end up with a lot of wasted bits (since most characters probably wouldn't even need the 3rd dimension, or at least as much of it as the heaviest users.) Another option would be to supplement the metadata that already accompanies Unicode characters (which block it is in, the name of the character/block, etc...) This could be done in practice now, but the information would almost certainly just be ignored if it needed to be looked up in a supplemental table. Furthermore, it's difficult to agree on just about anything in Unicode, and classifying all the characters based on concept seems like a Herculean task for a slow-moving body.
This already exists in Unicode, it's called "Variation Selectors" and they have their own block and are used to select emoji skin tones amongst other things.
But it would be wrong to use them in this case because an IPA G and the letter G are semantically different things and should not be unified into a single character just because they look similar.
Some time ago I registered http://www.goolge.io/. Still haven't done anything with it, I guess at some point I'll just redirect it to duckduckgo. [EDIT: now it's redirected to duckduckgo.]
This can of course be used in a malicious way. I thought about rebuilding the homepage of the bank Credit Suisse on www.credit-siusse.ch, but that's probably illegal.
Nope. The browser is behaving sensibly, since you can't register that domain. It's applying the same rules that the registrars do.
ICANN require that registries follow RFC3491 and related RFCs for name prep before allowing a name to be registered https://www.icann.org/resources/unthemed-pages/idn-guideline... . What that one does is (among other things) NFKC normalization and case-folding:
Interesting. So, out of interest, why is the same not being applied for ɢ? (When I ran it through Python's unidecode I got the roman symbol all the same).
Because 'small capital g' doesn't have a compatibility decomposition to G, but wide letter P does have a compatibility decomposition to 'normal' P. Unicode normalization kills large classes of homograph attacks but by no means all. conventions over mixing scripts from different languages stop some more, but there's no single answer.
The problem is that the RFCs aren't restrictive enough, partly because the IETF doesn't have much authority over registrars. The domain name rules really ought to be something like "one script, plus numbers, in a domain name part". But this runs into such things as the tendency in Japan to mix kanjii with English words. Then there's the whole right-to-left mark business, which has to coexist with left-to-right TLDs.
No. When you apply NFKC normalization to that string, you get just 'paypal', so Paypal have already registered the result. You can try that here: http://mct.verisign-grs.com/ - notice how the output is not the same as some online converters based on punycode.js, because that doesn't have nameprep support https://github.com/bestiejs/punycode.js/issues/40
That looks a lot like the sort of trademark use that authorities have deemed infringing, I'd expect the registrar to "recover" that unless you've got a clear explanation (like Goolge is your name, even then ... (remember Nissan, Mike Rowe, etc.)).
I registered http://www.4ppl3.com a while back. No potential for abuse really, but I just thought it was fun to have a l33t-speak version of the domain name of one of the world's most litigious companies.
That said, I haven't done anything with it, and I'm not a domain squatter, so if anyone wants it I can hook you up!
l33t-speak is so far off my radar that I was wondering if there was some hot startup called '4 people 3' or somesuch. I doubt anyone at Apple remotely cares about l33t-speak from a branding perspective.
This would be a great solution. Allowing unicode characters in domain names is just inviting trouble. I understand that people with non-Latin scripts want domain names in their own language and alphabet, but there are way too many unicode characters that will confuse people about legitimate-looking domain names.
Showing non-ascii in red would be an easy solution for everybody.
It seems like a reasonable compromise would be to allow domain names in non-Latin languages as long as the entire name is in the character set of a specific language. So, if your name is in English, that's fine. If it's in, say, Cyrillic, that's fine too. But if you mix English and Cyrillic characters, that's not allowed. It wouldn't necessarily eliminate all name look-alikes, but it would get rid of most of them.
That's supposed to be one of the rules at the registrar level, but it's one that gets ignored in practice.
I have heard proposals that mixed-script IDNs get converted to punycode in URL display, but I don't know if any browser has fully implemented that yet.
Wouldn't a good compromise to be to somehow highlight any characters that are outside of ASCII and the character set of the language you are using in the browser/os?
Use blue, the most common types of color blindness are red-green issues(there's a really tiny percentage that doesn't perceive colors at all but really really tiny. And other than those, nobody has trouble with blue)
Source: I'm colorblind(protanope) and red would definitely be an issue. Android studio, for example, is really annoying for me because the particular red they use for errors is very hard to distinguish from black
That kind of violates the intuitive "red/orange/yellow is alarm, blue/green/black is expected" notion though. What you could do is put ASCII characters in blue and non-ASCII characters in red.
Isn't this a general argument against ever using red as a warning? Seems to prove too much.
Especially in this case, where there is unlikely to be a specialized class of scammers who go phishing only for people with red-green colorblindness. So long as browsers implement a feature that stops the phishing in 99% of cases, the scammers will try something else.
It's an argument against using red as the only warning sign.
Compare to Chrome's https indicator- it turns the "https://" part of the URL green (which I can barely distinguish as different, so it is useless to me) and adds a padlock icon.
Colorblind-friendly graphs might use both color and symbols to distinguish elements.
> people with non-Latin scripts want domain names in their own language
I've yet to see a useful site with Cyrillic domain. Theoretically it sounds good, but practically everyone still uses Latin domains. May be it'll change with time.
Except by a third of all people who live in China and India. Not everyone speaks a language that is representable in the latin alphabet. In fact, a very large percentage of people do not.
And it is then worth noting that as it stands, the attitudes of western developers with respect to text input and name lookup has so horribly screwed the Chinese with respect to domain names that they started using numbers instead of letters for their major web properties.
I live in India. I have never seen a non-Latin alphabet domain, except when I opened <some hindi word>.<tld> and <poop emoji>.com just out of curiosity. Could you show me some non Latin alphabet domain names that are used?
I am not claiming that everyone speaks a language that is representable in the Latin alphabet.
China and India don't pose a problem since Pinyin uses standard ASCII characters and neither Chinese characters nor Brahmic scripts have any symbols that resemble ASCII characters.
For the same reason that my email client occasionally tells me "this may be a scam," even though sometimes it's not and I act accordingly. Based on whatever criteria it's using, the data received has a somewhat higher chance of being illegitimate.
We as (technical) humans can recognize (hence this discussion) that the use of this uncommon G is meant to mislead you into thinking you're going to Google, when in fact you're going to Hell. I'd like to be warned of that possibility.
In this case, the extremely oversimplified algorithm might be "does the domain, as filtered down to canonical characters, represent one of the top five destination domains, yet go somewhere else if not canonicalized?"
Chinese people will be fine since all Chinese URLs are either ASCII compliant or use Chinese characters, which can't be confused with any ASCII characters.
To my understanding the unicode standard encodes an ASCII transliteration of an Unicode symbol to itself, but what about typographical similances? Wouldn't that be a hard problem? Perhaps there are two unicode characters that look exactly the same (using a given typeface) but have different transliterations. Or vice versa - two totally different looking characters share transliterations and gave false alarms.
It's not a great solution since it requires knowing the difference between ASCII and Unicode... I would argue that a user who is vulnerable to falling for unicode characters in domain names won't have that knowledge.
No reason for me to try to pick out specific characters. I won't notice. Plus, it won't work for zero width characters, and I might miss it for really tiny ones.
Give me a popup warning explaining the problem when I try to visit the site, same as I get for certificate problems.
That would be very confusing for multilingual users. Just because my OS is configured to use a certain locale, doesn't mean I don't read text in scripts not considered part of it.
They are of course, but if you use that list instead of a single locale, you end up with a solution that only highlights 'strange' characters when they are not part of your language/locale set. So for someone who speaks only Latin character based languages you could highlight all Cyrillic characters, but for someone who speaks Russian you still have the original problem (it's not as if you can just highlight all Latin characters in their case!).
The characters are from the Latin character set, but non-ASCII. Highlighting the Å in red would look pretty confusing. And in many countries you want the entire domain name written in non-ASCII characters, depending on the language. E.g. websites in Russia, China, India, etc...
This is something that's been bugging me for years.
Why are there multiple representations of alphabet characters in Unicode? It seems reasonable to include accent marks, but what's the benefit in having a Cyrillic 'o' alongside a standard 'o' or the 2 or 3 other ASCII-lookalike sets of characters?
The most important reason is semantics. If "O" and "0" look alike in a certain font, should we use the same character code for both? No, because they have different meaning.
Here are some contexts in which this semantic difference is important: search (compare search results for "cop" and "сор"), alphabetical sorting, text-to-speech, spellchecking, case conversion ("ATOM" -> "atom", but "АТОМ" -> "атом", note the difference between t-т and m-м).
There will never be agreement what's the set of distinct characters (also, what characters should be included, bitcoin logo, facebook logo?)). I see Unicode as a necessary evil. Due to its complexity most applications should treat Unicode text as black boxes.
I never rely on Unicode for computation. When receiving Unicode I always make sure it's in the ASCII range. It could be argued that there should never have been Unicode domain names but I guess Western people are very lucky that ASCII includes most of their characters...
> When receiving Unicode I always make sure it's in the ASCII range. [...] Western people are very lucky that ASCII includes most of their characters...
Please don't spread the myth of Western languages being encodable in ASCII, and don't pretend to support Unicode (or anything else than English) if you filter everything to ASCII.
The _only_ Western language that is encodable in ASCII is English.
Corollary: English is the only language that can be encoded in ASCII.
The other western languages have endless issues with text being encoded/stripped down to ASCII. e.g. French, Spanish, Portuguese, German...
As a german I can attest that I can very well converse (e.g Email) in ASCII. Although it's convenient to use Umlauts, which I do. And I also agree that French or Spanish might be less convenient.
But that was not my point. The point was about identifiers, such as DNS names.
One goal of Unicode has been lossless round-tripping between legacy encodings (to encourage adoption). If such an encoding contains both Latin and Cyrillic, they must be separate to enable that.
The (seemingly obstinate) answer is that they are different characters. The Russian H sounds like an N in English.
If you're transcribing a conversation at the UN and there is a mix of different languages the fact that "Het" is transcribed as a latin character set is information. Het may be a southern American group of people, or it could just be a Russian dude saying "no", even if it looks the same.
I understand that we're still burdened by intralanguage homonyms, but I appreciate the fact that it isn't complicated further.
I got tons of referral header spam (that shows up in e.g. Google Analytics) for all sorts of social media buttons and EU cookie law scare tactic sites. And then there was Vitaly who just spammed me with ilovevitaly.com, which if I recall correctly actually was a site about himself at the time.
Unicode URLs are the devil. Too many indistinguishable characters. URLs should stay full ASCII imho. And I say that as someone whose language requires non-ASCII symbols.
Or, in Bruce Schneier's words: "Unicode is just too complex to ever be secure."
But think about the poor underrepresented folks using foreign character sets?
You really need to support this 'sub café {} café()' => Undefined subroutine café in your friendly and social programming language, otherwise you will be accused of discrimination.
Of course the two é are not normalized.
Which unicode-friendly language does really check for mixed script confusables? Java only is my guess. Even perl6 falls into this trap.
When it is just accents, it's ok. But when your users have a language that uses à radically different alphabet, sometimes they can't even read Latin transliterations.
agree. but then you need to declare your exoting encoding somehow, such as in perl via use encoding 'greek'; and then the parser does not need to guess about mixed scripts encodings on every identifier.
there's only latin and greek valid, everything else invalid.
how many languages even check for mixed script confusables?
for dynamic languages this check is much too expensive, but they are leading the "good cause", allowing everything, and checking nothing.
I haven't used slack, but I think both are doing the best practices around there: Chrome copies the punycoded URL to clipboard, Slack will decode pasted punycode-URLs into a nicer presentation.
There has been talk at the IETF of redefining IDNA2008 (the current way you prevent issues like this) in terms of the PRECIS framework (RFC 7564). This wouldn't exactly "solve" the problem, but it would mean that IDNA could be more agile with respect to Unicode versions and would make it easier to react to new problems, new confusable characters, etc. as they happen.
I frequently have to deal with lots of scientific, mathematical, and many other unusual characters.
I use http://unicode-table.com to help figure out what's what. The official Unicode specifications[1] is impenetrable, and it's really hard to deal with.
I'm not sure how feasible this is, but wouldn't it make sense for .com/.net/etc to be latin alphabet only and allow other domains to be localized with unicode? I wouldn't really have a problem with 新浪首页.cn, and I doubt I would confuse ɢoogle.ru or whatever with google.com
Doesn't that yield a whole class of problems though that we're trying to solve with obtuse solutions such as "let's make that character set in red so people don't get phished"? How is that any more international and/or easy to use?
It seems that putting the allowed character set into the tld would be a pretty user-friendly way of doing that.
Edit: As an added bonus, tlds are centrally managed, and are already western/latin encoded. So why not customize it with a localized abbreviation for the language or tld type?
Cool! I want a cool non-alpha unicode domain. I guess "square-root" is already taken, but there must be some cool domains left (even though nobody can actually type them in).
Actually, some of these would probably be nice aliases for some math / science oriented sites.
I'm curious, why is it allowed to register domain names with mixed character sets? I am behind allowing Unicode characters in domain names for the obvious reasons, but are there compelling use cases for allowing them to be mixed?
Technically, Unicode is only one character set. If you want to disallow mixing, you have to disallow it on some other basis, like script. There are many edge cases to consider, though, and many legitimate reasons to mix scripts.
Maybe browser should have a security option to whitelist characters in URL. When a URL uses another character, there would be popups with explanations and choices.
TR36 bidi spoofs are usually worse than TR39 confusables. Move over with your cursor over it. http://www.unicode.org/reports/tr36/#Bidirectional_Text_Spoo...
That's why browsers or dns tools use libidn, just programming languages not.