Hacker News new | ask | show | jobs
by dvt 20 days ago
Who implements transliteration rules? I assume operating systems? Or text renderers?
1 comments
kccqzy 20 days ago
The ICU library. It’s so ubiquitous that it’s somewhat difficult to avoid this library if you are doing anything advanced with human text.
link
ks2048 20 days ago
And ICU uses data from CLDR, which is mentioned in the blog. Here, there are 380 xml files: https://github.com/unicode-org/cldr/tree/main/common/transfo...

Yes, ICU is ubiquitous. But, some NLP projects use various other libraries, such as uroman (just for romanization - to Latin script).

link
poizan42 20 days ago
Windows has its own library built-in: NLS. There have been a lot of back-and-forth about whether to use NLS or ICU in dotnet on Windows because it resulted in different behavior between Windows and other platforms. I think they ended up using ICU by default from .NET 5 on.
link