Hacker News new | ask | show | jobs
by wooorm 4272 days ago
You are completely right, franc doesn’t state how language are detected. The detection is based on (1) unicode-script usage and (2) trigram-counts. Some scripts are only used by one language. Other scripts, such as Cyrillic, come with many more: those are detected by the top 300 trigrams per their corresponding UDHR (Universal Declaration of Human rights, the most translated document).

Shameless plug, the page clearly stated you can fork franc to support 300+ languages ;)

1 comments

Thanks for the information, don't get me wrong, here I'm not trying to play at who has the biggest. Just that at first without knowing how the data file was generated, I thought the "you can fork to support 300+" languages was a statement like "well provided you find a way to provide the data file for that much languages, which we didn't because doing it is hard/require a huge corpus", but if it's just parsing more UDHR translations, then sure it can easily be forked to reach 300+ languages :)
Yeah, so I’d like to add an easier way to support more, or less, languages through the Node API. Currently, there’s a number (1e6), the amount of speakers of a given language, which is hard-coded in the generation file (I added a link this morning in the statement about forking to the actual line).

If you set that number to 0, or 100,000 and execute `npm prepublish`, your franc supports more languages :) That’s it!