Hacker News new | ask | show | jobs
by wolfgarbe 559 days ago
Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.