Hacker News new | ask | show | jobs
by journalctl 2476 days ago
Sorry, I misread. Indeed, actually tokenizing text is accomplished with regular expressions (although some parsers don’t need a tokenization pass, but details :).