Hacker News new | ask | show | jobs
by stavros 330 days ago
You're actually applying a grammar to the token. If you're outputting, for example, JSON, you know what characters are valid next (because of the grammar), so you just filter out the tokens that don't fit the grammar.
1 comments
joshred 316 days ago
How do you get the alternate tokens?
link