|
|
|
|
|
|
by gliese1337
4048 days ago
|
|
|
Yes. For a given definition of "word", anyway, but "word" is fiddly enough we can assume they mean something sensible. A stricter definition would be "lexers work at the lexical level" for NLP (in which case it's nearly tautologically obvious), or "lexers work at the token level" for programming languages. (Creds: lexing / wordbreaking is the topic of my masters research in linguistics. Tries are a good starting point for that, but it turns out that they are a special case of simple morphological analyzer, and you need more complicated ones to do a good job on natural languages.) |
|
|