[jahewson]

Lexers work at the lexeme level, which are generally understood to be words, assuming that a word is a non-decomposable unit of language.

[agumonkey]

Similar to the atom / compound forms described by McCarthy.