[microtherion]

I agree. The tokenizer in the article completely misses the point of how Forth works: the tokenization is supposed to be driven by the words themselves, i.e. ." is looking for a " delimiter, ( is looking for a ).

Not to mention that the rest of the compiler also misses the point of how Forth works. This compiles a fixed subset of Forth and entirely misses out on the extensibility of the language.