[Someone]

Good text, but let’s nitpick. I’m too lazy/confident/arrogant to verify, so let’s potentially embarrass myself here.

    def getToken(self):
        self.skipWhitespace()
        self.skipComment()

How does that handle multiple consecutive comments? Whitespace following a comment?

(returning ‘comment’ and ‘whitespace’ tokens would fix this, and would make it possible to reuse the lexer for pretty-printing/syntax coloring)

[azhenley]

It works because a comment by definition goes until a newline, so you can’t have consecutive comments without a newline token in between.

I’ll look into if there’s a better way to do it than this based on your suggestion, thanks!