[anonymoushn]

I hope lexers aren't only for parsing regular languages! I definitely want my lexer to be able to parse balanced parens.

[nandemo]

Interesting. You mean you use a language where "sequence of balanced parens" is a token?

[anonymoushn]

No, I use many languages that care about whether parens (each one of which is its own token) are balanced. The language of balanced parens (a language that includes "" and "((((()()(()))())))" but not "()())(()" or "(((()))))") is a simpler language that also cares about parens being balanced. It was also the first language I saw in compilers class that was not regular.

[nandemo]

Then you don't need the lexer to parse balanced parens. The job of the lexer is to turn

    ("Foo(" + bar)

into something like

    OpenParen String Op Identifier CloseParen

Then the (synyactical) parser takes over. This is pretty standard.