[steinroe]

in our specific case, we needed something handwritten for the statement-level parser anyways. And the requirements for the LL parser are very simple: extract individual sql statements from a source input. we just compare the next n tokens with a list of tokens from which any statement starts, which is straightforward to implement with a handwritten LL parser.

so after all, I would say its a decision based on the special requirements we have working around the limitation of libpg_query. I think its also the fastest one, but this was not the main reason.

[epgui]

Thanks for your response!

I'm a little confused by it though... Maybe I am misunderstanding what you're telling me.

I would definitely consider a parser implemented with parser combinators to be "handwritten". The main difference for me is that parser combinators allow you to express a language grammar in a more declarative way, which makes refactoring and correctness verification much simpler. I think what you describe as "straightforward" without combinators would be "completely trivial" with combinators.

I know from reading recent comp sci papers (particularly in the Journal of Functional Programming) that "the parser problem" is not fully solved yet (!) and that there are various theoretical limits to all currently known formal approaches. I suspect that performance could be a limiting factor for certain types of syntaxes, although I'm not sure if Postgres' would be particularly problematic in this regard.

[steinroe]

sorry, I think I misunderstood your question!

after all, we did not implement a "real" parser. we just use libpg_query, the actual Postgres parser, and work around its limitations as good as possible. The implementation thereby required maximum flexibility. we never define any grammar other than "a select statement starts with a SELECT keyword".

[ahoka]

Parser combinations are very powerful! For example your function whitespace_tokens could be something like:

whitespace_tokens = many1(choice(‘ ‘, ‘\t’, ‘\n’)).map(m -> token(m))