[kevindamm]

I wonder how much of this is syntactic familiarity (from training) and how much of this is needing to attend to balanced parentheses.

I don't use lisp often enough to have played with getting GPT to lisp with me, but I have played a bit with getting it to read and write Datalog (which I suspect is even more scarce in The Pile dumps). It's ok at recognition but misses details. I haven't seen it produce much of value yet. But it can write JavaScript for days, and has no problem balancing parentheses and brackets there, even without compiler/tree-sitter support.

If I had spare experimenting bandwidth I would look into whether fine-tuning for Lisp format and conventions would show a significant boost in performance..

[nathan_compton]

I find its Lisp generation to not just be syntactically wrong but conceptually inadequate. It's inability to generalize across languages is one of the big reasons I'm skeptical about language models' general intelligence.