[zingar]

Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write.

Yet I still had the idea that LLMs should be better at lisp than other languages.

A fascinating contradiction, thanks for pointing this out.

[ijk]

Yeah, unfortunately "LLMs are bad at counting" seems to apply to counting parentheses.

With modern tool calling I wonder if a better way to go about it is for the LLM to express the program changes as a function or otherwise use an editor that auto-balances parens. There's a lot of relatively simple tooling that makes it easier to write in a Lisp. The languages tend to lend themselves to being straightforward to check like that.

What's special about Lisp's repl is that it's perfectly possible to construct your entire program in the repl, testing each addition live as you write it. (Many Lisp-focused editors assume you'll want to do this, such as Emacs making it easy to run the interpreter on a single function in a file.) That tooling is lost if you just try to one-shot the file, and before 2026 the majority of LLMs originally just tried to one-shot every file.

But, just like a lot of early LLMs had huge problems with whitespace and numbers because the tokenization was taking efficiency shortcuts that made sense for text but absolutely wrecked code syntaxe, I wonder if the current optimizations are badly formatted for Lisp.

At the very least, using a varient like Clojure that also uses [] and {} in addition to () might help.

[cma]

This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better:

Neural Networks and the Chomsky Hierarchy

https://arxiv.org/abs/2207.02098

[galaxyLogic]

Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?

[cma]

They tested them on formal languages of different power, and saw where they could generalize beyond the training data. Transformers failed to generalize pretty early on at stack machines/brace matching.

It was a good bit older of a paper though, if I remember it's somewhat expected from the pure feed forward nature of them and limited circuit depth, where LSTMs have some recurrence.

[Zambyte]

I believe the problem stems from ) and )) both being one token wide. I wonder if repeated parentheses being removed from the token vocabulary would improve accuracy for Lisp code generation.