Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
They tested them on formal languages of different power, and saw where they could generalize beyond the training data. Transformers failed to generalize pretty early on at stack machines/brace matching.
It was a good bit older of a paper though, if I remember it's somewhat expected from the pure feed forward nature of them and limited circuit depth, where LSTMs have some recurrence.