[iLemming]

> LLMs to be bad at balancing parenthesis

Because you treating Lisp just like any other (non-homoiconic) PL. Give an agent a true Lisp REPL to mess around, and you'd be surprised. Things get very interesting. I still don't understand why more people don't do that - isn't that obvious first thing anyone should figure out? Like I can't even imagine working with Lisp without a REPL and structural editing - I'd immediately fail at balancing parens. Why do you expect a [dumber] machine would do any better?

[klibertp]

Yeah. In my GToolkit setup, I started with giving it eval. At first, it tried to "rebuild the world" for each task, but we gradually settled on a set of images for specific tasks that can be rebuilt as a separate step and are otherwise cached. That gave us fast eval, and it kinda snowballed from there. I thought I would need to implement some IPC into a live image, but the startup is fast enough that it doesn't matter too much. The agent now has both the textual source (in Tonel) on disk and can easily query a live image via CLI.

What do you think about making the agent write type annotations? There are built-in forms in CL, and in Smalltalk, I settled for pragmas for now. They are not checked, but since I started using them, I think the rate of one- or two-shotting solutions has gone up.