[abraxas]

It seems that something that does away with human friendly syntax and leans more towards a pure AST representation would be even better? Basically a Lisp but with very strict typing might do the trick. And most LLMs are probably trained on lots of Lisps already.

[verdverm]

Generally seems a bad idea to have your LLM write languages you do not understand or write yourself

[catlifeonmars]

Doesn’t that apply to the OP as well?

[verdverm]

Yes, I'm not going to fill my precious context with documentation for a programming language

This seems like a research dead end to me, the fundamentals are not there

[catlifeonmars]

It seems kind of silly that you can’t teach an LLM new tricks though, doesn’t it? This doesn’t sound like an intrinsic limitation and more an artifact of how we produce model weights today.

[verdverm]

getting tricks embedded into the weights is expensive, it doesn't happen in a single pass

they's why we teach them new tricks on the fly (in-context learning) with instruction files

[catlifeonmars]

Right, it sounds like an artificial limitation.