[abeppu]

I'm still surprised by the approach. I mean, great that it works this well -- but program synthesis is one of those rare domains where you can observe exactly what the outcome is after you generate something. You can see execution traces, variable values, what the JIT produced, etc. And all of this is relatively cheap -- often executing a code snippet should be far cheaper than an extra pass through a giant DNN right? So it's fascinating to me that they train entirely from dealing with code as text.

Imagine learning to develop recipes, not by ever cooking or eating or even seeing food, but only reading a giant library of cookbooks. Or learning to compose music but never hearing or playing anything -- only seeing scores.

[wantsanagent]

FWIW execution guided code synthesis is a thing. Get a few possible outputs and ditch those that don't pass a parser as an example. At least in the SQL generation realm this is well worth the time it takes to tack onto a large language model.