[hyperthesis]

Are they actually running the code, and evaluating the output? Or is it debug-by-code-review?

Beware of bugs in the above code; I have only proved it correct, not tried it. - Knuth

[cs702]

They're doing both. Quoting from Figure 1, "the model first generates new code, then the code is executed and the model explains the code. The code explanation along with the execution results constitute the feedback message, which is then sent back to the model to perform more debugging steps. When unit tests are not available, the feedback can be purely based on code explanation."

[hyperthesis]

So only evaluating output with unit tests - a fitness function. AITDD.