[zakisaad]

How do you define "correct" code?

[newsicanuse]

The code that gets stuff done instead of beating around the bush making unxpected errors

[vanuatu]

i suspect this is highly dependent on what you're working on

from my experience if you give the models a way to self-verify correctness they succeed basically 100% of the time

[maccard]

> from my experience if you give the models a way to self-verify correctness they succeed basically 100% of the time

My experience is that if you can get the model to one shot the task, you'll do fine but if it has to iterate it leaves things worse than before and almost always requires human intervention after burning through an enormous amount of tokens