[xtagon]

A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?

[throwaway09223]

Yes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake.

Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.

[jhugo]

My experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?

[kubrickslair]

It does it pretty well for Python with code interpreter, especially for quant/ ML heavy code.

[tomjakubowski]

I wonder if anyone has done a feedback loop with GPT and compiler errors to train this sort of thing

[aldarisbm]

they have, with langchain