[bpfrh]

Isn't "code compiles" an insufficient criteria?

e.g you would need to prove that for all inputs the code produces the correct output which would in turn make the problem way more complex

[pressbuttons]

Not if the problem as written is "does this code compile", which is still a useful stepping stone for some workflows. Yours is certainly a more useful query in most cases but repositioning or re-scoping the original question can still lead to a net win.

[logicchains]

It's not a sufficient criteria by itself, but where no better criteria is possible it would still produce better results in reinforcement learning than if the model has no reward for producing correctly compiling code vs code that failed to compile.