[nathanmcrae]

True, but verifying proofs is usually easier (or at least more mechanical) than generating proofs. Personally I think this is the best possible use of AI: to have it generate systems that have verifiable properties in some formalized language. That way we can just come up with the properties we want satisfied (which, granted, is not generally trivial) and use the formalized language to verify that the properties are satisfied in the AI-generated code.

[chii]

but then what would stop the AI from running those verification as part of it's operations? Why does it need to be human run?

I think there's some level of halting problem equivalency here imho...

[scrollaway]

Nothing stops that! It just has to be done. I’m working on exactly this: taking GPT output and making sure a second layer of AI can run and verify that output.