That's about as advisable as asking it to write firmware for a pacemaker. Smart contracts are some of the most delicate codebases - even a tiny bug can cause you to lose a lot of money. With a model like this bugs are very likely, especially in such a niche domain.
I agree that at first it seems to be a bad idea, but the more I think about it, the more I think it makes for a great test case to prove the AI is robust enough and that it can be trusted for real life scenarios. It also feels kind of inevitable.
It seems like a safe playground, maybe it will lose a few tokens, but if they are no legal consequences, who cares ? The best place to move fast and break things, and with great reward potential.
If I take some freelancer to write them what's telling me that they are not already using something like codex or copilot. It's like when a factory release its used water in the river nearby. Maybe we shouldn't drink from the river anymore, but it would be better to test the water the factory release to make sure it's OK.
Sure some contracts will have bugs at the beginning, but those faulty contract won't get reused.
In theory you can create a new token per contract to cap the maximum potential loss. Then you increase this maximum potential loss as the contract get used in the open and therefore become more robust.
You probably can write some guarantee fund smart contract to compensate for when bug happen.
Sooner or later we will have to implement a high-level error-correction mechanism to handle the bugs generated by the weak AI : Like a paradigm shift where you expect bugs to happen and handle them instead of expecting no bugs.
That has got to be one of the worst possible use cases one could imagine. In page 33 of the appendix, the authors note that nearly 40% of RSA encryption keys created by Codex are clearly insecure.
As a contract user, I'd probably have more trust in a contract written by an independent AI from a short natural language specification which can't hide intent, than a contract with hidden backdoor, or a subtle bug.
Also the AI will probably improve with usage.
You probably can generate multiple version of your contract, and maybe a high level bug correction scheme like taking the median action between those version can increase bug robustness and find those edge cases when action differ.