[joshka]

I've found similar success in using this approach rather than going direct to "write me some code AI-code-monkey". You can also take the following (which are the same sorts of things you'd want to do as a programmer writing this code yourself:

- Rewrite the problem as a list of constraints

- Point out any ambiguous or missing constraints

- Write an example that demonstrates each constraint

etc.

Effectively this article is really asking the question when posing a novel problem to the LLM, how deep does the internal state go in producing the completions. When it doesn't go deep enough, the trick is to make it do the things that deepen the internal state.

I'd guess that supplying a good system message to GPT-4 (waiting for my access) would help. Something like:

You're an engineer responsible for writing correct code from a specification. Break down the specification into small chunks that can be explained simply. If there are ambiguities, seek clarification. Only write code once all ambiguities are resolved and each part of the problem is described simply.