[professoretc]

> Even if it succeeded, it kept changing the output by doing ops which I never mentioned in the prompt (like reordering and merging duplicated values to a single key)

That's something I don't see mentioned enough; if you change the input to a LLM, that may potentially change the probabilities of all the output tokens. Most of us would be surprised if we told a junior developer to fix a bug in a specific module, and they submitted a PR which modified literally every file in the source tree, but that's entirely plausible with a LLM. Asking it to "fix" one thing may change/break completely unrelated things.