[xpct]

I don't think we're approaching the limit of deterministic prompt -> source code mapping any time soon. Small variability in prompts produces medium variability in outputs. Building on previous outputs only extends the variability, where behavioral differences start appearing a few prompts down the line.

And the AI companies aren't focused on making this process any more deterministic, where repeated prompts are sampled using different seeds. I don't know if it's because the randomness makes the models explore a larger space and perform better, but it's probably intentionally done because it'd be quite easy to remove the stochasticity.

[Frost1x]

There’s another issue where models and transparent wrappers around models that get exposed are shifting around often. Versioning is highly questionable, and not all closed models will be supported indefinitely… so determinism becomes highly questionable at a purely prompt level.

[xpct]

Oh, I didn't even consider this. Now that you mentioned it, I wouldn't be surprised if the stochasticity is intentional so users don't treat it as deterministic, and the company is allowed to continuously update the model (or even a small part of its weights, perhaps to fix/censor something?).