[afavour]

> If we consider the prompts and LLM inputs to be the new source code, I want to see some assurance we get the same results every time.

There’s a related issue that gives me deep concern: if LLMs are the new programming languages we don’t even own the compilers. They can be taken from us at any time.

New models come out constantly and over time companies will phase out older ones. These newer models will be better, sure, but their outputs will be different. And who knows what edge cases we’ll run into when being forced to upgrade models?

(and that’s putting aside what an enormous step back it would be to rent a compiler rather than own one for free)

[devsda]

> New models come out constantly and over time companies will phase out older ones. These newer models will be better, sure, but their outputs will be different.

IIUC, same model with same seed and other parameters is not guaranteed to produce the same output.

If anyone is imagining a future where your "source" git repo is just a bunch of highly detailed prompt files and "compilation" just needs an extra LLM code generator, they are signing up for disappointment.

[carlmr]

>IIUC, same model with same seed and other parameters is not guaranteed to produce the same output.

Models are so large that random bit flips make such guarantees impossible with current computing technology:

https://aclanthology.org/2025.emnlp-main.528.pdf

[nerdsniper]

Presumably, open models will work almost, but not quite, as well and you can store those on your local drive and spin them up in rented GPUs.