[scheme271]

> Every commit is reproducible. The prompt is preserved with

> some extra attributes about which model and agent was used.

> You can re-run any commit against a fresh checkout to see

> what Claude generates from the same instruction.

I don't see how this is true. LLMs can generate different outputs even with the same model and inputs.

[ares623]

the models themselves don't even last a few years. Do they even last a few weeks? Who knows how much they subtly change behind the scenes.

[fragmede]

How different? So do compilers.

[Levitating]

Compilers of the same version should preferably not produce different output with the same input.

[fragmede]

"preferably" doesn't mean t that's the ground truth. Compilers aren't absolutely deterministic, only sufficiently so for most programmers to use.

[scheme271]

If you have a compiler the same source code and the same options, it should generate the same output everything provided you aren't using some compiler pragmas or something similar that embeds timestamps or random numbers or similar. If you give an LLM the same input, it can generate different outputs (controlled by the temperature setting).

[fragmede]

So provided you don't do anything that might cause it to be non-determinist, it's deterministic. Got it.

[scheme271]

I'll be charitable here but you need to go out of your way to introduce non-determinism. Bit reproducible builds and distros exist so it is possible to have an entire distro that can be reliably reproduced bit-by-bit on different systems and at different times.

[fragmede]

It's the other direction. You have to put in an extreme amount of effort, like Debian has, which you can just piggyback off of, to cause determinism to be introduced. 2013 they started that initiative. They're reasonably there, thirteen years later, but to disregard the amount of effort it took to get there would be to forget history. Give ChatGPT thirteen more years to iterate, and see where it is then.

[ares623]

Folks who bring up these "gotchas" should be forbidden from using or taking advantage of the things they are disingenuously whataboutism-ing. Reminds me of sovereign citizen behavior.

[croisillon]

as do most humans

[lefrenchy]

And? The claim is about AI.