[TZubiri]

2 caveats:

1st: Even with temp 0 and seed set, you can still get variations, I've heard the explanation relates to batching with other tenants and race conditions in the gpu parallel processing.

Also slower int quantized models may reduce or eliminate this fluctuation.

2nd: we need to talk about the difference between determinism and chaos. Because I don't give a fuck if "Write me the firmware for a TC scanner" always produces the same output, I care whether it will be different if it's not capitalized.

[orbital-decay]

Issue #1 requires batch-invariant kernels. They're already implemented in some inference engines with negligible performance loss, e.g. DeepSeek v4.

For the issue #2, identical outputs are impossible to guarantee for semantical variations, by definition. If you reduce your requirements to the acceptable difference between model's and human's understanding of the prompt, then it becomes a solvable (but not solved) training policy issue. Moreover, it doesn't necessarily need to be solved, because retrying with a slightly different prompt is a thing you don't want to do in the first place.