[dist-epoch]

As someone which tried really hard to get deterministic outcome out of them, they really are not.

Layers can be computed in slightly different orders (due to parallelism), on different GPU models, and this will cause small numerical differences which will compound due to auto-regression.

[delusional]

Could someone elighten me on how to compute layers in parallel? I was under the impression that the linearity of the layer computation was why we were mostly bandwidth constrained. If you can compute the layers In parallel then why do we need high bandwidth?

[dist-epoch]

https://developer.nvidia.com/blog/mastering-llm-techniques-i...

[throwawaymaths]

all things being equal, if you fix all of those things and the hardware isn't buggy, you get the same results, and I've set up CI with golden values that requires this to be true. indeed, occasionally you have to change golden values depending on implementation but mathematically the algorithm is deterministic, even if in practice determinidm requires a bit more effort.

[dkersten]

But the reality is that all things aren’t equal and you can’t fix all of those things, not in a way that is practical. You’d have to run everything serially (or at least in a way you can guarantee identical order) and likely emulated so you can guarantee identical precision and operations. You’ll be waiting a long time for results.

Sure, it’s theoretically deterministic, but so are many natural processes like air pressure, or the three body problem, or nuclear decay, if only we had all the inputs and fixed all the variables, but the reality is that we can’t and it’s not particularly useful to say that well if we could it’d be deterministic.

[orbital-decay]

It's definitely reachable in practice. Gemini 2.0 Flash is 100% deterministic at temperature 0, for example. I guess it's due to the TPU hardware (but then why other Gemini models are not like that...).

[throwawaymaths]

Anyways, this is all immaterial to the original question, which is if LLMs can do randomness [for single user with a given query], so from a practical standpoint the question itself needs to survive "all things being equal", that is is to say, suppose I stand up an LLM on my own GPU rig, and the algorithmic scheduler doesn't do too many out of order operations (very possible depending on the ollama or vllm build).