[tibbar]

If you use a deterministic sampling strategy for the next token (e.g., always output the token with the highest probability) then a traditional LLM should be deterministic on the same hardware/software stack.

[extraduder_ire]

Wouldn't seeding the RNG used to pick the next token be more configurable? How would changing the hardware/other software make a difference to what comes out of the model?

[tibbar]

> Wouldn't seeding the RNG used to pick the next token be more configurable?

Sure, that would work.

> How would changing the hardware/other software make a difference to what comes out of the model?

Floating point arithmetic is not entirely consistent between different GPUs/TPUs/operating systems.

[roywiggins]

Deterministic is one thing, but stable to small perturbations in the input is another.

[dragonwriter]

> Deterministic is one thing, but stable to small perturbations in the input is another.

Yes, and the one thing that was asked about was "deterministic" not "stable to small perturbations in the input.

[kokada]

This looks "fun" too: commit fixing a small typo -> the app broke.

[lvncelot]

So nothing's changed, then :D