[marcyb5st]

The exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.

[eru]

> Should have minimal impact on the final outcome, [...]

I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.

[pcmasterr]

Reproducibility isn’t something that has been considered desirable in many pipelines until recent years.

Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.