[kerkeslager]

The algorithms are definitely not deterministic. That said I agree with your general point that experimenting on LLMs as if they're black boxes with unknown internals is silly.

EDIT: I'm seeing another poster saying "Deterministic with a random seed?" That's a good point--all the non-determinism comes from the seed, which isn't particularly critical to the algorithm. One could easily make an LLM deterministic by simply always using the same seed.

[dist-epoch]

> all the non-determinism comes from the seed

not fully true, when using floating point the order of operations matters, and it can vary slightly due to parallelism. I've seen LLMs return different outputs with the same seed.

[onionisafruit]

That’s an interesting observation. Usually we try to control that, but with LLMs the non-determinism is fine.

It seems like that would make it hard to unit test LLM code, but they seem to be managing.

[kerkeslager]

Oh, that's really interesting. Hadn't thought of that.