[JumpCrisscross]

> There's a big difference between "deterministic + noise" non-determinism and "intelligent agent" non-determinism

I think I agree, but could you expand?

My point is humans have made progress on understanding systems without clean determinism. Genetics and cosmology is technically deterministic, practically speaking, but I don’t think that’s accessible to pre-modern societies. Instead, trends and tendencies were observed, acted and capitalized on.

[mrob]

One big difference is the former doesn't "know" it's being experimented on. Gregor Mendel was able to grow large numbers of pea plants and identify patterns in their inheritance because the non-determinism followed consistent statistical laws. A plant can't suddenly decide to follow different laws of genetics because it saw a biologist.

On the other hand, an LLM "knows" what a benchmark is, and is capable of detecting benchmark-like scenarios. A prompt that appears reliable in testing may fail unexpected in the real world. In this way, an LLM is like a stock market (also composed of intelligent agents): the act of experimenting on it changes its properties. Such systems are never reliably predictable.