[dtagames]

In fact, nondeterminism goes all the way down to the training level where every token and weight starts as a random number, a concept I wrote about in this intro to how AI works article [0].

The reason for this is that randomness is crucial to getting emergent data out of a system. Those are unexpected, unpredictable, but often useful results. This is how an LLM can answer a question it's never been asked before.

We have had deterministic databases forever so there would be no AI advance if LLMs were just databases. The AI models of the 1960's tried that very approach, called rules based and it doesn't work. We can never come up with or write down all the rules. The failure of those methods lead to the "AI winter" and no further real progress in AI until the invention of transformers at Google.

[0] https://levelup.gitconnected.com/something-from-nothing-d755...