[343rwerfd]

> Next-token prediction cannot handle Kuhnian paradigm shifts

The training datasets contain and reflect human imperfections, including implicit mistakes (which could not be fully extracted in the refining work because ambiguity), and failures in practical logic. Most probably many hallucinations are just the evidence that LLM intelligence or capability to solve (a less controversial way of say it), is actually quite aligned to the way humans think.

I think this is key for new knowledge generation, this could be the "secret sauce" behind the non-deterministic ouput we see when we feed the same prompt to a LLM and we get different outputs, I think is what "allows" the LLM their "generative" vein: a inherent, implicit mathematical structure well hidden in the human substract present in the datasets, that creates a way out of the fixed paths, not merely a recombinatorial behavior, but a true generative behavior.

Hence, the generative AIs we already have, could be the very evidence of the LLM technology being capable of generating new knowledge.