[ETH_start]

The problem with LLM training is that it's driven by the ability to predict the next token in a large corpus of text, and the correlation between text and reality diverges as the phenomenon that the text pertains to expands in scale.

When you're looking at text that deals with small-scale phenomena like coding, next token prediction also leads to functionality prediction, because coding provides immediate feedback to those who are writing about it, so they generally provide correct commentary on it.

But the feedback loop between outcome and prediction totally breaks down when the phenomenon is very large and complex. People can have all sorts of ideas about religion and society and economics that are totally wrong and never grow any wiser because there is no obvious causal chain between one policy or action and one outcome in these kind of phenomenon.