[aetherson]

People need to divorce the training method from the result.

Imagine that you were given a very large corpus of reddit posts about some ridiculously complicated fantasy world, filled with very large numbers of proper names and complex magic systems and species and so forth. Your job is, given the first half of a reddit post, predict the second half. You are incentivized in such a way as to take this seriously, and you work on it eight hours a day for months or years.

You will eventually learn about this fantasy world and graduate from just sort of making blind guesses based on grammar and words you've seen before to saying, "Okay, I've seen enough to know that such-and-such proper name is a country, such-and-such is a person, that this person is not just 'mentioned alongside this country,' but that this person is an official of the country." Your knowledge may still be incomplete or have embarrassing wrong facts, but because your underlying brain architecture is capable of learning a world model, you will learn that world model, even if somewhat inefficiently.