[moffkalast]

There's only so much information content you can get from a mug though.

We get a lot of high quality data that's relatively the same. We run the same routines every day, doing more or less the same things, which makes us extremely reliable at what we do but not very worldly.

LLMs get the opposite: sparse, relatively low quality, low modality data that's extremely varied, so they have a much wider breadth of knowledge but they're pretty fragile in comparison since they get relatively little experience on each topic and usually no chance to affirm learning with RL.

[ForceBru]

Yep, LLMs have a greater breadth of knowledge, but it's shallow. Humans are able to achieve much greater depth because they have more data about the subject.