I think this is at least a major part of the problem, and I share this hypothesis (and have been saying as much for the past two years). We never train them to know what they know and what they don't know (meta-cognition). We only train them to output the next token, not to self-reflect and ask: Is this true based on my prior knowledge?