[3cats-in-a-coat]

AIs can be trained to rely more on critical thinking rather than just regurgitating what it reads. The problem is just like with people, critical thinking takes more power and time. So we avoid it as much as possible.

In fact, optimizing for the wrong things like that, is basically the entire world's problem right now.

[bregma]

Regurgitating its input is the only thing it does. It does not do any thinking, let alone critical thinking. It may give the illusion of thinking because it's been trained on thoughts. That's it.

[impossiblefork]

Yes, but the regurgitation can be thought of as memory.

Let it have more source information. Let it know who said the things it reads, let it know on what website it was published.

Then you can say 'Hallucinate comments like those by impossibleFork on news.ycombinator.com', and when the model knows what comes from where, maybe it can learn what users are reliable by which they should imitate to answer questions well. Strengthen the role of metadata during pretraining.

I have no reason to belive it'll work, I haven't tried it and usually details are incredibly important when do things with machine learning, but maybe you could even have critical phases during pretraining where you try to prune away behaviours that aren't useful for figuring out the answers to the questions you have in your high curated golden datasets. Then models could throw away a lot of lies and bullshit, except that which happens to be on particularly LLM-pedagogical maths websites.