[overgard]

You certainly can ask it what it was thinking, the problem is just that it's more likely to make up a plausible sounding fabrication than to say "I don't know" or "my reasoning is hidden for business reasons" (frontier models hide a lot of their chain of thought). Which is the fundamental problem with LLMs though, if the data doesn't exist or it's sparse they make things up.

[muellero]

Choosing plausible sounding fabrication over an admission of ignorance is not an uncommon modality among the human beings I interact with, so I'm not surprised this pattern is found in models trained on human interactions.

[OtomotO]

Totally fine. Then let's just not pretend these "AI"s are somehow better at it.

That's the whole problem with all of these discussions. It's whataboutism and "You're holding it wrong" allegations.