That's not a useful description. The AI doesn't "know" what it "knows". It's not even filling some gap in its knowledge. It's just putting words together that statistically can go together.
Who cares if "it knows" if it's apparently impossible to get it to use that knowledge to stop hallucinating?
To me, the end user there is no practical difference between not having the data and not being able to use the data it has. If it can't use it, or refuses to use it, it may as well not exist.
If you'd bothered to look at the linked papers, you'd see it's not "impossible" to get much better calibration. Just difficult and something that could be further worked on.
> Who cares if "it knows" if it's apparently impossible to get it to use that knowledge to stop hallucinating?
It sounds like you're describing the MSM. In any case, the same problem is true for a fair number of people. The difference is, silicon training has a much better chance of evolving beyond its current limitations much sooner than the typical human.
Yes it does.
https://news.ycombinator.com/item?id=37874556