[chongli]

Having a probability distribution to sample from is not the same thing is knowing, because they don’t know anything about the provenance of the data that was used to build the distribution. They trust their training set implicitly by construction. They have no means to detect systematic errors in their training set.

[adastra22]

You are talking about something different. If I ask you a yes/no question, and then ask you how certain you are, the answer you give is not an objective measurement of how likely you are to be right. You don't have access to that either. If you say "I'm very confident" or "Maybe 50/50" -- that is an assessment of your own internal weighted evidence, which is the equivalent of an LLM's softmax distribution.

[chongli]

The difference is: I know the provenance of my evidence. Some of it I may have read in a book, some online, some I may have heard from a teacher or a professor, but some evidence I may have gathered directly from experiments I performed myself.

If you ask me “how certain are you that the standard model of particle physics is true?” I’ll answer “I don’t know” because I don’t have any subject matter expertise, and philosophically I tend to hedge on questions like this anyway (“all models are wrong, some are useful”).

However, if you ask me “how certain are you that food is bland with no salt added, tastes better with some salt, but tastes bad with too much salt?” I would answer “very certain” because I have loads of direct experiments on this question in the kitchen. Furthermore, between these two extremes

To an LLM these are identical kinds of questions. All evidence has the same provenance: the training set. As of yet, we don’t have embodied AIs (robots) with multi-modal sensory inputs and online training. Until then, what we have remains a “brain in a vat fed on tokens” which, to me, is extremely weak from an epistemic perspective.