[leereeves]

How can a neural network evaluate "confidence"?

The parameters don't store any information about what inputs were seen in the training data (vs being interpolated) or how accurate the predictions were for those specific inputs.

And even if they did, the training data was usually gathered voraciously, without much preference for quality reasoning.

[habitue]

I don't know for sure, but here's a plausible mechanism for how:

Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance.

There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work

[brookst]

That’s good. Also maybe an architecture that runs the query through multiple times and then evaluates similarity of responses, then selects (or creates) the most-generated one, along with a confidence level of how many of the individual responses were aligned.

[XenophileJKO]

Actually you can get a very good proxy by looking at the probability distrobution of the "answer" tokens. The key here is you have to be able to identify the "answer" tokens.

https://arxiv.org/abs/2402.10200

[pbhjpbhj]

Phind gives me ChatGPT answers with relatively authoritative references to works on the web that (usually!) support the answer. Could it have a post-filter to fact check against the references?

I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material.

I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.

[montjoy]

IIRC confidence in video is related to predicting what happens next vs what actually happens. If the two seem to correlate to the model it would give it a higher confidence ranking, which would then be used further for self-reinforced learning.