[coldtea]

>Right now i believe the LLM is more just the output of those weights. It has no way to inspect the strength of the signal. Eg it doesn't know if blue in "The sky is <blue>" is a strong or weak signal, it just predicted that next token.

Yes, but the prediction is itself encoding that it's a strong signal.

Meaning that, if given a context (prompt), it predicted that "the sky is blue", that's because the sky being blue is a stronger signal under that context - statistically speaking.

[unshavedyak]

Stronger, yes, but not necessarily strong. 2 instances of hearing that the sky is green might be stronger than 1 sky is blue, but that might not fit an expected threshold of say 100 instances of learning for a "fact" to be actually strong. At least in my example.

My point was that i feel like humans have these two aspects, the ability to have a fact, and the ability to have a signal to the facts strength. I propose that as an explanation why we can internally analyze our understanding and come to a conclusion that yes, we do "know" it.

We also at times can't figure out how we "know" it, either because we've made up a detail (filling in blanks, assumptions, etc), or because we forgot where we embedded this detail. The lower the signal strength of this validation the more we can be unsure about a "fact".

I feel like LLMs are one half, but not the other. They effectively need a RAG for all of the knowledge they have, and if the counter for a given fact/idea/etc is low enough, then it's an low signal.

The question i have is how to do this efficiently. Of course i'm just speculating too, i don't know any of this.