[coldtea]

>I suspect (ignorant armchair) we have some ability to signal strength of those facts, via repetition. Without this layer of introspection

Isn't that precisely what the LLM training does? It signals strength of those facts, via repetition.

[unshavedyak]

Yea but i don't think the LLM can inspect that. Eg it's similar to a code function , the code executes but the code doesn't have the ability to inquire about the code itself.

In that silly example/thought, the LLM would effectively need the ability to query the strength of a fact/spatial region/etc.

Right now i believe the LLM is more just the output of those weights. It has no way to inspect the strength of the signal. Eg it doesn't know if blue in "The sky is <blue>" is a strong or weak signal, it just predicted that next token.

If we could somehow encode strengths along with every token, eg "The sky is <blue:1.0>" or something we could perhaps give it a sense of [un]certainty. Though i imagine it would look differently than that since we'd want to encode this information in some sort of multi-dimensional space, rather than purely by token - since tokens aren't that valuable. Eg the "knowledge" in an LLM goes beyond tokens, and so too should signal strength.

I'm of course speculating on all of this and i have no clue on anything.

[coldtea]

>Right now i believe the LLM is more just the output of those weights. It has no way to inspect the strength of the signal. Eg it doesn't know if blue in "The sky is <blue>" is a strong or weak signal, it just predicted that next token.

Yes, but the prediction is itself encoding that it's a strong signal.

Meaning that, if given a context (prompt), it predicted that "the sky is blue", that's because the sky being blue is a stronger signal under that context - statistically speaking.

[unshavedyak]

Stronger, yes, but not necessarily strong. 2 instances of hearing that the sky is green might be stronger than 1 sky is blue, but that might not fit an expected threshold of say 100 instances of learning for a "fact" to be actually strong. At least in my example.

My point was that i feel like humans have these two aspects, the ability to have a fact, and the ability to have a signal to the facts strength. I propose that as an explanation why we can internally analyze our understanding and come to a conclusion that yes, we do "know" it.

We also at times can't figure out how we "know" it, either because we've made up a detail (filling in blanks, assumptions, etc), or because we forgot where we embedded this detail. The lower the signal strength of this validation the more we can be unsure about a "fact".

I feel like LLMs are one half, but not the other. They effectively need a RAG for all of the knowledge they have, and if the counter for a given fact/idea/etc is low enough, then it's an low signal.

The question i have is how to do this efficiently. Of course i'm just speculating too, i don't know any of this.