[_superposition_]

I can't prove it but I have strong beliefs that the logic and intuition required for abstracting for future changes is not possible in a stream of predicted tokens. Mental models aren't built from text. Something's missing that can't be measured.

[tau5210]

I believe similar.

Maybe not exactly the same thing, but I think one related representation of this in any good codebase is documentations in the code: Good documentation always focuses on the why and not the what. i.e. things that are not represented by or representable in code. It may describe not only why a certain choice was made, but also why not something else. In general, the whole point of these documentation is to account for things that are beyond what can be understood from just reading the codebase including other existing documentation, that you may never even imagine with all the tokens in the world otherwise, which were yet a critical part of our coding choices.

If we are to believe that LLMs can build complex systems, then it's equivalent to saying that LLMs can make similar decisions equally well without any of that... which also then implies that we essentially never needed these documentation to begin with other than as redundancy for human needs.

[_superposition_]

I can't really explain it but when I am working with a system in my head it's not represented as text. It has an almost physical dimension to it similar to imagining how an engine works or what have you. This is obviously beyond the grasp of an LLM.

[tau5210]

Interesting! Makes me curious what you do for work.

[Mikhail_Edoshin]

The real understanding builds on not knowing. Once I realise I don't know something I cease thinking and start watching the thing. Then somehow I understand.

We do have a mechanism similar to LLMs but it provides existing knowledge; it's a search mechanism, basically. New knowledge happens when we turn that mechanism off.

[hahahaa]

LLMs can learn but only by copy paste into that tiny context. Actually theoretically they could self fine-tune but it would be very expensive/slow.