[fnordpiglet]

Are you saying the discovery of relativity happened because Einstein was reacting to some reward / stimulus in his environment? Galois’ discoveries were a stochastic parrot regurgitating stimulus from his life?

There are known faculties humans have that LLMs especially do not, such as actual memory, the ability to simulate the world independently via the imagination and structured thought, as well as facilities we don’t really understand but AIs definitely don’t have which are the source of our fundamental agency. We are absolutely able to create thought and reasoning without direct stimulus or as a response to something in the environment - and it’s frankly bizarre a human being can believe they’ve never done something as a reaction to their internal state rather than extrinsic.

LLMs literally can not “do” anything that isn’t predicated on their training set. This means, more or less, they can only interpolate within their populated vector space. The emergent properties are astounding and they absolutely demonstrate what appears to be some form of pseudo abductive reasoning which is powerful. I think it’s probably the most important advance of computing in the last 30 years. But people have confused a remarkable capability for a human like capability, and have simultaneously missed the importance of the advance as well as inexplicably diminished the remarkable capabilities of the human mind. It’s possible with more research we will bridge the gaps, and I’m not appealing to magic of the soul here.

But the human mind has a remarkable ability to reason, synthesize, extrapolate beyond their experience, and those are all things LLMs fundamentally - from a rigorous mathematical basis - can not do and will never do alone. Any thing that bridges that will need an ensemble of AI and classical computing techniques - and maybe LLMs will be a core part of a part of something even more amazing. But we aren’t there yet and I’ve not seen a roadmap that takes us there.

[strangescript]

Or all these properties are emergent from ever increasing compute. The underlying architecture doesn't need to fundamentally change. The roadmap is increasing compute. Agency is simply never ceasing input and resulting output, both internally and externally. Do you have Agency when you sleep? I still don't know what "extrapolate beyond their experience" means. Does Einstein discover relativity if he is never taught math? More to the point, why was he so brilliant in the first place. Why could he "extrapolate beyond their experience" better than others in his field?

We want AI to hoop jump something we don't even understand ourselves. The only empirical evidence we have is as we increase compute, the results get better.

[fnordpiglet]

I would note you absolutely have agency when you sleep - in fact a lot of the purpose of dreams is to simulate situations for you to practice in safely - that’s why you’re often replaying situations you’re worried about or things you commonly experience. Also lucid dreaming is still fully immersed dreaming but you have total agency and awareness. The key to lucid dreaming is just training yourself to not give up aware of the non dream reality control while dreaming.