[simonh]

I’d say sophistication.

Observing the landscape enables us to spot useful resources and terrain features, or spot dangers and predators. We are afraid of dark enclosed spaces because they could hide dangers. Our ancestors with appropriate responses were more likely to survive.

A huge limitation of LLMs is that they have no ability to dynamically engage with the world. We’re not just passive observers, we’re participants in our environment and we learn from testing that environment through action. I know there are experiments with AIs doing this, and in a sense game playing AIs are learning about model worlds through action in them.

[FloorEgg]

The idea I keep coming back to is that as far as we know it took roughly 100k-1M years for anatomically modern humans to evolve language, abstract thinking, information systems, etc. (equivalent to LLMs), but it took 100M-1B years to evolve from the first multi-celled organisms to anatomically modern humans.

In other words, human level embodiment (internal modelling of the real world and ability to navigate it) is likely at least 1000x harder than modelling human language and abstract knowledge.

And to build further on what you are saying, the way LLMs are trained and then used, they seem a bit more like DNA than the human brain in terms of how the "learning" is being done. An instance of an LLM is like a copy of DNA trained on a play of many generations of experience.

So it seems there are at least four things not yet worked out re AI reaching human level "AGI":

1) The number of weights (synapses) and parameters (neurons) needs to grow by orders of magnitude

2) We need new analogs that mimic the brains diversity of cell types and communication modes

3) We need to solve the embodiment problem, which is far from trivial and not fully understood

4) We need efficient ways for the system to continuously learn (an analog for neuroplasticity)

It may be that these are mutually reinforcing, in that solving #1 and #2 makes a lot of progress towards #3 and #4. I also suspect that #4 is economical, in that if the cost to train a GPT-5 level model was 1,000,000 cheaper, then maybe everyone could have one that's continuously learning (and diverging), rather than everyone sharing the same training run that's static once complete.

All of this to say I still consider LLMs "intelligent", just a different kind and less complex intelligence than humans.

[kla-s]

Id also add that 5) We need some sense of truth.

Im not quite sure if the current paradigm of LLMs are robust enough given the recent Anthropic Paper about the effect of data quality or rather the lack thereof, that a small bad sample can poison the well and that this doesn’t get better with more data. Especially in conjunction with 4) some sense of truth becomes crucial in my eyes (Question in my eyes is how does this work? Something verifiable and understandable like lean would be great but how does this work with more fuzzy topics…).

[FloorEgg]

That's a segue into an important and rich philosophical space...

What is truth? Can it be attained, or only approached?

Can truth be approached (progress made towards truth) without interacting with reality?

The only shared truth seeking algorithm I know is the scientific method, which breaks down truth into two categories (my words here):

1) truth about what happened (controlled documented experiments) And 2) truth about how reality works (predictive powers)

In contrast to something like Karl friston free energy principle, which is more of a single unit truth seeking (more like predictive capability seeking) model.

So it seems like truth isn't an input to AI so much as it's an output, and it can't be attained, only approached.

But maybe you don't mean truth so much as a capability to definitively prove, in which case I agree and I think that's worth adding. Somehow integrating formal theorem proving algorithms into the architecture would probably be part of what enables AI to dramatically exceed human capabilities.

[simonh]

I think that in some senses truth is associated with action in the world. That’s how we test our hypotheses. Not just in science, in terms of empirical adequacy, but even as children and adults. We learn from experience of doing, not just rote, and we associate effectiveness with truth. That’s not a perfect heuristic, but it’s better than just floating in a sea of propositions as current LLMs largely are.

[FloorEgg]

I agree.

There's a truth of what happened, which as individuals we can only ever know to a limited scope... And then there is truth as a prediction ability (formula of gravity predicts how things fall).

Science is a way to build a shared truth, but as an individual we just need to experience an environment.

One way I've heard it broken down is between functional truths and absolute truths. So maybe we can attain functional truths and transfer those to LLMs through language, but absolute truth can never be attained only approached. (The only absolute truth is the universe itself, and anything else is just an approximation)

[pbhjpbhj]

>A huge limitation of LLMs is that they have no ability to dynamically engage with the world.

They can ask for input, they can choose URLs to access and interpret results in both situations. Whilst very limited, that is engagement.

Think about someone with physical impairments, like Hawking (the now dead theoretical physicist) had. You could have similar impairments from birth and still, I conjecture, be analytically one of the greatest minds of a generation.

If you were locked in a room {a non-Chinese room!}, with your physical needs met, but could speak with anyone around the World, and of course use the internet, whilst you'd have limits to your enjoyment of life I don't think you'd be limited in the capabilities of your mind. You'd have limited understanding of social aspects to life (and physical aspects - touch, pain), but perhaps no more than some of us already do.

[skissane]

> A huge limitation of LLMs is that they have no ability to dynamically engage with the world.

A pure LLM is static and can’t learn, but give an agent a read-write data store and suddenly it can actually learn things-give it a markdown file of “learnings”, prompt it to consider updating the file at the end of each interaction, then load it into the context at the start of the next… (and that’s a really basic implementation of the idea, there are much more complex versions of the same thing)

[TheOtherHobbes]

That's going to run into context limitations fairly quickly. Even if you distill the knowledge.

True learning would mean constant dynamic training of the full system. That's essentially the difference between LLM training and human learning. LLM training is one-shot, human learning is continuous.

The other big difference is that human learning is embodied. We get physical experiences of everything in 3D + time, which means every human has embedded pre-rational models of gravity, momentum, rotation, heat, friction, and other basic physical concepts.

We also learn to associate relationship situations with the endocrine system changes we call emotions.

The ability to formalise those abstractions and manipulate them symbolically comes much later, if it happens at all. It's very much the plus pack for human experience and isn't part of the basic package.

LLMs start from the other end - from that one limited set of symbols we call written language.

It turns out a fair amount of experience is encoded in the structures of written language, so language training can abstract that. But language is the lossy ad hoc representation of the underlying experiences, and using symbol statistics exclusively is a dead end.

Multimodal training still isn't physical. 2D video models still glitch noticeably because they don't have a 3D world to refer to. The glitching will always be there until training becomes truly 3D.

[skissane]

An LLM agent could be given a tool for self-finetuning… it could construct a training dataset, use it to build a LORA/etc, and then use the LORA for inference… that’s getting closer to your ideal

[ako]

Yes, and give it tools and it can sense and interact with its surroundings.

[FloorEgg]

Oh, I just realized you maybe we're referring to Kopple when you said sophistication?

If so, then yes, that might be a good measure. I'm not deep enough in this to have an opinion on if it's the best measure. There are a few integrated information theories and I am still getting my head wrapped around them...