[nopinsight]

It's far from obvious that thought space is much less complex than physical space. Natural language covers emotional, psychological, social, and abstract concepts that are orthogonal to physical aptitude.

While the linguistic representation of thought space may be discrete and appear simpler (even the latter is arguable), the underlying phenomena are not.

Current LLMs are terrific in many ways but pale in comparison to great authors in capturing deep, nuanced human experience.

As a related point, for AI to truly understand humans, it will likely need to process videos, social interactions, and other forms of data beyond language alone.

[visarga]

I think the essence of human creativity is outside our brains - in our environments, our search spaces, our interactions. We stumble upon discoveries or patterns, we ideate and test, and most ideas fail but a few remain. And we call it creativity, but it's just environment tested ideation.

If you put an AI like AlphaZero in a Go environment it explores so much of the game space that it invents its own Go culture from scratch and beats us at our own game. Creativity is search in disguise, having good feedback is essential.

AI will become more and more grounded as it interacts with the real world, as opposed to simply modeling organic text as GPT-3. More recent models generate lots of synthetic data to simulate this process, and it helps up to a point, but we can't substitute artificial feedback for real one except in a few cases: like AlphaZero, AlphaProof, AlphaCode... in those cases we have the game winner, LEAN as inference engine, and code tests to provide reliable feedback.

If there is one concept that underlies both training and inference it is search. And it also underlies action and learning in humans. Learning is compression which is search for optimal parameters. Creativity is search too. And search is not purely mental, or strictly 1st person, it is based on search spaces and has a social side.