[xienze]

> "If what LLMs do today isn't actual thinking, what is something that only an actually thinking entity can do that LLMs can't?"

Invent some novel concept, much the same way scientists and mathematicians of the distant past did? I doubt Newton's brain was simply churning out a stream of the "next statistically probable token" until -- boom! Calculus. There was clearly a higher order understanding of many abstract concepts, intuition, and random thoughts that occurred in his brain in order to produce something entirely new.

[danielbln]

My 5 year old won't be coming up with novel concepts around calculus either, yet she's clearly thinking, sentient and sapient. Not sure taking the best of the best of humanity as the goal standard is useful for that definition.

[omnicognate]

"It's an unreasonably high standard to require of LLMs": LLMs are already vastly beyond your 5 year old, and you and me and any research mathematician, in knowledge. They have no greater difficulty talking about advanced maths than about Spot the Dog.

"It's a standard we don't require of other humans": I think qualitatively the same capabilities are used by all humans, all the time. The special thing about novel mathematical thinking is that it is verifiable, requires genuine insight and is a text generation task, not that you have to be able to do it to be considered intelligent.

[hshdhdhj4444]

> Newton's brain was simply churning out a stream of the "next statistically probable token"

At some level we know human thinking is just electrons and atoms flowing. It’s likely at a level between that and “Boom! Calculus”, the complexity is equivalent to streaming the next statistically probably token.