[tikwidd]

ChatGPT-style AI should be called AC - Artificial Communication. It is trained on an extremely limited selection of utterances - most of our thinking is never written down. But no matter how many utterances are plugged in, the system will never be able to learn how to think. Its capabilities are stochastic and generalising, not systematic and generative.

An utterance is the expression or outcome of an internal thought process. A machine learning model is trained on the outcome, but not the process. Utterances look like thoughts to us; like pareidolia, we can't help but find thoughts in utterances. This is what makes ChatGPT so compelling.

People don't learn how to think either - it's a capability that grows from a genetic endowment. After the next AI winter, when we get over the spectacle of Artificial Communication, we may begin to examine our own generative capacities for thought. We'll know we're on the right track when AI is slow, uncooperative and childish, fails to thrive when exposed to nonsense training data, and requires very little of the right kind of training data to acquire its language.

[ano88888]

Our internal thought process is an emergent property of a bunch of thoughtless chemical reactions. You're making a big assumption that LLM will never produce similar internal thought process as an emergent property.