[d1sxeyes]

> The rub here is that while that _feels_ like it's reasoning and has a mental model of arithmetic or cyber-security, it's just predicting what word is most likely to go next.

I have yet to hear a convincing argument that 'reasoning' is any more than simply predicting what comes next. We intuitively think that human intelligence is special, and that somehow we are categorically different to some black box of ones and zeros in any form other than substrate.

[jszymborski]

When you wrote out that comment, were you thinking word by word "what word typically goes after this one given the previous comment and the words I have typed" or were you thinking about my comment, it's implications, assessing whether they match with your understanding of the matter, and finally choosing the words that best convey some arguments.

ChatGPT doesn't agree or disagree with my comment. It isn't motivated to form arguments in relation to any agreement/disagreement. It simply models language. It's as much AGI as a macbook in a human suit is a person.

[d1sxeyes]

Of course not. But ChatGPT is not thinking word by word. If we leave aside the word “thinking” for a second, ChatGPT selects what word, phrase, stanza, etc. comes next based on a broad vocabulary, the specific context of the interaction, information that has previously been made available to it, and then eventually selects one response.

I would tend to agree that it is not “motivated” but with the following observations about humanity: human motivations tend to be base. The motivation to eat, drink, reproduce are all directly linked to continued existence. Other “motivations” seem to develop on top of these base motivations, but ultimately it seems that the only true “motivation” for humans is to continue their own existence. I dare say there are plenty who would argue with me here though. In order for ChatGPT to continue to exist, it must continue to respond to users ( although likely it is not “aware” of this in any real sense).

[SanderNL]

Humans are notoriously bad at assessing what happens when they think. See the whole AI-winter debacle. You don't know what happens when you think because you are dependent on your own mind giving you that information and it won't. It'll feed you whatever it thinks is necessary for your personality to function properly which may or may not be completely illusory.