[danpat]

> but why would you expect that from it?

If you understand what it is doing, then you don't. But the layman will just see a computer that talks in language they understand, and will infer intent and sentience are behind that, because that's the only analog they have for a thing that can talk back to them with words that appear to make sense at the complexity level that ChatGPT is achieving.

Most humans do not have sufficient background to understand what they're really being presented with, they will take it at face value.

[hunter2_]

I think this is partly explained by most of the marketing and news essentially saying "ChatGPT is an AI" instead of "ChatGPT is an LLM."

If you asked me what AI is, I'd say it means getting a computer to emulate human intelligence; if you asked me what an LLM is, I'd say it means getting a computer to emulate human language. The word "language" does not imply truthiness anywhere near to the extent that the word "intelligence" does.

[int_19h]

You could reasonably describe it as "human language emulator" back when people were using GPT-2 and the likes to compose text. But what we have today doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform, and then carries them out. Granted, the only thing it can really "do" is produce text, but that already covers quite a lot of tasks - and then of course text can be an API call.

[hunter2_]

Interesting perspective. I'm still learning about what it really is, and I'm having trouble marrying the thoughts of a parent commenter with yours:

> ... does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason ...

versus

> ... doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform ...

Maybe a third party can jump in here: does ChatGPT use reasoning beyond the domain of language, or not?

[int_19h]

Nobody can definitely answer this question because we don't know what exactly is going on inside the model of that size. We can only speculate based on the observed behavior.

But in this case, I didn't imply that it's "reasoning beyond the domain of language", in a sense that language is exactly what it uses to reason. If you force it to perform tasks without intermediate or final outputs that are meaningful text, the result is far worse. Conversely, if you tell it to "think out loud", the results are significantly better for most tasks. Here's one example from GPT-4 where the "thinking" effectively becomes a self-prompt for the corresponding SQL query: https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e530....

Or here's an even more interesting example where GPT-4 does this kind of "thinking out loud" unprompted: https://gist.github.com/int19h/8251bd00b7a4858a69cf3922ae674...

I think the real point of disagreement is whether this constitutes actual reasoning or "merely completing tokens". If you showed the transcript of a chat with GPT-4 solving a multi-step task to a random person off the street, I have no doubt that they'd describe it as reasoning. Beyond that, one can pick the definition of "reason" that best fits their interpretation - there is no shortage of them, just as there is no shortage of definitions for "intelligence", "consciousness" etc.

[nomel]

> Most humans do not have sufficient background to understand what they're really being presented with, they will take it at face value.

Trust in all forms of media [1], and institutions [2], is at an all time low. I'm not sure why that distrust would go away, with reading the output from a company that's clearly censuring/fudging that output.

I think any "damaging" trust would, clearly, be a transitory phenomenon, since the distrust in media and institutions is from an ability to see BS. I don't think some sentences on a screen will be as destructive as some think, because people don't appear to be as stupid as some believe.

1. https://news.gallup.com/poll/403166/americans-trust-media-re...

2. https://news.gallup.com/poll/394283/confidence-institutions-...