[drew-y]

Totally agree with the sentiment. I find the constant debates on "is AI conscious" or "can AI understand" exhausting. You can't have a sound argument when neither party even agrees on a concrete definition of consciousness or understanding.

Regarding this line:

> ChatGPT can already tell you a lot about itself (showing awareness) and will gladly walk you through its “thinking” if you ask politely.

Is it actually walking you through its thinking? Or is it walking you through an imagined line of thinking?

Regardless, your main point still stands. That a program doesn't think the same way a human does, doesn't mean it isn't "thinking".

[xigency]

> Is it actually walking you through its thinking? Or is it walking you through an imagined line of thinking?

You can prompt an LLM model to provide reasoning first and an answer second and it becomes one and the same.

Worth keeping in mind that all of these points are orthogonal to the quality of reasoning, the bias, or the intentions of the system builders. And building something that emulates humans convincingly, you can expect it to emulate both the good and bad qualities naturally.

[Davidzheng]

In split brain experiments (where the corpus callosum is cut), sometimes the half which is nonverbal is prompted and the action is taken. Yet when the experimenter asks for the explanation the verbal half supplies an (incorrect) explanation. How much of human reasoning when prompted occurs before the prompt? it's a question you have to ask as well.

[intended]

why does performance improve after chain of thought prompting?

Because a human is measuring it unfairly.

The output without CoT is valid. It is syntactically valid. The observer is unhappy with the semantic validity, because the observer has seen syntactic validity and assumed that semantic validity is a given.

Like it would if the model was alive.

This is observer error, not model error.