[djusk]

The author addresses this, you can get explanations that seem logical, but ChatGPT is not telling you what it was actually "thinking" about when generating the responses. It's just looking at what it's said and coming up with an explanation after the fact.

[ciex]

Which is the same as what humans do? How would you tell anybody what you are „actually“ thinking? People are not very good at understanding their own motivations, thought processes, ulterior motives and on top of that we routinely fabricate or alter our own memories. We compensate for that by building social systems to collectively error collect, plan and model the world. That’s what AI will probably end up doing as well.

[nevertoolate]

So if solve a problem you can’t retrace the thought process? I mean do you not take notes or scribble?

[ciex]

Oh yes, but again, that’s actually very similar between how humans and llms think. Both can use notes to improve coherence and increase the complexity that can be handled in a thought process but there is no strong guarantee that these notes are internally consistent.

[chillacy]

Surprisingly similar to the results of humans in split brain studies, the brain seems to provide and accept pretty nonsensical explanations for behavior.