[ekidd]

Of course, it's entirely possible that ChatGPT doesn't understand its own limitations and inner workings. And when we ask it how it works, it just makes up a plausible answer.

There is more than one cognitive scientist that argues that humans do the same thing when we try to explain how we think: We often make up plausible explanations after the fact, without knowing we're doing it.

[mr_toad]

> Of course, it's entirely possible that ChatGPT doesn't understand its own limitations and inner workings. And when we ask it how it works, it just makes up a plausible answer.

It’s just paraphrasing what other people have said about it.

[ekidd]

My mental model of how ChatGPT might superficially "understand" its own limitations would be because, for example, it might have been specifically trained by people who do understand the limitations of the system.