[matthewmacleod]

I think ChatGPT does not understand anything here

This is a good way to think about it generally (or at least how I do it) - ChatGPT doesn’t understand anything, ever.

It produces coherent text which is usually a reasonable response to a prompt. It’s quite amazing at this, and it’s fascinating to see how effective that can be. But it doesn’t have an abstract model of whatever it’s discussing, and can’t reason about it. It’s a vaguely spooky uncanny valley effect.

[JoeAltmaier]

Isn't that splitting hairs? It digests some kind of 'understanding' of human language and expectations. It responds appropriately in most cases, at least as often as an actual human if not more so.

To claim it doesn't understand anything, is akin to digital bigotry. It's a machine, so what it does cannot be 'real' in some undefined sense.

I don't mean to upset anyone; this comment is just to open the conversation about when exactly we're content to permit artifical constructs to be worthy of respect.

[matthewmacleod]

I don’t think it’s splitting hairs, but I also don’t think it’s a criticism to say that a model like this lacks “understanding”. It’s not a bad thing; if anything, I’d say it’s less “respectful” to imbue a digital construct with characteristics it doesn’t have, rather than embracing those which it does!

What I mean by lacking understanding is that ChatGPT responds to most queries the same way I'd expect if I asked a human being without understanding of a topic to write about it. The content it produces feels much like the sort of writing I'd do myself when I end up having to write a few paragraphs about something I don't really know or care about – a kind of veneer of reasonableness, like a short high-school five-paragraph essay.

What I find fascinating when it produces very confident, very incorrect output – that's what highlights a lack of what I'd call "understanding". As an example – last week someone shared with me a regular expression it had generated to match a specific string. It did this correctly, and included a detailed explanation about what it had done that was written in complete, comprehensible sentences but was wildly incorrect, in a sort of self-contradicting way that even the worst human writer would struggle with. It seemed very obvious that it was stringing together words that kind of made sense together, but that there was no abstract model of the thing it was describing.

[JoeAltmaier]

I've noticed the same. It speaks loudly and confidently, with no logic or evidence whatsoever. Just like a blowhard who 'heard something somewhere' and repeats it as fact.