[int_19h]

Try something like this as input sometime:

   I want you to replace the word "right" in your output thereafter as follows:
   if it indicates direction, say "durgh;
   if it indicates being near or close, say "nolpi";
   if it indicates correctness, say "ceza".
   I will also use these replacement words accordingly and expect you to be able to understand them. 

And see how well it can maintain a conversation, solve a task, or write a story with these constraints.

ChatGPT seems to get this wrong most of the time, but Bing AI is consistently better (although may need to be jailbroken to accept the idea of word substitution to begin with). It still makes occasional mistakes, but on the whole I'd say that it has to somehow "understand" what the words mean conceptually, whether when generating them or when processing them as input; it's hard to see how this trick could work in an extended conversation if it were a mere "stochastic parrot".

[civilized]

I agree. There are many indicators that it has some sort of deeper understanding of the meaning of language. Even in the conversation I had, for all its flaws, it was able to correctly perceive inconsistencies in its statements based on my prompts and make somewhat coherent attempts to correct them. It's just that the understanding can be so fragile, and its attempts to resolve inconsistencies are superficial, incurious, bullshitty.

"Density matters for weight but not mass" is a perfect example - it's ridiculous, but I can understand how it logically inferred that from its own previous statements. I'd bet plenty of money that it didn't get this crazy idea from its training data.

To be fair, humans have the same sort of issue sometimes. But ChatGPT seems to have more extreme versions of the issue and perseveres confidently with no self-awareness.

Really though, not bad for an autoregressive text model trained on terabytes of internet data.

[int_19h]

I think a big part problem here is that "understand X" is just a shorthand way to say "has an internal model of X" - but the degree and accuracy of said understanding depends entirely on the quality of that model.

Now there's a good reason to believe that ChatGPT does have such a model, based on the Othello experiment. But, firstly, the size of that internal model is inherently constrained by the size of the neural net, and I doubt that the limit is anywhere large enough to allow a truly accurate approximation of the real world.

And then on top of that, said model is created based on inferences from text only, which is several steps away from the original data (audiovisual, sensory etc), and one short snippet of text at a time. Some things retain meaning better in this format than others, and I think this might explain why ChatGPT and Bing are both hilariously bad at spatial navigation beyond 1-2 steps even in simple tasks.

It will be very interesting to see how this evolves as the models are scaled up and get large enough to handle things other than text.