[int_19h]

Thing is, they can still solve the problem, even if the problem was not one from its training set.

And, more importantly, they solve the problem much better if you tell them to reason about it in writing first before giving the final answer.

[simonh]

Yes I know, as I said they are very knowledgeable and in some ways very intelligent. We just need to bear in mind their processing architecture is radically different from our. This makes our intuitions about their abilities highly error prone.

[int_19h]

Absolutely. The shoggoth metaphor is extremely apt here.

What I was specifically responding to is the claim that they can only solve certain kinds of problems because those kinds of problems (and their solutions) were in the training set. By now there's plenty of counter-examples of unique problems that are nevertheless solved. At which point I think we do have to call it "understanding" and "reasoning", even as we acknowledge that it is a very alien form of understanding and reasoning that we just barely managed to squeeze into something that kinda sorta feels humanish.

[giardini]

simonh says >"We just need to bear in mind their processing architecture is radically different from ours."<

The hardware architectures are certainly different but there is a possibility that at least parts of the "software" architectures may be remarkably similar.

Time and research will tell.