[simonh]

Quite, the idea that mathematical modelling of language will lead to general AI is absurd. The simplest way to defeat chatbots and mathematical language generation models is to teach them something new, like a game or other rules based system and ask them questions about it and then play it. They fall flat on their face immediately because they have no ability to build, interrogate and adapt models of systems.

The authors’ credence of Searle’s Chinese Room argument is telling. The Chinese Room is misdirection. We are invited to consider an agent in a room manipulating symbols on cards and asked could such a system be considered conscious. In fact there might need to be trillions of these agents in rooms covering an area many orders of magnitude larger than the Earth, manipulating millions of trillions of symbols every millisecond. Asking if a system like that could be conscious is a whole different question.

“Here however Turing commits the fallacy of petitio principii, since he presupposes an equivalence between dialogue-ability (as established on the basis of his criterion) and possession of consciousness, which is precisely what his argument is setting out to prove.”

Sigh, no. Dialogue ability isnt claimed to be _equivalent_ to possession of consciousness, that’s putting the cart before the horse. It’s a possible product of consciousness. You could have a conscious system incapable of sensible dialogue, but the point of the test is you can’t have sensible dialogue without consciousness. That’s a claim and it’s arguable, sure, but dialogue ability doesn’t lead to consciousness. That’s daft. They and Searle look at this from entirely the wrong direction.

[roywiggins]

And everyone knows there are conscious agents who can't hold a sensible conversation: toddlers. They're more conscious than any chatbot could ever be, but they'd fail the Turing test. So would dogs, and dogs show more recognizably cognitive ability than a chatbot. Let alone nonhuman primates, who are all much, much smarter than a chatbot and would all fail the Turing test.

It would be one thing if we had built an apelike intelligence and found it impossible to make something smarter, but as we can't model them either, worrying about not entirely understanding language seems beside the point.