[roywiggins]

And everyone knows there are conscious agents who can't hold a sensible conversation: toddlers. They're more conscious than any chatbot could ever be, but they'd fail the Turing test. So would dogs, and dogs show more recognizably cognitive ability than a chatbot. Let alone nonhuman primates, who are all much, much smarter than a chatbot and would all fail the Turing test.

It would be one thing if we had built an apelike intelligence and found it impossible to make something smarter, but as we can't model them either, worrying about not entirely understanding language seems beside the point.