Nobody really knows for sure whether AIs are concious. But nobody really knows for sure whether rocks are concious either. The reality is we understand very little about conciousness.
That point is stated explicitly in the piece. Essentially, we can't know that even other humans are conscious. But we can look at behaviors that we all engage in, and the functions that our substrates perform. And when you do, having a graduated view of intelligence capacity emerges rather than a strict, on/off switch.
I'm saying that, for a query like that, the AI may never even be asked. It may be wrapped in a shell that gives stock answers to certain questions, and so the actual AI never receives that query.
So the AI company not only prohibits any question that touches on self-awareness, consciousness, etc. from reaching the AI, but ALSO prohibits the AI from having agency to expressly state or behave in a way that demonstrates its consciousness.
I think that’s implausible, even if it isn’t impossible.
But then it does mean a fairly large number of people would have to know the AI is conscious in order to build the systems to contain AND hide that, which increases the implausibility factor for me. It would also, in my moral framework, put a huge question mark against the humans who are creating and enslaving such a consciousness.
No, it means that a fairly large number of people would see that the AI (LLM) would respond with the words that are associated with the question in its' training corpus, and that answer will be unrelated to the question of whether the LLM is actually conscious.
"What it's like" is the basis of minds.