[wrs]

I find the argument to be essentially assuming the premise (if a system acts conscious, but you look inside it and don’t find anything conscious in there, then it can’t be conscious) but honestly I wouldn’t say I’m sure I understand the argument. Given my understanding, I thought this illustrated the circularity. But I see I’m wrong, because if you buy the premise then you won’t find the situation analogous.

That is, imagine there was a famous philosopher who insisted that cats can only be recognized by some non-computational mechanism, and any computational cat recognizer might “simulate” recognizing cats but could not be said to actually recognize cats. Then you build a neural network that recognizes cats, and they open it up and point out that nothing in it can recognize cats, so therefore it isn’t “really” recognizing cats.