[ziotom78]

I believe that what we can reasonably expect in the future is machines that act as if they experience consciousness. But the fact that this is true consciousness is highly debatable, as the thought experiment of the Chinese room [1] explains.

[1] https://en.wikipedia.org/wiki/Chinese_room

[OrangeMusic]

Not sure if it's really "highly debatable" that an entity that appears conscious is therefore conscious.

The Chinese room thought experiment is obviously flawed to me. It's not the "computer" that is conscious, it's the running software.

[mjh2539]

The point of the Chinese room is to show you that the guy who has memorized the standard responses to various questions in Chinese does not in fact know Chinese. He's mindlessly parroting things.

[RugnirViking]

what would it mean to "know" a language? one could imagine a series of increasingly complicated questions designed to relate various words, subjects, associations, maybe culture and history. But one could just as easily imagine a sheet of paper with the answers on answering them - and our friend answering them, in seemingly fluent Chinese. Im not convinced there is any experiment one could perform to convincingly separate the two (without removing the man or his translation aids from the box) - thus does your idea of "knowing" exist?

An AI cannot be removed from its box, because it doesn't have one. It really does have enough information inside of its essence to reply. In fact, that information makes up its essence.

I agree that in some sense their knowledge is distinct and of a different character to human knowledge. But what that means conceptually or morally exactly is very complicated, and cannot be dismissed easily