[woeirua]

Ted Chiang's argument basically boils down to: I won't recognize an AI as conscious until its desires/behaviors reflect situations that I'm already personally comfortable with. I personally think most humans are incapable of recognizing consciousness in creatures that do not mimic human emotional states. Most people would say their dog is at least somewhat conscious. No dog is capable of vocalizing how it feels, but we all recognize fear and happiness in dogs. Claude can write how it "feels" but we immediately dismiss it as hollow mimicry.

I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.

[krapp]

An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.

But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.