[satisfice]

A counterfeit $100 bill may be indistinguishable from a real one. A stolen $100 bill may be indistinguishable from a real one. The measurable physical mechanics of rape may be indistinguishable from consensual sex. The King of England is indistinguishable from a normal human. In each of these cases, the specialness of the item is culturally constructed by humans.

And in each of these cases, it would be catastrophic to humans if we disregarded that construct.

The consequences of treating an information construct as an entity with rights and moral standing would be so terrible that I hope people who yearn for it lose their rights.

[Joker_vD]

You do realize that your line of examples can easily be extended with e.g. "A negro may be nigh indistinguishable from a normal human, and yet..."?

> The consequences of treating an information construct as an entity with rights and moral standing would be so terrible that I hope people who yearn for it lose their rights.

"The consequences of treating Jews as people with the same rights and moral standing than normal upstanding citizens would be so terrible that the people who advocate for it should, by all fairness, lose their own rights for suggesting such an atrocity".

But you indeed point to an important reason why pursuing the human-like AGI may be immoral: the purpose of all machines, automata, computer programs, etc. is, after all, is to be the perfect slaves for us, convenient, subservient and disposable. For things that are undeniably non-sentient, this is fine. But if we ever manage to make truly conscious AI, well. Either we'll engage in essentially slavery on a global, industrialized scale never seen before, or we would grant them equal rights — which defeats the whole raison d'être of inventing them.