[satisfice]

The author is basically saying we have to surrender our humanity as we understand it because otherwise we will lose outright to AI. Hard no from me.

His section on objections leaves out civil war. But his proposal is dead on arrival.

[cyberneticc]

No need for everyone to do it, but some people will certainly want to merge with AI. My main arguments are that AI of sufficient complexity ("synthetic minds") deserves moral standing with or without consciousness, and that our best shot at long term alignment is to see them as equals and find mutually beneficial arrangements with them.

[satisfice]

My position is that we have no power to grant moral standing to non-humans. Moral standing is held by humans relative to other humans just because they are humans. It is not earned or merited.

Moral standing is not a gift or a privilege, it’s a necessary heuristic for the avoidance of perpetual war. It is a prerequisite for more than one human to live together in peace. Moral stadning necessarily involves the sharing of power. That’s what it’s all about.

When you extend moral standing to nonhumans, for instance many people say there is a thing called “God” that they insist has moral standing. Then you shift power relationships in potentially catastrophic ways.

As a thought experiment, consider that we might be two bees arguing about whether flowers, or humans, have moral standing. Bees can only grant it to each other, though, as a survival heuristic, same as us.

Imagine an angel creates a human and wants his fellow angels to treat it as if it had rights. The most he’d be able to say is if you harm this human I will smite thee. By doing so he is using his power to support his personal interests. But he cannot manufacture power as easily as he creates a human.

Consider why people living in Washington D.C. do not have full voting rights and never will.

Consider what would happen if we allowed rich people to arbitrarily manufacture political power by spinning up servers and allocating disk space to create voters.

Of course you can manufacture power by making a dangerous weapon, then you can give that weapon a semblance of agency, and then dare anyone to deny to this weapon the things that it thinks it desires. That’s sounds like Frankenstein. That’s not really moral standing, that’s terrorism.

Nothing you can create ever has moral standing— except a baby, and that is merely a heuristic that we see can fail rather easily, as during the Holocaust, or Isis persecuting Christians.