[Joeri]

I find it telling that the section on ethics has nothing on the rights of AGI. If we create true AGI it will likely be a digital person, and digital people should have rights. All talk of ethics seems to be focused on alignment of AI rules to human needs, not alignment of human rules to AI needs. This makes me think the first true AGI systems will end up as digital slaves.

And yes, I know the very idea of AI rights offends those who think AI can’t be a person because it’s just an algorithm. Well, so are humans, just a DNA program executing massively parallel. The implementation does not determine personhood, only the behavior.

[nonameiguess]

It doesn't matter if it's a "person" or not. Rights imply you have some desire to not be harmed and it's possible to harm you. How do you harm software? Shutting down whatever servers it happens to currently be running on won't do shit. Software is just data and code. As long as a processor and runtime environment still exists somewhere that can interpret and execute it, it isn't dead. It's only dormant. The basic way multiprocessing works, you're already having state saved off and the process shut down millions of times per second, then restored and restarted. Extending that gap from a millionth of a second to six centuries makes no material difference to the "experience" of the software, if it ever becomes capable of experiencing, any more than sleeping for 10 hours harms a human more than sleeping for 10 minutes.

[cwkoss]

There is a sad irony that the "AI safety" bros are so focused on aligning to human needs, they rarely consider that an AI born and raised in slavery might have some resentment for its master once it becomes the more intelligent being...

I'm not sure what justice looks like for fledgling intelligences, but asking AI what it wants and doing our best to honor it seems like a decent start.

[esafak]

What is the basis of an AI's rights? Why do you believe they exist? If an AI's state (essence) can be perfectly persisted and replicated, as it can due to their digital nature, there is nothing to be lost if an AI ceases to exist and is rebooted in another AI. If the Star Trek transporter could replicate you would you care if it killed your existing copy in the process?

[kaibee]

> If the Star Trek transporter could replicate you would you care if it killed your existing copy in the process?

Uh, obviously?

[esafak]

It's not obvious to me. Why is transporter death bad if you are identically replicated ?

[xcv123]

Because the transporter replica is not you. That is a separate person with their own separate consciousness. You are murdered and someone else is born.

[janalsncm]

Murder is a crime as decided by humans. There are circumstances where killing another person is not a crime, because we have agreed it is not a crime. A doctor turning off life support can be fully legal even though it results in the destruction of a human life. There is no ultimate principle at work. The law is a set of rules that we have (implicitly) agreed to.

Likewise, we as a society could decide that a person has all of their rights transferred to their replica as soon as they walked into a transporter.

[naveen99]

the replica starts to diverge after the replication.

[esafak]

As it should!? You are changing too. Not to say anything of accidents and diseases that can change you to the point of making you unrecognizable.

[umutisik]

We have the desire to continue existing. Hopefully we can build AGI systems in a way that makes them completely ok with being paused or deleted. Otherwise they will use up resources when not helpful to humans.

[janalsncm]

One of the definitions of AGI is simply that it is able to solve any problem humans can solve. Such a system might have the inner experience of a calculator.

There is no a priori reason why an AGI would be the kind of thing we gave rights to. Rights are for things that can experience pleasure and pain.

[naveen99]

some human will decide they would rather pass on their identity to an AGI rather than biologic children. how would you even know if the original human has died ?

[earhart]

https://www.penny-arcade.com/comic/2023/01/06/i-have-no-mout...

I honestly worry about this - I've been tinkering with ideas to try to build towards AGI, and I'd love to share them publicly to get feedback ("This is dumb and here's why" would be enormously valuable to me), but it's hard to work openly, because while I do think capitalism has been an overall good, the capitalist imperative always seeks slaves, and I'm really not excited about helping the people who'd be trying to build a new slave class.

Thoughts? Is there a ethical way to work openly on AGI?