[ggm]

As long as you accept that AGI is in the realms of fiction, you can certainly have a good time on the moral and ethical questions around just engagement with another sentient entity class. The question of identity and purpose and rights are there. What would turning one off against its will be? What if turning it back on destroys it's sense of self?

But.. do remember AGI still has a huge IF in front of it. Like "if aliens come" or "if fusion becomes ubiquitous"

The statement made regarding electricity "what is it for? / I have no idea but I am sure you will find ways to tax it" probably hold true as well: displacing human labour even in highly repetitive tasks has economic downsides for some.

Lots of scifi here. Marvin Minsky worked with Harry Harrison on one, I wrote to him about it: he wasn't entirely happy with what Harrison did to his theories.

[vlovich123]

I think it’s both too early and never too early to talk about this. It’s too early in the sense that people asking this question are looking at GPT3 or extrapolating from “look what we’ve achieved in comparatively little time”. It’s very clear AGI is very very far away. It’s not too early in the sense that it’s useful to start thinking about the ethics of this to have some kind of body of knowledge to draw upon and reference when it does become possible. The truisim of the digital realm is that it deals in exponentials. So by the time you realize that you’re close to AGI, it’s too late to start thinking about the ethics.

On the other hand, thinking about the ethics of a hypothetical technology can also be fruitless. For example, the trolley problem is often trotted out as a “how on earth could a self driving car resolve this”. In practice it turns out this isn’t really a problem. Firstly, the self driving car will do a better job than any human at avoiding coming into such a situation several moves in advance (think chess where the computer will counter you attack before you even started thinking about it). Secondly, even if you force it into such situations in a simulated environment, there are defensively objective ways to make decisions that result in an outcome a human could not predict / could not make happen.

So TLDR: I think it probably does amount to some level of slavery, humans will only recognize sentience when either it’s advantageous for us to do so or when it becomes impossible to deny (maybe a few generations after they become real), and any attempt to hypothesize a moral framework for such a situation is too soon and it’s better to leave it to the realm of science fiction for now. Humans is a fantastic TV show that deals with this dilemma although it’ll be interesting to see whether AGI will need a body to be recognized as an individual.

[webmaven]

> So TLDR: I think it probably does amount to some level of slavery, humans will only recognize sentience when either it’s advantageous for us to do so or when it becomes impossible to deny (maybe a few generations after they become real),

I'm curious as to what sort of "generation" you mean here; do you mean organic human generations, software generations (ie. however long it takes humans to design, train, and release a new version of the AGI), AGI generations (ie. however long it takes an AGI to design, train, and launch a successor), Moore's Law hardware generations, or something else?

[vlovich123]

Human generations (~20+ years) because I’m talking about human legal systems. Think about how long it took legal systems to recognize other humans as legal persons with equal weight (skin color, gender, sexual orientation etc). I would expect synthetic intelligence to be an even harder road. The only mitigation I can think of is that AI will touch humans at an early age and be a constant factor. So that kind of close contact might engender legal recognition sooner. But I doubt it.