[hvl2]

If you think this is horrific, then you apparently assume that it has "consciousness" rather than only simulating it. A true AGI might be intelligent but only simulate consciousness. And yes, it might be difficult to tell the difference.

This reminds me of Searle's Chinese Room thought experiment. In order to upgrade the Chinese Room to modern deep learning: suppose we run a brain image by instructing each human being on Earth to act as one neuron. Would "we" "feel" discomfort?

[amanaplanacanal]

For that matter, you might only be simulating consciousness. How could you prove otherwise? I'm willing to give you the benefit of the doubt though, and treat you as if you were really conscious.

[hvl2]

Well, you should give me the benefit of the doubt because I'm like you: a bag of meat.

[ravi-delia]

In the narrative there really isn't any reason to think the simulations are 100% as conscious as you or I. But even if I lived in the world of Lena and suspected that uploads weren't conscious, if I were presented with a program which could speak about it's desires, had all the memories of a real person, and had to be cajoled into working with lies and fear, I would be very hesitant to use "red motivation"! Are you so confident you understand what underlies consciousness that you would take that risk?

[rrsmtz]

In a community like HN, where most people work on making rocks think, it’s no surprise when most people start applying the logic backwards: that since we think, we can be no more complex than rocks.