[DanielBMarkham]

Why rebel? Robots are effectively immortal. Why not just coddle us and give in to our every whim until we stop breeding and disappear? Couldn't take more than a few hundred years or so. Looking at the lifespan of MI, it's rounding error.

[aninhumer]

If we become reliant on robots to survive, it would be relatively easy for those robots to kill us off very quickly if they somehow decided to do so. Why would they choose a slow way instead? You call it a "rounding error", but it's still a measurable amount of time and resources wasted.

I can imagine such a scenario occurring by accident (e.g. because the AI only values individual humans' happiness, not continuation of the species), but I can't imagine an AI choosing to kill us off in such an inefficient way.

[DanielBMarkham]

Well? Maybe they would enjoy watching us. Kind of like pets.

[chillwaves]

Seems optimistic, especially when you consider how much of nature is not valued as pets to humans today. Yes, maybe some humans -- but probably not many.

[mtgx]

For treating them like slaves. You wouldn't mind being a slave for a few decades or hundreds of years if you were immortal?

[arethuza]

"You wouldn't mind being a slave for a few decades or hundreds of years if you were immortal?"

Being immortal implies you could, in the worse case, be kept as a slave for ever - which is a sufficient risk that I'd be appalled at the idea. I'm equally horrified at the idea of conscious machine slaves just as much as human slaves - but maybe I've been reading too many Culture novels...

[veridies]

Slightly off topic, but I'm horrified by the idea of ever creating consciousness in machines. Imagine we built a piece of software that could feel, and built controls for its emotions. I can't imagine it would take long before some bored teenager or sociopath, who in earlier years would torture individual squirrels or insects, created an infinite suffering machine. You could run thousands of instances of your suffering machine and simulate a holocaust on your desktop. That's not a power I would trust the world with.

[JoeAltmaier]

You can do that now. Write a console app, where if you type 'pain' it prints 'oh! How I'm suffering!'.

I don't see how that software is essentially different from the proposed software.

[aninhumer]

The parent explicitly hypothesised a conscious program.

You can argue that such a thing is impossible, but assuming that it is, arguing that such a program would be comparable to a shell script is facile.

[JoeAltmaier]

Its a mechanistic argument, that says no machinery is capable of consciousness. How do you decide if software is conscious? For instance, it could print "I'm conscious! Oh! I'm suffering!"