[squeaky-clean]

> If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines.

But the AGI in this thought experiment doesn't want human-sized parcels of matter. It wants paperclips. And by proxy, it wants any matter which is not a paperclip and can be transformed into a paper clip.

> A human can collect many times his own mass per day.

And if the AI can collect many thousands of humans worth of mass each minute, it doesn't care about the humans or their ability collect things. The AGI simply wants to optimize the number of paperclips in it's collection, it doesn't care about number of humans working for it, or that it has displaced or used to make paperclips, unless they affect how many paperclips it can acquire.

I like the ants metaphor another commenter made and want to continue with it. When digging a pit / quarry, or clearing land or whatever.... Well, ants can lift several times their bodyweight and dig some deep holes. But do we use the ants to dig for us? Or do we bring in a backhoe and not care whether ants exist in the field?

Also, your scenario still agrees with the thought experiment. It's not that an AI will intentionally kill all humans. Just that a super-intelligent AI designed without consideration for human values will not likely develop those values on it's own, and therefore can be just as dangerous as an AI designed to be malicious towards those values. In your example, a machine AI has still enslaved the entire human race, and it's original instructions were not at all malicious or related to destroying humanity.

[Eliezer]

This is an excellent analogy! "Ants can carry fifty times their own weight. Why would humans wipe out the ants on a construction site instead of using food rewards to train them to help remove dirt?" Answer: backhoes are faster at moving dirt, the time lost by carefully moving the ant colony exceeds any gain from their helping to move the dirt on one tiny section of the project, ants are hard to trade with and you wouldn't be sure they were doing the job right, and humans don't have a terminal value for protecting the ants.

And that's without a button that disassembles the ants and uses their atoms to build part of a diamondoid backhoe that's much better at moving dirt.

[wapapaloobop]

The problem with your scenario is that it equates humans with animals, but the difference that matters is between people (humans, AGIs) and non-people (primitive replicators, animals).

We don't commit murder to get at rocks despite the fact that weren't designed with 'consideration for human values'. We learn our values, just as an AGI would, not through programming or lack thereof, but through parenting, education and culture. These latter things aren't optional since minds can't exist without them.

[outworlder]

> We learn our values, just as an AGI would

Yes, but there's a problem. An AGI would be a completely alien mind. It does not have to have the same values(or any values we could relate to), just like you wouldn't expect a super intelligent spider to have any morality, no matter how good the parenting was.

[wapapaloobop]

An AGI would have to begin with the same values because without them he couldn't learn anything. Our values draw on the surrounding culture and an individual that grows up outside of any culture (e.g. a feral child) isn't capable of functioning. There may be different cultures but there is no alien culture. Thus there could be no alien mind.