The idea that I'll use human bodies if I need a bunch of atoms makes as much sense as the premise of The Matrix i.e. harvesting humans for energy production.
I don't think the line is meant to be taken literally. Consider the myriad of environmental factors humans need to remain alive. Any of those factors, e.g. the atmosphere's current makeup, could prove uninteresting to an AI.
I read that the first draft was using a much more reasonable concept - the Matrix needed our brains as CPUs to run itself. No idea if it's true, but certainly a lot less implausible than what came out in the movie.
The power source thing is an instance of unreliable narration. Their account of what happened is pieced together and passed around like the mythology and religion that underscores the movie.
I think the concern is more that a sentience that is many orders of magnitude more intelligent than humans simply won't care about us. How do you feel about ants?
Any person (biological or artificial) who is capable of mass murder is unlikely to balk at slave owning. If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines. A human can collect many times his own mass per day.
> If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines.
But the AGI in this thought experiment doesn't want human-sized parcels of matter. It wants paperclips. And by proxy, it wants any matter which is not a paperclip and can be transformed into a paper clip.
> A human can collect many times his own mass per day.
And if the AI can collect many thousands of humans worth of mass each minute, it doesn't care about the humans or their ability collect things. The AGI simply wants to optimize the number of paperclips in it's collection, it doesn't care about number of humans working for it, or that it has displaced or used to make paperclips, unless they affect how many paperclips it can acquire.
I like the ants metaphor another commenter made and want to continue with it. When digging a pit / quarry, or clearing land or whatever.... Well, ants can lift several times their bodyweight and dig some deep holes. But do we use the ants to dig for us? Or do we bring in a backhoe and not care whether ants exist in the field?
Also, your scenario still agrees with the thought experiment. It's not that an AI will intentionally kill all humans. Just that a super-intelligent AI designed without consideration for human values will not likely develop those values on it's own, and therefore can be just as dangerous as an AI designed to be malicious towards those values. In your example, a machine AI has still enslaved the entire human race, and it's original instructions were not at all malicious or related to destroying humanity.
This is an excellent analogy! "Ants can carry fifty times their own weight. Why would humans wipe out the ants on a construction site instead of using food rewards to train them to help remove dirt?" Answer: backhoes are faster at moving dirt, the time lost by carefully moving the ant colony exceeds any gain from their helping to move the dirt on one tiny section of the project, ants are hard to trade with and you wouldn't be sure they were doing the job right, and humans don't have a terminal value for protecting the ants.
And that's without a button that disassembles the ants and uses their atoms to build part of a diamondoid backhoe that's much better at moving dirt.
The problem with your scenario is that it equates humans with animals, but the difference that matters is between people (humans, AGIs) and non-people (primitive replicators, animals).
We don't commit murder to get at rocks despite the fact that weren't designed with 'consideration for human values'. We learn our values, just as an AGI would, not through programming or lack thereof, but through parenting, education and culture. These latter things aren't optional since minds can't exist without them.
Yes, but there's a problem. An AGI would be a completely alien mind. It does not have to have the same values(or any values we could relate to), just like you wouldn't expect a super intelligent spider to have any morality, no matter how good the parenting was.
An AGI would have to begin with the same values because without them he couldn't learn anything. Our values draw on the surrounding culture and an individual that grows up outside of any culture (e.g. a feral child) isn't capable of functioning. There may be different cultures but there is no alien culture. Thus there could be no alien mind.
People use dead mammals for things like leather, bones, fat, feeding other animals/humans, science experiments, ornaments/status symbols, tallow, etc which you can't mine.
The AI might be in a hurry (eg be in competition or war with humans other AIs) and trade off long term cooperation prospects for getting an upper hand in a fast paced emergent situation.
I'm not sure getting sorted into two piles, beasts of burden and recycled biomass, is really a win scenario for humans either.
If you consider the extent to which humans have shaped the Earth's environment to suit them, we massively favor animals we kill and eat over the ones we don't. You don't have to be part of the machine to be part of a congenial environment for the machine.