[NumberWangMan]

Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment>).

An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.