[NumberWangMan]

It's not hostile in the sense of hating us, it's just a maximizer, and when you're smart enough and try to maximize one thing (or some function of multiple things), anything that's not in your utility function gets sacrificed. Humanity maximized for cheap energy, and for a long time, the carbon composition of the atmosphere was not in our utility function. Now that's clear we might be killing ourselves by doing it, we're working it in, but if it turned out that CO2 was only dangerous to, say, several species that we didn't care about, it's very likely we'd just accept it as the cost of progress.

Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore.

It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.

[anony23]

The artice is literally about how this will lead to a hostile superhuman intelligence that will wipe out humanity. It's not about AI tooling as an accelerator, which is a valid concern that imo can't be avoided.

[NumberWangMan]

Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment>).

An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.