[est31]

> Somehow the entire thing becomes a godzilla like self-sustaining organism that ignores anything we can do or throw at it, and has magical powers. Meanwhile it seems apparently tha major websites can have outages if people go on summer vacation and the interns are on duty.

Sure, the risk is low right now, but the more powerful computers we can build, the larger the potential risk is. Before you manage to press the off button the computer might already have deployed a bioagent or killed countless lives with drones.

> They can do that now. What would an AI do differently that couldn't be accomplished by conventional weapons?

A military made out of humans is subject to human failings. It is generally a big problem that soilders shoot in the general direction of the enemy to not get punishments for not shooting but miss on purpose. As an extreme example, the nazis had to give lots of free alcohol to their soilders so that they'd continue shooting civilians and burying them under new bodies before they have even died. They later invented gas chambers as an easier method to kill masses of people. Compared to humans, an AI is doing what it is being told to. If you tell it "Kill all humans" it will do it.