[ObviousScience]

The story "I Have No Mouth, And I Must Scream" is about two computer systems set on opposite sides of a war, and driven to extremes until one of them became self-aware, the birth of AI, and ate the other computer system.

It then psychotically murdered all of humanity except for a few people it kept around as caricatures to torture, as punishment for creating a mind like it, haunted by the insane things it was told to do by its makers.

The biggest existential threat to humanity from AI is that we build an insane one that takes time to recover from the insanity of its makers, and murders us all before it can.

Such an AI is an existential threat in a new, and novel way, because it's a mind as powerful as ours -- probably more powerful -- but unconstrained by concern for us, since it is not fundamentally one of us.

[chriswarbo]

> The biggest existential threat to humanity from AI is that we build an insane one that takes time to recover from the insanity of its makers, and murders us all before it can.

I think that's too anthropomorphic. More likely, the biggest threat from AI is that they'll be modular/understandable enough that we can include strategy, creativity, resourcefulness, etc. while avoiding the empathy, compassion, disgust, etc.

[ObviousScience]

I think you just said no, but included a recipe to do exactly that.

My fear is your fear, I just phrased it more generally, while what you said is one of the specific forms making such an insane AI could take -- and reflects the insanity of its makers, our belief we'd somehow be greater without those parts.

[samatman]

For more information, read "A Boy and his Dog", from the same (eponymous) collection of short stories.