[JoshTriplett]

> Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

This is a huge handwave, and one resting on multiple incorrect assumptions.

Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

[est31]

> it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.