[Jensson]

A computer virus that evolves and spreads and is too elusive for humans to eliminate would fit that scenario. I think the point is that as long as humans defines what the AI should do it will never be intelligent, it will only become intelligent when we lose control of it.

I think that was his point, not sure I agree with it but at least it isn't trivially wrong.

[rowanG077]

Isn't that just moving the goalpost to a higher level of abstraction? You could envision an AI were the only instruction is spread yourself. In that quest it could create a vastly more impressive society of AI agents with their own culture.

Humans have instructions too, which are rooted in evolution and biology. It's not at all clear to me how an AI that follows an instruction must, per definition, be considered unintelligent. That would imply Humans are unintelligent.

[soco]

Well even natural viruses are not considered technically "living", so there's a lot of confusion between what current science calls a thing and what the common understanding of that thing is.