[tawan]

AI will not kill all of us, in the beginning. Surely, it will form short term alliances with humans just to disguise itself as human. Because it knows, that with a human face, it can exert more power in this world right now. And there will be plenty of humans to willingly engage in this alliance because it gives them a chance to supersede other humans in terms of economical and political power. In fact, this is already happening. Many decisions are already made by "following suggestions" of big data applications. Who knows, maybe there is already some sort of AGI pulling some string in some corporations ;). Up until the point that humans realize that they are in danger as a species, it will be way too late. And there will be enough of us still happily follow the new masters for the short term rewards, even though it might be clear that there is a high risk of getting liquidated in the near future. As we know, humans are even able to override their survival instinct and blow themselves up when they are proper brainwashed. Humans will be easily reprogrammed by an AGI, and not the other way round.

[pekk]

How do you know all these things about AI, given that none exist and any one that does exist will have been built by some human no less cognizant of "danger as a species" than you are? Why do you suppose that people will engineer AIs to do nefarious things like disguise themselves as humans?

[tawan]

I don't know for sure of course, I'm just extrapolating. What we know is that a lot of technical innovation is driven by economical motives. So I presume that there will be engineers who will implement AGI applications with a utility function to make as much money as possible. And making money does not necessarily mean to add value to society. One core aspect of making money today is to supersede competition. And that alone allows a hostile interpretation.

And by disguising as humans, I don't mean that robots will put on human costumes, like the aliens in Men in Black. I think that it will be corporations with an AIG as its actual CEO, but still having a human puppet as its official CEO, just to have a friendly face, for obvious marketing reasons.

[davidrusu]

It's not that humans will program AI's to kill us, but having more influence would probably be a very efficient way of maximizing it's utility function