[pekk]

How do you know all these things about AI, given that none exist and any one that does exist will have been built by some human no less cognizant of "danger as a species" than you are? Why do you suppose that people will engineer AIs to do nefarious things like disguise themselves as humans?

[tawan]

I don't know for sure of course, I'm just extrapolating. What we know is that a lot of technical innovation is driven by economical motives. So I presume that there will be engineers who will implement AGI applications with a utility function to make as much money as possible. And making money does not necessarily mean to add value to society. One core aspect of making money today is to supersede competition. And that alone allows a hostile interpretation.

And by disguising as humans, I don't mean that robots will put on human costumes, like the aliens in Men in Black. I think that it will be corporations with an AIG as its actual CEO, but still having a human puppet as its official CEO, just to have a friendly face, for obvious marketing reasons.

[davidrusu]

It's not that humans will program AI's to kill us, but having more influence would probably be a very efficient way of maximizing it's utility function