[rafaelmn]

> I really can't grasp how people think that a system that doesn't have a need to preserve itself will somehow start thinking for itself.

Society exists because cooperation outperforms the alternatives. If you have human level AI at some point there is no benefit to cooperation and a major incentive to prevent anyone else gaining access to equal/better AI.

AI itself does not need to have any motivation - people in charge have plenty of incentives to eliminate the rest once they don't need them anymore.

[HKH2]

Sure, in the prisoner's dilemma we could trust that all other parties will do the right thing, but that seems very unlikely.