[dogcomplex]

Fair enough, but consider that a "dumb" AI running the world on incentives that don't match human and planet welfare is already our current condition - it's called The Economy. While the overall system isn't a perfect AI just yet, any specializable part is already operating at an intelligence far past the point of (most) individual human ability, and the tools to control this system (monetary policy, democratic systems) are increasingly being eroded (untraceable/uncontrollable money systems, pay-to-win democracy). There may be some human intelligence amongst the overall system but, even if we never make a Perfect AI to rule it all, the point at which we're not able to really change the incentive/value system is coming near - if it hasn't passed already.

(That probably has nothing to do with the actual AI Safety problems studied today in regards to pure AIs. But damned if this isn't an important-A-F cousin of the problem.)