| >Diversity and independence of AI Agents mitigates against the Terminator-scenario danger I don't see how this is the case. The AI's still are likely to be more similar to each other than they are to humans. And they could choose to cooperate to kill humans, and then split the resources of the world among themselves. Having multiple AIs is no guarantee of safety! Unless those AIs want to keep humans alive and care about our future. The second assumption is that the world of the future will be stable. It could be that a small technological advantage is enough to win against every other group. E.g. if your AI can invent better nanotechnology first, or discovers some entirely unknown technology. It's unlikely that a random set of superintelligences would be perfectly evenly matched. Some would be much larger, smarter, or have access to more resources, or have a head start in time. Lastly and most importantly, is the idea that superintelligence won't be built quickly. Once you have the first smarter than human AIs, it may be only a matter of weeks or days before you have superintelligence. That's the whole idea of an "intelligence explosion". The smarter than human AI's can do AI research and improve themselves, and then improve themselves even more, then even more. Not to mention plugging more GPUs into their brains to make them smarter. So it seems very likely to me that once you have a lab that solves the last technical problems in creating AGI, a single superintelligence will rapidly emerge from that. The superintelligence will then be able to get whatever it wants, because it's so much smarter than humans. It doesn't need to be "wired into" anything if it's 10x smarter and faster than the best human hackers. And if it's 2050 and everything and your toothbrush is connected to the internet. |