[tendom]

> A more pessimistic outcome is that powerful AI arrives before we've really thought through the moral and ethical issues, and it possibly destroys us, or is developed and exploited by military interests to who knows what outcome.

But it's not a very likely one. Spontaneous AI is almost impossible, primarily because we design and control the substrate. AI is very very hard, and when we get there, it will be very unlikely that it turns on us, or at least not for a millennia or two. This fear is a result of some seriously good sci-fi stories, but not a lot of logic or fact,

[Houshalter]

Once we get smarter than human AI, it can improve itself or make even better AIs, which make better AIs, and so on. It could very rapidly become far, far more intelligent than us. A being that powerful could do whatever it wants. Manipulate humans, hack the entire internet, design nanotechnology, etc. It's very unlikely that it would have human morality, and so wouldn't care about eliminating us to turn the Earth into a giant supercomputer, or just prevent us from ever creating competing AIs.