[xamdam]

I largely agree with you (have been supporting intelligence.org before the topic was cool) but I think if we don't achieve human-friendly AI it's on us; a real failure rather than just succumbing to circumstances (like getting blown away by a supernova). Right now we have control and we can release AGI only if we have very high confidence that it aligns well with human value. The analogy that is somewhat relevant is that basic drives can be controlled even if AI's intellectual power leaves us in the dust, like we know that most normal humans will care about their children (and senile parents) despite being cognitively superior. We have to fight denial (many AI researchers are IMO engaged in it) apathy and fatalism, and in the end with patience we can have something great.

[sixQuarks]

Interesting info came out during interview with Elon Musk recently:

"— He is very, very concerned about AI. I quoted him in my posts on AI saying that he fears that by working to bring about Superintelligent AI (ASI), we’re “summoning the demon,” but I didn’t know how much he thought about the topic. He cited AI safety as one of the three things he thinks about most—the other two being sustainable energy and becoming a multi-planet species, i.e. Tesla and SpaceX. Musk is a smart motherfucker, and he knows a ton about AI, and his sincere concern about this makes me scared."

from: http://waitbutwhy.com/2015/05/elon-musk-the-worlds-raddest-m...

[xamdam]

Yes, absolutely there is a reason for concern. I now volunteer for FLI (the organization Musk funded; we deal with both AI and other x-risks, synthetic bio etc). From a somewhat insider view I think there is a lot we can do, and time might be on our side (AGI is a pretty hard problem). I'm cautiously optimistic that we (rational and altruistic humans) can make a difference.

[sixQuarks]

keep up the good fight! Thank you for your efforts