[blakeweb]

For those interested in another very different analysis of the evidence for and against superintelligence being a worthwhile concern, here's Holden Karnofsky, executive director of GiveWell and the related Open Philanthropy Project:

- Up until 2016, Holden held the opinion that it was not a worthwhile concern. He describes why he changed his mind here: http://www.openphilanthropy.org/blog/three-key-issues-ive-ch...

- And he outlines his current arguments in favor of treating it as a significant concern here: http://www.openphilanthropy.org/blog/potential-risks-advance...

In that lengthy write-up, he addresses both the shorter-term risks from things like misuse of non-superintelligent AI, along with longer-term risks from superintelligent AI.

[mcguire]

What does he suggest doing about it?

That's always my question for this sort of topic. It seems to me that the only reasonable behavior for someone accepting SAI as a legitimate concern is to advocate defunding all AI researchers, as well as criminal punishments for AI research.

[blakeweb]

There's a section on Tractability--he can speak to his opinions better than I can: http://www.openphilanthropy.org/blog/potential-risks-advance...

TL;DR: He talks about several avenues of technical and strategy research that seem plausibly very useful and are not currently being pursued by more than a handful of people in the world. Many of these currently-engaged people are precisely the folks the author of this post disparages for being weird or insular.

One of the avenues of technical research he mentions is "transparency, understandability, and robustness against or at least detection of large changes in input distribution" for AI/ML systems. In other words, technical research to produce methods capable of reducing the likelihood of advanced systems behaving in severely bad, unexpected ways.