[JamesBarney]

Just for context AI is ~10% of EA donations.

I'm not a huge AI risk guy, but it doesn't seem irrational to me. It doesn't seem unreasonable that you'll have a 5% chance of > than human IQ in the next 100 years. Assuming a 5% risk of it going rogue you end could end up in a pretty terrible situation.

[HideousKojima]

Given how seriously mamy rationalists take certain things like Roko's Basilisk I have a hard time imagining that AI risk charities use their funds effectively (the E in EA, after all). That's assuming I agree with your 5% estimate (I don't).

[ben_w]

I have literally never encountered someone who really takes Roko's Basilisk seriously.

Other than the person who came up with it, I've not even heard of such people.

I don't doubt they exist — I've met people, people are dumb and believe dumb stuff even when they're smart — but I've never once encountered that.

[HideousKojima]

>Other than the person who came up with it, I've not even heard of such people.

Yudkowsky took it seriously enough to ban it from discussion on LessWrong as it was an "information hazard," so I think you might be a bit naive about how seriously many rationalists took/take it.

[ben_w]

“an incredibly costly way for me to learn that deleting a stupid idea is treated by people as if you had literally said out loud that you believe it, but Roko being right was never something I endorsed, nor stated.” - Yudkowsky

https://web.archive.org/web/20140626092902/http://www.reddit...

So no, not naïve.

[HideousKojima]

He didn't just delete a post about it, he outright banned discussion of it as an information hazard, so you'll forgive me if I don't believe him.

[JamesBarney]

I spend a lot of time in EA adjacent spaces and I've read the Basilisk there once, but I've seen it in several popular magazines reporting about EA 4 times. I really don't think it's a top concern at Givewell.

I've never seen seen a survey of AI experts where less than 50% thought we wouldn't have AGI by 2100. Why are you so confident that AI is an order of magnitude less likely in the next 100 years then the vast majority of experts in the field?

https://nickbostrom.com/papers/survey.pdf

[cauthon]

Are they AI experts? The authors of the paper you linked are philosophers, not CS. Yudkowsky doesn’t have even a high school education, and I’ve never seen any programming or ML work by him.

Frankly, the “research” in this area all appears to be science fiction that takes itself too seriously.

[JamesBarney]

Two of the groups they surveyed were AI researchers, it break it does by those groups, and the majority of those groups think AGI will happen in 100 years.

1. EETN: Members of the Greek Association for Artificial Intelligence (EETN), a professional organization of Greek published researchers in the field, in April 2013. Ca. 250 members. 2. TOP100: The 100 ‘Top authors in artificial intelligence’ by ‘citation’ in ‘all years’ according to Microsoft Academic Search(http://academic.research.microsoft.com) in May 2013. We reduced the list to living authors, added as many as necessary to get back to 100