[notahacker]

The relevant probability isn't "probability of superhuman AI being developed this century" but the probability my marginal dollars donated to this specific research org will lead to positive developments in superhuman AI that would not have happened if the org did not receive my money. That one's a much higher bar to cross especially with EAs having satisfied themselves that other marginal dollars definitely are efficiently saving lives. Hence "longtermism", which is a quite explicit argument conceived within the EA movement to reconcile the low probability of stuff like a particular AI research paper saving any lives whatsoever by arguing the utilitarian maths still work if you can multiply that low probability by a large enough future of humanity safeguarded.

Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...

[kalkin]

Depends on the question. I agree that the argument for a random person giving money to try to reduce AI risk seems dubious, and touched on this in another comment - https://news.ycombinator.com/item?id=32394429 - although I'm not up to speed on the "longtermism" debate as such.

But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).

[notahacker]

Agree on both points. (There's a perfectly good reason for EAs with a computer science skillset to want to work in AI with a focus on safety; it's a well compensated industry where they can probably solve important real world problems more reliably than AI researchers less concerned about the ethical implications of what they do even without any sort of singularity, it just the movement seems to have invested a lot more in talking about longtermism than anything else lately, and Yudowksy's always had some very... individual lines of reasoning)