[robbensinger]

I agree that "that's Pascal's wager!" isn't a reasonable response to someone arguing that, say, a 1% or 10% extinction risk is worth taking seriously. If you think the probability is infinitesimally small but that we should work on it just in case, then that's more like Pascal's wager.

I think the whole discussion thread has a false premise, though. The main argument for working on AGI accident risk is that it's high-probability, not that it's 'low-probability but not super low.'

Roughly: it would be surprising if we didn't reach AGI this century; it would be surprising if AGI exhibited roughly human levels of real-world capability (in spite of potential hardware and software improvements over the brain) rather than shooting past human-par performance; and it would be surprising if it were easy to get robustly good outcomes out of AI systems much smarter than humans, operating in environments too complex for it to be feasible to specify desirable v. undesirable properties of outcomes. "It's really difficult to make reliable predictions about when and how people will make conceptual progress on a tough technological challenge, and there's a lot of uncertainty" doesn't imply "the probability of catastrophic accidents is <10%" or even "the probability of catastrophic accidents is <50%".