[dTal]

This has a Pascal's wager sort of feel. The possible negative consequences of a rampant AI are infinite, so we all ought to take it Very Seriously. The trouble is that there are lots of other scenarios with unbounded negative value - for instance, tight centralized control of technology supports a stable dystopia that keeps nearly all humans in a perpetual state of suffering. It's not at all clear, when you add it all up, that "prevent 'misaligned' AIs from being deployed at all costs" is the correct strategy to minimize risk.

[circuit10]

The difference to Pascal’s wager is that with that the probability is vanishingly small, but given current progress the probability of AI causing an existential threat is actually quite high. You can’t just call anything that has very bad negative consequences Pascal’s wager

Video about this: “Is AI Safety a Pascal’s Mugging?” https://youtu.be/JRuNA2eK7w0

[circuit10]

I sort of mixed up “wager” and “mugging” here but you get the point

[dTal]

Not at all, "Pascal's mugging" was exactly the term I would have used, had I remembered it :) That video is a very good reply. One response that comes to mind is that there is an implicit subtext behind any discussion of AI safety that we must do something about it, and soon. A good "anti-god" for his payoff matrix would be "the act of calling for AI safety will inadvertently be used as political capital by those wishing to be the gatekeepers of powerful technology" - the chances of that are quite high, and the negative consequences quite bad, if not necessarily apocalyptic.