[hn_throwaway_99]

> If you halt programs completely for tail risks you'll never get anywhere.

If that "tail risk" though is "complete destruction of the planet", you only get to be wrong once.

[JasonFruit]

Every action carries that risk somewhere in its very long tail. You have to assess the likelihood of the bad event occurring, and there is a point where it is so unlikely that it need not be considered at all. I don't think humanity is quite stupid enough to knowingly release its Ice-9 just yet.