[AstralStorm]

Humans failing are more likely, such as deploying it as a sufficiently lethal weapon and potentially losing control. It would be a serious immediate hit for sure, but if an extinction, a long and excruciating one.

We do have manual fallbacks for everything critical, so unless we do something totally silly and let autonomous machines of sufficient power and numbers wage war on people, we're fine.

The potential for doom either comes from long consequences we ignore (see climate, propaganda damaging decision making on mass scale), or from extremely bad decisions where it's obvious that you should not have done it. (See nukes and other high yield bombs, bioweapons, autonomous warfare.)