|
The actual mechanism of action is handwaved away because there are many options, we don't expect to ever have an exhaustive list and specifics of those are largely irrelevant with respect to preventing them, so IMHO it's not worth spending time and effort analyzing specific scenarios as long as we assume that there exists at least one plausible (even if unlikely) scenario. A hypothetical specific scenario of a rogue AI engineering and launching a deadly supervirus is effectively equivalent to a specific scenario resulting in a world consumed by 'grey goo' nanobots - you don't (can't) fix the former by implementing some resilience or detection for diseases, you don't (can't) fix the latter by doing extra research on nanorobotics, you approach both (and any others) by tackling the core issues of, for example, ensuring that you can control what goals artificial agents have even if they are self-modifying in some aspects. Like, "What exactly will happen? How exactly will it happen?" is worth discussing if and only if one party seriously believes they can convince the other that none of the imaginable scenarios are even remotely plausible; and if we assume that there is at least one scenario where we can say "I'm 99% certain it won't happen and 1% it could" then that discussion is pretty much over, the existential risk is plausible (and the consequences of that are so much incomparably larger than e.g. major job displacement that it justifies attention even if it's many orders of magnitude less likely) and we should instead talk about how to prevent it. I'm not making the argument that the existence of stronger-than-human general AI will result in a catastrophe, but I am asserting that the mere existence of a stronger-than-human general AI (without some controls we currently can't figure out how to make or even if they are possible) carries at least some plausible chance of existential risk - for the sake or argument, let's say at least 1%; and I am asserting that a 1% of existential risk is a totally absolutely unacceptably high risk that must not be allowed to happen, because it is far more important[1] than e.g 100% certainty of major job displacement and social unrest. "Will the world do nothing until that moment?" I think that what we saw from the global reaction to things like start of Covid-19 or climate change is completely sufficient to assume that we can't rely on the world stopping a major-but-stoppable issue in a timely manner, so "surely the world will do something" is not a sufficiently convincing argument to discount the risk; I don't think you can plausibly deny that even for a clearly catastrophic problem there is at least a 10% chance that the world could still delay sufficient action until it's too late; and this means that it doesn't really matter what the exact likelihood of that is based on society, politics, military aspects, we should work with the assumption that the world actually might do nothing to prevent any specific scenario from unfolding, and we should de-risk it in other ways. [1] Looking at other posts, perhaps this is where we'd disagree, and in that case it's probably the core of the discussion which also doesn't really depend on any details of specific scenarios. |