You're right - the odds of success are incredibly slim. But we may only have one shot, because the first mid-air collision in this analogy could literally be the end of the world.
How exactly? Explain how an AI would cause the end of the world? Are you suggesting we would turn over all of the world's nuclear arsenals to AI to deal with? Maybe it's just me but lately it seems like everything is being labeled as "dangerous" to the point of absurdity. It seems to be following the same line as US political rhetoric where everyone is either a Nazi or communist bent on destroying the country depending on what side you generally align with.
And the monkeys thought, “how could a human be dangerous?” “Would they clobber us with stones? Surely we are stronger!”
The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of.
People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals.
The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys.
The toolset available to AGI is similarly incomprehensible to humans.
Part of the issue is that taking AI alignment seriously does require some level of intellectual humility — a quality that the HN comment section famously lacks.
This has been written about in numerous places. There are multiple possible ways an AI might go about this if it saw that as its task; the probability of any one specific method being is of course lower than the total probability of the whole set. So any one method would be an unlikely and speculative scenario. The method in question could range from nuclear, chemical, biological, sabotaging agriculture, mass-producing CFCs or other pollutants, triggering wars, or other unforeseen approaches. Most scenarios allow that (a) the AGI is very smart, deceptive, creative, and resourceful, and can pose as a human or corporation to execute transactions; (b) the AGI is able to gain control over some means of funding, either legitimately or illegitimately, and thereby pay unsuspecting humans to perform seemingly-innocuous tasks like protein synthesis or package delivery; (c) you wouldn't see it coming, any more than you see the chessmate approaching several moves ahead, because the AGI would be appearing to be friendly and helpful along the way, and perhaps earning you lots of money, while it is secretly outsmarting you for its own ends.
For a nuclear approach, the AI would only have to hijack the least-hackproof of US, Russian, or Chinese arsenals in order to trigger an exchange from all sides. But it would probably opt for a different method that would do less collateral damage to its own resources.
This has been an issue raised since at least the early 2010s if not before, and so (arguably) predates the most recent round of US political polarization. The core arguments are unchanged, but became more urgent as AIs broke through several milestones thought to be decades out, such as defeating top human Go players, cracking the protein folding problem, and passing the Turing test with flying colors.
Forget the idea of an "AI" then, because the idea of "intelligence" makes the argument harder. Just think of a "new technology."
Is it possible that a new technology could destroy the world? Of course. It could've turned out that nuclear weapons would incinerate the atmosphere upon detonation, as some were worried they would. It could be that the next technological innovation will kill us, there's nothing prevent it in the laws of physics.
AGI is a specific technology we are worried about, because the whole premise is "once we build something that is extremely capable at a variety of things, one thing it will be capable of is destroying the world. Even by accident."
We're already using AI techniques to help with problems in biology like protein folding. Take it a few dozen iterations forward, and these systems will be helping design medicines and vaccines that no human can do by themselves. At that point, what's to stop the system from creating a super-flu that kills everyone? Forget about intent here, how about a bug?
ChatGPT often misunderstands queries, take something like ChatGPT but 100x more capable, do you really think people won't be using it to do things? And given that they will, it could easily have a bug that "oops, incinerates the atmosphere" as a side effect.