If only AI safety research had a mechanism this clear. "We have proof that building the machine will kill everybody, so get to work making a provably safe version."
Except that you have the logic backwards. It's an argument that something ("safe" general purpose AI) can't exist rather than that it has to.
People want AI to be able to do every good thing but no bad thing, which is impossible twice. First because false positives and false negatives trade against each other, so a general purpose AI which can do anything approximating all the good things is going to have the bias leaning heavily towards being able to do things in general and therefore being able to do many things that are bad. And second because "good" and "bad" aren't things that anybody can agree on and then some people will demand that it must do X while others demand that it not do X (e.g. "help the rebels win the war"), which means someone is inherently going to be unsatisfied and it's not a thing that can be sensibly regarded as everyone working towards a common goal.
Only that doesn't work either, because what people want is for themselves to have it but not their opponents, and you not building it while your opponents do is the opposite of that.
It's like calling for a general halt to the production of military equipment. How do you expect that to actually happen?
The first one is a difficult balance but not really impossible. The second is basically utilitarianism: Of course you can't maximize all wishes because they often contradict each other, but there can be a reasonable trade-off. Some tradeoffs are clearly better than others.
> The first one is a difficult balance but not really impossible.
It's a direct trade off. If you want it to do more "good" things you make it able to do more "bad" things.
> Of course you can't maximize all wishes because they often contradict each other, but there can be a reasonable trade-off. Some tradeoffs are clearly better than others.
The easy tradeoffs are the ones nobody disputes and everybody is already trying to do. There is no lobby for having it hallucinate more or give you ingredients that will combine to make poison when you ask for a tasty recipe.