[FeepingCreature]

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm.

With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.

[danaris]

Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe.

From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge.

This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.

[FeepingCreature]

I agree with your description of the AGI/ASI position.

My take on it is the other way around: every step of the argument seems plainly obvious.

[yifanl]

Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence.

So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.

[FeepingCreature]

I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research.

I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.

[pvg]

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.

[FeepingCreature]

That's a difference of degrees, not kind. You can always ask for a higher class of proof.

[pvg]

How is it a difference of degrees? This is just plain inaccurate:

The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.

[FeepingCreature]

The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible.

My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.

[pvg]

The disagreement is about the difficulty of "proving"

It's not, it's just a turn of phrase you've latched on to. There's no discussion of a standard of proof, etc. The comparison between the Szilard Einstein letter and 'AGI safety' is specious and it's one you brought up! You have to outright misrepresent what the letter was about just to make it. It was not a letter about a 'hypothetical danger they had no way of proving'. There just isn't any reasonable reading of the letter, the context, the history in which that's an accurate statement.

[FeepingCreature]

My whole point and the emphasis on why I brought it up was that yes, the letter was about a "hypothetical" danger they had no way of "proving".

By any reasonable standard, at that point the possibility of nuclear fission was well established. But the argument is that there is no central committee that defines the reasonable standard and checks theories against it. It's all just opinion, and that specific opinion can be arbitrarily goalshifted.

I'm not saying the letter was about a hypothetical danger they had no way of proving, I don't think that. But that's specific to my interpretation of those terms, and I could easily see the exact same argument levered against the letter if that debate had been done in public.