[pvg]

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.

[FeepingCreature]

That's a difference of degrees, not kind. You can always ask for a higher class of proof.

[pvg]

How is it a difference of degrees? This is just plain inaccurate:

The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.

[FeepingCreature]

The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible.

My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.

[pvg]

The disagreement is about the difficulty of "proving"

It's not, it's just a turn of phrase you've latched on to. There's no discussion of a standard of proof, etc. The comparison between the Szilard Einstein letter and 'AGI safety' is specious and it's one you brought up! You have to outright misrepresent what the letter was about just to make it. It was not a letter about a 'hypothetical danger they had no way of proving'. There just isn't any reasonable reading of the letter, the context, the history in which that's an accurate statement.

[FeepingCreature]

My whole point and the emphasis on why I brought it up was that yes, the letter was about a "hypothetical" danger they had no way of "proving".

By any reasonable standard, at that point the possibility of nuclear fission was well established. But the argument is that there is no central committee that defines the reasonable standard and checks theories against it. It's all just opinion, and that specific opinion can be arbitrarily goalshifted.

I'm not saying the letter was about a hypothetical danger they had no way of proving, I don't think that. But that's specific to my interpretation of those terms, and I could easily see the exact same argument levered against the letter if that debate had been done in public.