[danaris]

Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe.

From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge.

This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.

[FeepingCreature]

I agree with your description of the AGI/ASI position.

My take on it is the other way around: every step of the argument seems plainly obvious.

[yifanl]

Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence.

So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.

[FeepingCreature]

I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research.

I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.