[zrezzed]

> turn the door handle, and unplug the rack.

There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power.

I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself.

But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.

[mlsu]

Human systems. They are protected by human systems. Human beings are the ones who press the big red button.

We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone.

But what I'm talking about is categorically, fundamentally, not the eschaton!

[high_5]

Yes, they are human indeed, but those humans cannot agree on whether the plug should or shouldn't be pulled.