[Marha01]

> Once humans can be robustly replaced by machines, the military-industrial "meta build" will be a state with no humans, no cities, and 100% of all production dedicated to war.

We will nuke it long before it gets to that point. We don't nuke other countries because they still contain lots of humans. Any AI that actually becomes autonomous and starts to clear a nation state of cities and humans will immediately receive megatons in nukes.

[voidmain]

Nukes are not magic wands, they are mostly only worth it for targeting cities and other very dense concentrations of value and production. An enemy state going down this path will probably start out with cities full of humans (and will probably want to "mine" those cities for materials), but its growing robot production will be more broadly distributed (and EMP shielded) and therefore much harder to destroy. And they will of course have their own nukes. I agree it's possible that a global nuclear war, if it starts early enough, ends with no one still having the ability to operate AI. But that's not exactly the most attractive way to shut it all down, is it?

[apsurd]

Claude’s gonna code up some nukes?

Ok yes robot killer dogs from black mirror are terrifying and possible - they’re gonna enrich uranium from gpus and assemble nukes at the local toyota dealer?

what are we talking about

[voidmain]

Governments, initially made of humans, will deliberately create military and industrial robots and assign them to create and scale production chains that go from natural resources to finished war machines. Maybe your government will not do this; in that case it will be helpless against the robot armies of those that do (yes, even if it has a few or few thousand nukes).

[apsurd]

> … and assign them to create and scale production chains that go from natural resources to finished war machines…

so yes, you think claude’s gonna get a skill any day now to code up some nukes.

Your scenario assumes fully autonomous ASI. That’s just not what LLMs are. So then the discussion is more that ASI is inevitable. That’s much more interesting but very different from the point release of Opus.

[voidmain]

You don't need superintelligence to get this problem, but you do need human level intelligence across all domains. I don't know how far we are from this, and neither does anyone else.

[apsurd]

human level is much more manageable. at that scale, we’ve seen just how much steering AIs need.

And AIs have no will. That’s been my best response on the philosophical end. Without will, they’re bound by humans.

[idiotsecant]

'We' will not be players in this situation. Over the space of a few generations the only entities that will have any power will be the feudal corpo-state and the only entity that will have any power within that will be a thin crust of humans who control the central machinery of that corporation.

The rest of us will be irrelevant.

[Marha01]

Even if it would be true (dubious), as long as those entities are controlled by humans, they will have a strong incentive to nuke any fully autonomous AI with obvious military objectives.