[svara]

Yes, okay.

I think we're talking about different time scales - I'm talking about the next few, maybe two or three decades, essential the future of our generation specifically. I don't think what you're describing is relevant on that time scale, and possibly you don't either.

I'd add though that I feel like your dystopian scenario probably reduces to a Marxist dystopia where a big monopolist controls everything.

In other words, I'm not sure whether that Earth-spanning autonomous system really needs to be an AI or requires the development of AI or fancy new technology in general.

In practice, monopolies like that haven't emerged due to competition and regulation, and there isn't a good reason to assume it would be different with AI either.

In other words, the enemies of that autonomous system would have very fancy tech available to fight it, too.

[boothby]

I'm not fussy about who's in control. Be it global or national; corporate or governmental; communist or fascist. But technology progresses more or less uniformly across the globe and systems are increasingly interconnected. An AGI, or even a poor simulacrum cobbled together from LLMs with internet access, can eventually hack anything that isn't airgapped. Even if it doesn't have "thoughts" or "wants" or "needs" in some philosophical sense, the result can still be an all-consuming paperclip maximizer (but GPUs, not paperclips). And every software tool and every networked automated system we make can be used by such a "mind."

And while I want to agree that we won't see this happen in the next 3 decades, networked automated cars have already been deployed on the street of several cities and people are eagerly integrating LLMs into what seems to be any project that needs funding.

[svara]

It's tempting to speculate about what might happen in the very long run. And different from the jobs question, I don't really have strong opinions on this.

But it seems to me like you might not be sufficiently taking into account that this is an adversarial game; i.e. it's not sufficient for something just to replicate, it needs to also out-compete everything else decisively.

It's not clear at all to me why an AI controlled by humans, to the benefit of humans, would be at a disadvantage to an AI working against our benefit.

[boothby]

Agreed on all but one detail. Not to put too fine a point on it, but I do believe that the more emergent concern is AI controlled by a small number of humans, working against the benefit of the rest of humanity.