Hacker News new | ask | show | jobs
by PollardsRho 498 days ago
What incentives do any humans have to so totally delegate the functioning of the core levers of societal power that they're unable to prevent their own extinction?

"Better machine alternatives" implies that the police and military aren't first and foremost evaluated through their loyalty. A powerful army that doesn't listen to you is not a "better" one for your purposes. The same isn't true of the economy: one could argue that our current economic system is beyond any one person's ken, but even if I don't understand how my coffee came to me and no one person would be an expert on that entire pipeline it works.

The idea that AI could lead to power concentrating in the hands of a few oligarchs who use a robot army as a more effective version of the janissaries or praetorian guard of the past certainly seems broadly plausible, although I'm not sure that the effectiveness of the Stasi is the limiting factor on autocracy or oligarchy. I don't understand how that links to human extinction. For most of human history, most people have been unable to meaningfully impact the way their society operates. That is responsible for an incalculable amount of suffering, and it's not a threat to be taken lightly, but if anything one might argue it's likely to ensure some human survival for longer than a less stable, freer system.

1 comments

> What incentives do any humans have to so totally delegate the functioning of the core levers of societal power that they're unable to prevent their own extinction?

Because it'll be more effective at every step than the alternative. Just like specialization is more effective, so anyone who wants to avoid poverty needs to outsource their food growing to giant mechanized farms.

> "Better machine alternatives" implies that the police and military aren't first and foremost evaluated through their loyalty. A powerful army that doesn't listen to you is not a "better" one for your purposes.

Ah, I think there's a confusion - I'm saying that the police and military will stay loyal to the state, or head of state. But that even if there is a human nominally still in charge, that that human's hands will be tied by competitive pressures to gradually marginalize their own human citizens in favor of more productive machines. Maybe a good analogy is unpopular today would be free trade deals or immigration policies enacted for economic reasons.

I think the objection of "wouldn't the few remaining humans in charge become ever-more powerful, so they could enact UBI by fiat" is a good one. But I think it's just hard for third parties to treat unpromising, unproductive beings well - others will be constantly proposing other, more lucrative, uses of their resources.

> But I think it's just hard for third parties to treat unpromising, unproductive beings well

Strong similarity to the Resource Curse and despotic governments. The people aren't that useful if you have natural resources.

We actually mention the Resource Curse as an example of this in the paper.