[moonchrome]

If you get access to singularity level AGI you would have no incentive to cooperate with other people and a strong incentive in preventing others accessing it.

Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor.

"Open"AI is already beginning to exhibit this.

[1attice]

This puts in mind the earlier post on HN about modelling civ as either a one-shot or iterated prisoner's dilemma.

You make the case that, for sufficiently high N, possessing GPT-N is a one-shot prisoner's dilemma.

The only winning strategy is 'defect' --- which in this context, means chokepoint control.

[1attice]

I don't know who is reading this who can help. If you know someone closer to the fire, pass this along.

OpenAI -- its people, its buildings, its servers -- need nation-state level protection. This is an ICBM you could put on a thumb drive -- in fact, it's far worse than a loose nuke, because a nuclear weapon has a geographically limited range.

There need to be tanks and guards and, like, ten NSAs in a ring formation around this thing. At pain of x-risk, do not treat this like a consumer-facing product. This is not DoorDash.

This isn't a threat to national security. This isn't even a threat to the entire geopolitical order. This is a threat to the possibility of a geopolitical order.

OpenAI's assets -- its people, its servers, its buildings -- just became the most desirable resources on the planet. It behooves any actor with ambition to secure at least a copy, and ideally, capture at least some of the people who created it.

It doesn't matter if the threat actor is China, or Russia, extraterrestrials, or mermaids. You will find out who wants it shortly. But you know now -- you know from game theory, the body of mathematics that has kept the peace since the invention of atomic weapons -- what happens next.

[moonchrome]

Thanks for phrasing it better, yes my point is there is no social iteration past singularity level AGI.

[apatil]

Say you get access to a singularity-level AGI, meaning you have the power to render the entire human economy completely irrelevant. Given any task, no matter how big, small, novel, complex, simple or mundane, it's vastly more cost-effective to have your AGI to do it than to pay humans.

Do you really want to accumulate incomprehensible material wealth for yourself, whatever "wealth" means in this scenario where money is longer a token of spent human life energy, and let everyone else struggle and suffer? Or would you rather tell your AGI "please create a utopia in which all humans are fully actualized" and then go have a latte?

[visarga]

> "please create a utopia in which all humans are fully actualized"

That doesn't work well in reality because we do relative comparisons not absolute, so not all humans can be better than average.

And without competition we stagnate, there must be incentives to compete and take risks, and thus not everyone can be equally actualised, our level depends on our previous decisions.

[moonchrome]

That's basically the petting zoo outcome - you let others live because it makes you feel nice. But you still can't allow anyone access to the same level of tech because you can never be certain of their motivation. There's no nuclear deterrent between AGI only first movers advantage.

[apatil]

I don't disagree, but I think the framing is overly pejorative and makes the likelihood of this outcome seem more tenuous than it really is. We do lots of things to help others because they make us feel nice. "Mothers and Others" by Sarah Hrdy argues that this tendency isn't just a fluke or a game-theoretic equilibrium of some kind between fundamentally self-interested agents, it's an ancient and deeply ingrained aspect of human nature.

The thought of living in a constructed world that exists by the grace of a single human owner of a super-powerful AGI is distasteful, of course, especially if the human owner uses their power to impose some of their own opinions about how people should think and behave. Becoming dependent on AGI is probably inevitable at some point, but I don't see that as so different from the status quo. We're already dependent on systems created by other humans that are so complex and sophisticated that no individual can grok them all.

I guess I would like to think that we will move past any initial impulse that the owner of the AGI feels to control other humans. We will presumably change so much that old ideas about how we should think and behave will seem irrelevant and quaint. And the AGI, which presumably will have the social engineering superpower, will hopefully point out inconsistencies between the owner's desire to control human thought and behavior and their desire for humans to live their best lives. Hopefully.

[phs318u]

I think we already know the answer to that question if we can extrapolate from some (not all) of the tech bro billionaires. Narcissists need to stand out from the herd.

[grogenaut]

Have you noticed how fast everyone else was able to copy open ai. And that's just what we saw or someone leaked. History is full of parallel inventions... how long after the US had nukes did russia?

[moonchrome]

Sounds like more of an argument for preemptively wiping out the competition.

Comparing it to nukes doesn't hold since social norms/ethics/etc. become irrelevant if core tenant of society is broken.

One thing that could happen is AI defense outperforms offense long enough to develop multiple instances - have no idea what would happen at that point.

[grogenaut]

I wasn't talking about nukes in using them, I'm talking about nukes in that once it was known to be possible/useful everyone with the means did it.

[moonchrome]

I agree with you - I edited my comment above - if defensive capabilities allow multiple AGIs to develop I have no clue what the outcome would be - we are talking about predicting superhuman intelligence here.

[grogenaut]

An interesting thing to consider is if an AGI would be able to run on small nerfed hardware like recent optimizations can, or if you need absurd tier hardware to also run it. If it's the latter even if it's really smart there's only on or N of them. And it's still limited by the speed of light in it's thinking speed. If it's the former, when it leaks, it'd be everywhere.

[visarga]

> Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor.

But even without AI others are still a risk factor. AI could also act as a mediator between conflicting parties.

[moonchrome]

But that's the point - benefits of cooperation outweigh the risks and we have laws/ethics/drives to stabilize the system based on this.

If you have AGI then others are just a threat and can offer nothing you can't get from AGI.

[LordFast]

What you said made sense until I said it out loud to my wife.

Hoarding access to resources has /always/ been the name of the game.