[moonchrome]

That's basically the petting zoo outcome - you let others live because it makes you feel nice. But you still can't allow anyone access to the same level of tech because you can never be certain of their motivation. There's no nuclear deterrent between AGI only first movers advantage.

[apatil]

I don't disagree, but I think the framing is overly pejorative and makes the likelihood of this outcome seem more tenuous than it really is. We do lots of things to help others because they make us feel nice. "Mothers and Others" by Sarah Hrdy argues that this tendency isn't just a fluke or a game-theoretic equilibrium of some kind between fundamentally self-interested agents, it's an ancient and deeply ingrained aspect of human nature.

The thought of living in a constructed world that exists by the grace of a single human owner of a super-powerful AGI is distasteful, of course, especially if the human owner uses their power to impose some of their own opinions about how people should think and behave. Becoming dependent on AGI is probably inevitable at some point, but I don't see that as so different from the status quo. We're already dependent on systems created by other humans that are so complex and sophisticated that no individual can grok them all.

I guess I would like to think that we will move past any initial impulse that the owner of the AGI feels to control other humans. We will presumably change so much that old ideas about how we should think and behave will seem irrelevant and quaint. And the AGI, which presumably will have the social engineering superpower, will hopefully point out inconsistencies between the owner's desire to control human thought and behavior and their desire for humans to live their best lives. Hopefully.