[svara]

It's tempting to speculate about what might happen in the very long run. And different from the jobs question, I don't really have strong opinions on this.

But it seems to me like you might not be sufficiently taking into account that this is an adversarial game; i.e. it's not sufficient for something just to replicate, it needs to also out-compete everything else decisively.

It's not clear at all to me why an AI controlled by humans, to the benefit of humans, would be at a disadvantage to an AI working against our benefit.

[boothby]

Agreed on all but one detail. Not to put too fine a point on it, but I do believe that the more emergent concern is AI controlled by a small number of humans, working against the benefit of the rest of humanity.