[duvenaud]

Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities.

As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.