I actually have a game idea playing around with this idea. Sure, the AI is 'aligned' but what does that even mean? Because if you think about it humans have been pretty terrible.
Absolutely. The reason people worry about AI alignment is because we already have millennia of experience with the intractability of human alignment. So the concern is, what if AI is as bad as we are, but more effective at it?