[DalasNoin]

Even if alignment works and AI is "safe", what does that actually mean?

Frank Herbert in Dune wrote about AI: 1. "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."

If a few people control super intelligent systems, they could probably control all your thinking. Would these machines run all the media, all the politics, all of society? These machines could likely persuade you into any believe or action.

2. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments"

These early art generating AIs will only be the beginning to superhuman visual and audio art. All music, movies, games and whatever will come might be created by AI at a superhuman level. Will they control our sense of beauty and will you find human art to be dull in comparison?

This is just some perspective on what might be wrong with an AI utopia.