[jltsiren]

This is just the eternal conflict between people's freedoms.

Negative liberties ("freedom from") are limited and relatively easy to regulate. If you ban people from killing each other and stealing their property, almost everyone agrees it's not a huge burden. While these liberties sometimes come into conflict, such situations tend to be rare.

In practice, people care more about positive liberties ("freedom to"). In particular, they want the freedom to live a good life. Unfortunately people have different ideas of a good life, and those ideas usually require other people living their lives in a certain way and providing various services. If you try regulating this, you start quickly making choices who is allowed to live a good life.

Because laws are insufficient for a good life, people make voluntary agreements to ensure it. If certain kinds of agreements (such as HOAs) become popular, they can effectively prevent some minorities from living their idea of a good life. But the agreements are only a symptom, not the cause. The real cause are other people. Without HOAs, the same people would try getting actual governments regulate the same behavior. And failing to do so, they would often feel that the society prevents them from living a good life.