Right. Depending on the take, your aligns with the utilitarian perspective as well. You don't need to make everyone happy, as long as it leads to the "greatest good" (rule utilitarianism) or the "highest average happiness" (average utilitarianism). My point is that this perspective is not devoid of philosophical problems.
If society can't make everyone happy, what's the point.