Good point, and much harder to challenge. If the majority is against an authoritarian there's protests and sabotage of social structures. If the majority oppresses a fringe group, it's often socially encouraged
For these reasons, I personally believe authoritarianism cannot be opposed without a solid foundation of individualism. The problem becomes that explaining ideological nuance is rarely politically expedient or even rhetorically effective. Appeals to collectivism are more easily digested by the masses.