What experiences do you have to the contrary? I’ve moderated a few community spaces and this has been the way it’s gone every single time it’s come up for me.
Have you reflected on the fact that all those community spaces have had one thing in common - you moderating them? Perhaps listening to peoples concerns about lack of diversity of thought instead of labeling them bigots would allow you to see things differently. Most people don't feel as strongly about controlling the discourse of others that they end up moderating any not to mention multiple spaces.