Whoever is in power determines what is "ethical society" or what makes society "better". Censorship is a form of physically violent oppression and is never justifiable.
Society is nothing more than the aggregate form of its constituent parts, i.e. all of us. It’s not separate from us, it is us, and we all have an obligation to act to improve it. Censorship is a tool, just like free speech or liberty or law or justice or rhetoric or tradition, that we have to wield to those ethical ends.
Yes, parts of society absolutely have the right to censor other parts of society. Just as we have the right to imprison, or regulate, or chide, or squelch. None of these things are sacred and inviolable lines. All of them are tools, which we have the moral obligation to use, judiciously, to improve the human condition.
Free speech improves the human condition. Therefore, by your own argument, the authorities have the moral obligation to forcibly censor your own anti-free speech comments.
Free speech is an amoral tool, which can be used to improve or degrade the human condition, when wielded appropriately. It's our job to decide when and how to use it, and not act naïvely and pretend it's a virtue in itself.
Which means censorship vectors get to wrap things they want censored as "racist".
Soviet censorship vectors were able to wrap dissenting viewpoints as "counter-revolutionary" or "bourgeois" without regard for the consistency of content to the wrapper.
You want it to be, but in practice it's not. Enforcement of immigration policy is seen as racist by some and not by others, for example. So though _you_ claim to know (because obviously _your_ beliefs are correct), people disagree on this point.
Burning a book is physical violence. Banning a sub is physical violence. In censorship you are forcibly destroying a communication between two nonviolent third parties.