Yes, parts of society absolutely have the right to censor other parts of society. Just as we have the right to imprison, or regulate, or chide, or squelch. None of these things are sacred and inviolable lines. All of them are tools, which we have the moral obligation to use, judiciously, to improve the human condition.
Free speech improves the human condition. Therefore, by your own argument, the authorities have the moral obligation to forcibly censor your own anti-free speech comments.
Free speech is an amoral tool, which can be used to improve or degrade the human condition, when wielded appropriately. It's our job to decide when and how to use it, and not act naïvely and pretend it's a virtue in itself.
Which means censorship vectors get to wrap things they want censored as "racist".
Soviet censorship vectors were able to wrap dissenting viewpoints as "counter-revolutionary" or "bourgeois" without regard for the consistency of content to the wrapper.
You want it to be, but in practice it's not. Enforcement of immigration policy is seen as racist by some and not by others, for example. So though _you_ claim to know (because obviously _your_ beliefs are correct), people disagree on this point.