People can't start showing up because they need invitations to join your channel. If you don't want a radical group there, don't invite them. It's that simple.
"It's that simple" is why we keep getting into this mess. People are not simple. They're vindictive and petty, and unfortunately some are out to turn your platform into their toy with which they will do nothing but harass and abuse your user base.
I don't know what you're talking about. Nobody has ever harassed me on my Discord channel. Everybody on there was invited by me or one of my friends. If people you invite are harassing you, get rid of them.
There is no "turning the platform into a toy". Everybody on your Discord channel is somebody for which you gave consent to be there. There is no possible way for trolls or abusers to arrive on your Discord channel without first being invited. It really is that simple.