You can already control who and what you see by running your own server. If you choose to use someone else's infrastructure, don't act surprised when you're subject to their rules.
How many of these people are going through these blocklists to properly verify the claims?
And why should the users of my instance be penalized because someone found some reason to dislike me? If someone wants to block me, fine. But having the influence to push a whole server out of the wider network definitely counts as "too much power".
Users of an instance choose to place their trust in the instance moderator. If they feel they're being "punished", they're free to leave. It is insanely easy to change what instance you're on.
You're right, people do have the right to block whatever they want. But it's just shitty that someone else made the call, and now presumably many people are impacted. There's no recourse.