You are absolutely and perfectly allowed to speak out in public against such people. But you must be prepared to take responsibility if your speech is worded to spur people into violent action and to incite hateful action.
The court of public opinion is notoriously fickle. Rousing hate in the masses is not a reliable tactic, and often backfires.
There’s a difference between what’s good for a country and what’s good for an individual person - society at large can’t really function if it laws place the singular wants and needs of any given person as the metric of allowable. Just because you can justify a hatred doesn’t mean there’s some transitive property that means society should therefore tolerate it.
The problem starts if you go around in public, making hateful speeches that incite violence.