The difference is that speech, even unpopular, controversial, insulting, and dehumanizing speech, isn't violence. There is no legitimate claim to self-defense aside from direct threats to one's own life or property.
But we're not talking about speech, we're talking about tolerance. Speech is just the medium. And lack of tolerance is a threat to one's life and property: men did not tolerate women receiving an education so that they could have a career; redlining housing developers in the US did not tolerate black people buying homes in white neighborhoods; Nazis do not tolerate Jews to live. Intolerance is not "just" speech, it is speech as a direct encitement to actions that are intended to have negative consequences for the targeted party.