It's the well known correlation between stupid opinions that can do no good and stupid people that cannot control themselves. We should worry about the bad cases when oppressors are in power and able to do significant harm and/or manipulative sociopaths using hate speech as one of many tools.