Yes, the question is, why can't we? Becasue apparently we can't stop at that point. We've already gone so far past that that the slippery slope has been established in this case.
Yeah right, what's a bigger problem: not being able to say the N word or going to jail for no reason and feeling lucky that you weren't shot for no reason because you're Black?
They are related; there's a false equivalency in these arguments between the right to be a racist and the right to be free from racism. On that spectrum, our society is far better at ensuring the former at the expense of the latter, and we need to recalibrate.
Lest you think this is futile, our system of laws and the social contract itself is based on balancing rights. For example, we have the right to free expression, but perjury and fraud are crimes.
In other words, we work too hard to protect bigoted speech, and we don't work hard enough to protect Americans from bigotry and discrimination, and this is at least in part due to the false equivalency we maintain between them. I'm not saying I have a great policy answer -- although allowing civil suits for bigoted speech might work -- but we should at least be talking about it.