[rtsil]

The "harm" is in relation to other people's constitutional freedoms and rights. Freedom of speech isn't inherently superior to, say, freedom of assembly, freedom of belief, the right to safety in one's person and one's properties, the right to vote, and so on...

It's a question of value: either you think Freedom of Speech is the highest form of freedom and right that one can hold, and all the other freedoms must come second, or that all freedom and rights are equal, and the role of the Constitution and the law is to find a balance between those.