It's exactly how rights work. Freedom of speech doesn't have to justify it's value to society, it's an innate right that's not up for vote. Google the founding fathers and natural law.
If freedom of speech didn't have to justify its value, you wouldn't need legislators to enact it as a law. Legislators aren't going to legislate without justification, and they shouldn't.