Just because you can punish someone for political views doesn't mean you should. The freedom of speech can be an idea and an attitude that is bigger than the First Amendment.
That's fine. Even more so, then, freedom of speech can be a social norm (tolerance of other viewpoints) that protects people beyond the minimum standards the government sets.
Please don't assume that I am speaking about that.