The issue is personal rights vs. general good.
In the end an individual's liberty ends when his actions are having a serious negative impact on the community.