This is exactly right. Criminality is a very high bar! There are many behaviours that fall well short of criminality that we shouldn't accept in communities.
Like homosexuality, atheism, blasphemy, miscegenation, witchcraft, vagrancy, and a whole host of other "anti- social" behaviors, right? After all, who polices the morality police?