The principle of free association isn't just anyone should be able to associate with those that feel the same way, it's also that people shouldn't have to with those they don't.