When folks say stuff like this, I feel like they're imagining two buddies arguing over tax reform or something. I'm absolutely fine being friends with someone who has conservative economic beliefs.
What I'm not fine with is being friendly with someone who supports policies designed to hurt my friends and family.
It turns out that befriending people who hold opposite and even toxic opinions is actually a great way to bring them back to more moderate and rational thinking.
There is a Jordan Harbinger podcast about a black man who single-handedly dismantled multiple KKK clans literally by befriending them. The trick is to approach the other person with curiosity and reason, not with judgement and adversity. Not everyone can be talked out of being a racist (or whateverist) of course but at the end of the day we all have more in common than we have in difference and a surprising number of people are willing to change their views on an entire demographic if they aren't being shouted at, shamed, or just silently shunned.
"Disagree with a point of view" and "don't want to be beaten up" (an example) are not in the same ballpark.
People are undoubtedly victims if they get beaten up or have to live under daily threat of it. (Again, example).
That is relevant to the KKK a few comments up. It's not that long ago the KKK wouldn't stop at beating someone up, when they could get away with torture and murder and be let off by juries who approved - in some states.
Replace "beaten up" with any other kind of serious harm to get the point. There are plenty of examples today, whether you care to recognise them or not.
What I'm not fine with is being friendly with someone who supports policies designed to hurt my friends and family.