"Race" is a de-facto cultural category, at least in the U.S. It's not merely a description of someone's outward appearance; the reason people even care about the construct in the first place is its social aspects, which happen to be rather pervasive. It's not obviously wrong to advocate for separatism on the basis of such things, nor is separatism per se hateful or violent.
Except you can't, which is why for example "Islamophobia" is a thing. Things that should be ideologically separate are inevitably tied back to race/ethnicity, because nobody seems to know how to separate race/ethnicity from ideology in debate. Just listen to how much flak Sam Harris gets on the subject.