Well, that isn't really true, since the conception of what races exist and who belongs to which shows wide cultural variation.
More to the point, though, racism is an obvious manifestation of the kind of tribalism I think we are talking about and I don't think it's really controversial to argue that it is harmful at least to the marginalized groups. I'd personally even go further and say it harms all parties involved.