This basically does happen. For it to not happen, people would have to universally be ok with prejudice toward them based on their skin colour, and that's obviously not true. A certain percentage of them will develop more hostile beliefs toward people who they feel are persecuting them.