Once you accept and identify as a label then you are no longer part of the middle. That is one of the main arguments against race quotas and affirmative action. It taints the whole idea of equality and neutrality.
A woman that joins a company in mountain view that only hires women is the same as a man that joins a company in Manhattan that only hires men. Can't really say you are neutral after that.