[stcredzero]

I'm not sure how much this has to do with quantity of x as opposed to quality of x.

Quantity matters. Everything is contextual. If one is in a "hypo-minority" situation, then one is more likely to stand out. Being unique can be nice, but being irrevocably marked as unique, with no respite can feel like being trapped. Being singled out is simply more likely if you are truly singular.

Like if a company is predominantly composed of white men, does adding a white female make it more diverse than adding a black man?

I think that very much depends on the people involved. So in this, you are right. But quantity does matter a lot.