The users of the product that my team makes are overwhelmingly male, in fact I don't know that I've ever had a female user. Should I refuse to hire women on that basis?
You got rid of men in order to increase the number of women on your team. But when someone says they'd merely not hire women citing your own apparent justification, you disagree that it'd be OK and then respond to a strawman that nobody proposed.