Given that our shared culture seems quite committed to labeling all men as "unfair", I'd think that the converse description "the fairer sex" is only... well, fair, y'know.
Maybe, but English is not my native language so it doesn't matter here.
Fun fact: The word for 'woman' in my country is derived from '婦' - 女 (“woman”) + 帚 (“broom”), so even calling them that way is kinda male chauvinist already.