"rare" doesn't mean "never". Caster Semenya is an Olympic candidate who is getting disqualified from some women's racing events because she is intersex and has just a little bit too much biological masculinity to compete as a woman.
I don't think anyone's claiming that "biological sex is a spectrum", but some people "do not fit the typical definitions for male or female bodies". Some are born with ambiguous external genitalia, but there are also people who look completely female while having male (XY) chromosomes. Since various distinct medical conditions are involved, it doesn't make sense to call it a "spectrum" even from a medical point of view.