[jeremysalwen]

It seems like you have decided that sex is solely defined by gametes. Ignoring for now that this is not the sole criterion of "sex" used in most scientific and colloquial contexts, even on your own terms, it is not true that all humans are one of two "sexes".

At a bare minimum, you would have to make an allowance for the "third sex" (as you define it) of those who do not produce any gametes. Remember, we are talking about sex in humans, not sex in human gametes. I do not dispute that human gametes are strictly binary; the question is about classifying individuals, for which the binary classification is obviously not absolute, even on the question of gamete production.