Who is "we"? That is not established at all, in fact the opposite is objectively true and widely understood. Chromosomes are the key determinant of "gender", by even the most progressive definition of the term.
That's a specific cultural definition of gender. Taking a wider view of human history and behavior tells a more complicated story. The common thread is that Gender is something that people do which is only loosely correlated to human sexual dimorphism.
This doesn't convince people who see their cultural norms as objective reality tho. hence the controversy.
If anything chromossomes would be the key determinant of sex, not gender. But even that's not how we attribute sex: biologically sex is attributed by the capacity to generate small or big gamete, which means that someone who doesn't produce either type, such as a child or elderly person has no sex.
This doesn't convince people who see their cultural norms as objective reality tho. hence the controversy.