I don't think anyone really considers the memo itself to be a scientific fact. The scientific papers cited and arguments contained therein people might take as scientific facts.
It was a pretty scientific memo. It cited a lot of scientific research.
The sort of people who argue against his memo always end up looking foolish because they never argue against anything that was actually said. No specific claims are ever brought up except lying strawmen like "he said women can't program", a claim not found in the memo.
Instead it's all just slurs, insults, attacks on his character, guilt-by-association and so on. There's never any actual depth to it, it's all "don't look at the bad man".
Damore's arguments must be good, because in years of this coming up nobody has ever once managed to make an intellectual argument against the stuff in his memo.
When they're not conservative opinion pieces, they're cherry picked articles, even going to far as to misrepresent the data they support.