I don't think anyone is disputing that there are biological differences between men and women (on the average) that result in unequal distributions of some outcomes. An easy example is "fastest marathon" or "number of babies".
The dispute is that there are biological differences between men and women that are relevant to qualification to work at Google in software engineering.
Nobody is questioning their qualification. The memo doesn't.
There quite possibly are biological (and cultural) differences that ultimately affect the representation of women among Google software engineers that are not sexism. As a result, a diversity / hiring policy that assumes sexism is far and away the only relevant cause is not likely to work well.
That's the argument alt-right folk who actually deserve our hostility are trying to use Damore to make. Damore, however, is instead arguing that those biological differences make women less likely to pursue engineering as a field, not that those difference make them inferior.
Then why are the programs he's criticizing all about either successfully hiring or retaining women who are already interested in working at Google?
He also explicitly mentions ability: "Note, I’m not saying that all men differ from all women in the following ways or that these differences are “just.” I’m simply stating that the distribution of preferences and abilities of men and women differ in part due to biological causes and that these differences may explain why we don’t see equal representation of women in tech and leadership."
(Also, let's be clear, someone who seeks out the support of Stefan Molyneux to advocate his viewpoint cannot be meaningfully distinguished from alt-right folks who supposedly want to distort his viewpoint.)
People's qualifications are not all the same. Imagine that Google wants to hire 1,000 people, but 10,000 people apply for a job. Then Google logically ought to take the best 1,000 people that apply.
With diversity quota's, the company may decide to hire a less able woman in favor of a more able man.
The reality is that 2,000 of them are indistinguishably good and diversity programs are intended to make sure that google hires equally able women out of that group instead of choosing all the men.
The company may also decide to hire a less able man in favor of a more able woman.
Given that the vast majority of incompetent people I've worked with have been men (if not all of them), I'm surprised to see less attention to this phrasing of the problem. Maybe it is too politically incorrect to bring up?
Although, to your point, that doesn't really require critical thinking. It just requires impressive mental gymnastics to dispute.