[NaliSauce]

>That experiment was done in STEM academia recently, and showed a 2:1 hiring advantage for women.

Back when I was still at university I was on a hiring panel for a new prof.

In a first step we ranked all the applicants using qualitative and quantitative metrics (the rankings across qualitative/quantitative and people on the panel was very similar when we compared rankings at the end) and then we handed in a shortlist with the best male applicant and best female applicant. We did this to make the Gleichstellungsbeauftragte (="diversity officer") shut up and thought that since the ability gap was really noticeable it'd be a no contest. How wrong we were. Despite being in the upper part of the bottom half of our ranking, she got the job.

That was the day I found out that I wasn't at a meritocratic institution as I had previously thought and left a few months later. That sort of thing really isn't good for your self confidence.

[andrewprock]

When you say the ranking was quantitative, what sort of quantities did you measure? Did you ever evaluate the quality of your metrics, and how predictive they were for future success?

IIRC, Google did the same thing and concluded their hiring metrics had very little predictive value beyond setting a certain bar. Once you rose to that level, most discriminatory power of the metric evaporated.

If that was the case where you were, then is probably very little predictive value between "top scores" and "median scores".