| Those estimates are generally in line with what other independent investigators find, maybe a bit smaller. The elephant in the room in these kinds of discussions is the validity of the tests to begin with. MIT makes a statement that the general tests are predictive in combination with other criteria, but how predictive? Generally studies of these sorts of things find that they're moderately predictive of first-year GPA (like .4 correlation), and then trail off to zero as the interval between testing and outcomes increases. The effects are even smaller in studies in relatively unselected samples (due to court orders or legal decisions, for example). So one argument always goes that you should use whatever empirically-supported stuff you can to make fair decisions, but we treat them as if they're more than they are. Sure, you could have a lot of things that are significantly predictive but with small effect size, or where there's lots of noise, but why as a society to we pretend these are huge effects? The other thing about that paper is the hint that the practice effects are larger in higher-performing examinees, which also makes sense and is consistent with other studies. Why is this a problem? Because those are exactly the types of students for whom these issues are more applicable. A 12 point average gain with practice doesn't matter if you're including people who never had a chance at MIT anyway, but a 20 point gain might in a highly competitive group where small differences are being magnified tenfold. The issue in all of this isn't the people in the 99th percentile versus the 50th percentile, which is the bulk of what's going into these predictive models and effect sizes, it's the 99th versus the 95th. There's a ton of real-world noise, but society acts as if the noise is nonexistent. It's like we're idolizing the outcome of some kind of survivorship bias process. |