|
Depending on how you define it, you could say my research is in this general area. My sense of how to interpret these sorts of findings has changed greatly over time, and become much more moderate and humble. Discussions over these issues tend to be very heated and people oversimplify issues on both ends. These tests are predictive, and as you point out, it's disingenuous to claim otherwise. They're measuring something valid. However, they're doing so relatively poorly, or rather, roughly. Even the quote you cite suggests that the total amount of variance accounted for was about 19%, which isn't too shabby by any means, but is also very crude when you think about implications for real-world consequences. Interpreting that 19% also becomes very murky, when you consider factors like teaching to tests, that tests like those are used for academic and career selection, leading to self-fulfilling prophecies. In rare circumstances where researchers have been able to remove some of these effects (for example, due to unusual legal decisions about acceptable use of test scores), the relationship between test scores and career performance has been shown to be very weak in the short-term, and almost non-existent in the long term. Somewhere underlying the distribution of test scores is some distribution of ability that makes it easier for some people to thrive in certain settings. But ability is just one thing in a long list of things, along with practice, resources and other environmental factors that contribute to success, and sociological dynamics. Add to that test scores being very crude indices of ability to begin with, and you're in a very messy situation. What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between. |
This is 19% after range restriction. They're making the same point that the graph of quartiles does: even after you set an extremely high bar, differences in the test score are still predicting quite a bit of variance despite all the other possible diluting factors like geography/family/test-error/personality/interest/wealth/opportunity... If anything, it shows that institutions aren't being 'crude' enough - if they were using the test scores optimally and extracting all of the signal, the variance would be 0%.