| I actually took part in this study. A few notes: 1. The exam was supposed to be administered 3 times, Freshman, Sophomore, and Senior years. I took it my Freshman and Senior years. 2. The exam was not a normal exam. I won't say it was pointless, it definitely measured something, but it's hard to say exactly what. Imagine you have to make an exam to measure "learning" and this thing was about as close as you could come to getting one. But that's still far off the mark. 3. The sample was not representative. Taking the exam was opt-in, and the college put forth a variety of incentives to get people to take the exams. Despite this, they definitely had a high drop off between sign-up-for-everything freshmen and I'm way too busy to spend three hours taking another exam seniors. I believe there were schools where it was not opt-in, but even so, I can't imagine any of the students taking this exam had grades riding on their performance. And given that it was a single exam, it was dependent on the students' state of mind on the particular day they took the exam. Three data points per person, over four years. It's hard to draw any conclusions from this data. |
I'm not saying I don't agree that the test may not have measured anything useful, or that there wasn't some sampling bias, but there's nothing inherently wrong with 3 data points per person over four years.