[kxyvr]

There's a wonderful exchange in the movie Interstellar that speaks to this:

    Cooper: You're ruling my son out for college now? The kid's fifteen.
    Principal: Tom's score simply isn't high enough.
    Cooper: What's your waistline? 32? With, what, a 33 inseam?
    Principal: I'm not sure I see what you're getting at.
    Cooper: You're telling me it takes two numbers to measure your own ass but only one to measure my son's future?

The point being is that a person, and their future, should not be distilled into a single number like an SAT score because people are far more complex than a single number. I would also contend that splitting them into a few numbers, such as by subject area, doesn't help either.

[triceratops]

The SAT has 2 components with separate scores. The SAT score is usually the sum of both, but there's no reason colleges can't weight one section more than the other.

If you really care about multiple dimensions, add more types of standardized tests - general science (or specialized - separate physics, chem and bio sections), history, geography, whatever.

Or tell students to take the standardized tests that most closely map to their preferred major.

[kxyvr]

I still contend strongly that it is wrong to distill a person into a collection of numbers. They are not.

My mother was a special ed teacher who eventually became a diagnostician whose job it was to determine eligibility for services. Part of this job was to give a standardized test to a child to see if they met state set criteria. That said, this was only part of an assessment and was used along with parent preference, teacher recommendations, and other people involved with the process at the school. Outside of meeting state criteria, the numbers in the test were used to identify areas of strength and weakness to help understand a child and put this in perspective of a broader story. For example, if a child struggled with math, was it because they couldn't read well, struggled with the math itself, was distractable? There were also other instances where the test scores themselves were misleading. For example, did a child perform poorly because they were sick, or the AC of the building broke, or they set a test date on a school holiday when the child knew the other kids had a day off?

My point in telling this story is that the raw score in a standarized test can be helpful, but it does not tell a complete story. It is part of a broader assessment. The SAT and college admission is not different.

Purely using a test score does provide an objective score to rank a collection of applicants, but I still contend strongly that it misses the point. Depending on your point of view, college is meant to educate a populace, provide job training, give economic opportunity, or a whole host of other goals. One can use test scores a proxy to determine which applicants best approximate these goals, but I think we should consider whether they really help achieve this.

I'll also mention that using tests scores is not entirely "fair" either. I'd a friend who dated a guy who worked for Kaplan, one of the test prep companies. They had very good data about how much their test prep improved test scores. They were also very expensive. As a result, money and wealth can be used pretty directly to improve one's score. It's difficult for a university to fully address wealth inequality in society. However, since tests can be gamed so easily, I would caution against their over use.

[triceratops]

> I still contend strongly that it is wrong to distill a person into a collection of numbers

No one's saying a person is a collection of numbers.

> My point in telling this story is that the raw score in a standarized test can be helpful, but it does not tell a complete story. It is part of a broader assessment

And that's very noble and well-meaning. But it can also be used to discriminate and deny opportunity. Raw scores, on the other hand, are hard data. You either have them or you don't. Do factors outside of one's control go into them? Absolutely. Those factors are all-pervasive and will influence anything you might use to determine merit.

> Kaplan, one of the test prep companies. They had very good data about how much their test prep improved test scores.

Of course they would say that. Even assuming it's true, Kaplan test prep books are only around $50 on Amazon. You could also get them for free from a public or school library (or the high seas, if that's your thing).

As I understand it, Kaplan has apps and dashboards and more material if you're a subscriber. Is there data about the marginal score improvement using that versus just the books?

When you get right down to it, everyone's starting point is different. Reducing the number of variables improves opportunity IMO.

[kxyvr]

My continued assertion is that "hard data" can be misleading, wrong, and discriminatory. It requires judgement to understand what it tells us and whether there is error in the collection of the data itself.

As far as Kaplan, I don't have metrics to share, but what I saw shocked me at the time. It appeared as though they collected data as to how much their test prep improved scores and it was pretty dramatic. And, they were smart enough to distinguish between their test prep books and their in person tutoring. In person tutoring worked better and by a lot, but it all helped.

As far as tests, I've never contended that they shouldn't be done, but that these scores need to be part of an overall portfolio. I have in fact sat on admittance boards for graduate school and the two dominant factors other reviewers used were GPA and where they went to undergrad. Both of these were hard metrics and full of bias. I'll also contend that they weren't particularly effective in identifying students who successfully completed the program.

Something to consider, I do think test scores can help screen candidates. For example, if one is going to admit into a math program, then their quantitative score should probably be above a certain level. After that, I personally don't think it matters. In that way, one could just set minimum criteria to be accepted, which includes GPA, tests scores, research, jobs, whatever, and then randomly select from that group. That would certainly eliminate a huge amount bias, though not all, but most people hate this approach because it contradicts this belief that we live mostly in a meritocracy. They want rankings and they want to know who number 1 is. My continued argument is that it's impossible to distill a person down to a hard ranking in a reasonable matter, especially with test scores, and it's not necessary for college admission since one can select a group of applicants for acceptance without ordering them.

[triceratops]

I appreciate your willingness to engage and your detailed responses. I understand where you're coming from, and that you've seen how the sausage is made.

> In person tutoring worked better and by a lot, but it all helped.

Are money and other resources only helpful for getting a good test score? They don't matter at all for the factors that improve one's chances of admission? For example, going to a good high school district, participation in certain sports, hobbies, or access to internship, volunteer, or research opportunities?

My argument is simply: by making it about the test, you remove all the other factors where money could potentially have an influence. The rich students will get tutoring no matter what. But they can't gain any other advantages over the poor students.

> In that way, one could just set minimum criteria to be accepted, which includes GPA, tests scores, research, jobs, whatever, and then randomly select from that group.

That wouldn't be a terrible approach either. You are correct it would be unsatisfying.

> GPA and where they went to undergrad...were [both] hard metrics and full of bias.

I would argue they weren't "hard" at all. Undergrad is already gated by "squishy" criteria. Normalizing GPAs across universities (and countries, in the case of international applicants) is tricky and imprecise.

[kxyvr]

Money absolutely has an impact across easily available metrics. For example, money lets one purchase better coaching for sports or join a travel team. As we see from the lawsuit, athletes were accepted at a much higher rate than the rest of the applicants. Money lets one attend better schools either from a transfer or private school. Money allows for better tutoring. I completely agree.

Where I think we differ is that my contention is that money also allows one to game a higher test score on something like the SAT and by a significant factor. Yes, it would focus attention on this one factor, but I believe it also misses the point. College admissions is not meant to be a singular contest or game, but part of a broader ecosystem and that system has choices.

A university has a number of competing criteria and wants for its applicants. Generally speaking, a university does not want someone to drop out. They want graduates and part of screening applications is meant to determine whether they believe someone can successfully complete their degree. Some universities essentially operate like hedge funds that have classes. These universities have an interest in high net worth and connected applicants because these people can trade favors or give donations as alumnae. This is reflected in legacy admissions and automatic admission for certain classes of donors. Some universities want to give preference to in-state or in-city students. The purpose here is to ensure the university is locally connected because the state and city have the most influence on what is really a city inside of a city for most universities. Some universities want to give children of those who never went to college preference because it ultimately broadens their base and likely ensures their children will want to attend there as well. Some universities want to maximize their rating on the US News and World Report ranking because it means they can charge higher tuition.

None of those metrics above, moral or amoral alike, need a total ordering of applicants based on a test score.

As one other comment, when I sat on the graduate board, our department was flooded with applications from one particular country, more than half. Don't know why, but it was. Certainly, we accepted some of those applicants. However, consider the implications of accepting students purely from that one place. Would the local government be happy if we did not accept any local or domestic students? Would the federal? Probably not. However, if we accepted applications purely based on a singular test score, there would be incentive for that place to game the system and help with the test, legal or otherwise. No idea if they did then, but it wouldn't have mattered since we still read the broader application.

[ordx]

> I still contend strongly that it is wrong to distill a person into a collection of numbers.

Welcome to the adult world, where you are distilled into a collection of number each step of the way: look at credit scores, performance reviews, etc.

> I'll also mention that using tests scores is not entirely "fair" either.

Unfortunately we didn't come up with a better way to rank a group of people on scale in a reasonable amount of time. So tests are a necessary compromise. I also know plenty of examples where people aced their tests by just spending $30 on a prep book and studying hard.

[kxyvr]

Thankfully, I also work in the adult world and this world requires judgement.

As an example, my wife is a physician. She and her colleagues use metrics to help assess a patient, but ultimately treatment is based on their judgement. Purely relying on hard numbers, such as what an EKG gives, would lead to dead patients. Many of them. I am a mathematician. My world revolves around hard computational numbers, yet my algorithms will often given misleading results. Determining when that is requires judgement. Despite a push too the contrary, good hiring managers assess applicants and take responsibility for both good and bad hires. This requires judgement.

And, yes, I can also find examples where someone either didn't study or used a book and did well on their SAT. That said, in person test prep with a tutor absolutely has a large, positive effect on test scores, much more than simply a book. Kaplan has the internal data. It's absolutely worth the money if one can afford it. Most can't.

[ordx]

Your personal stories are very cute, but they don't explain what you're proposing instead of tests. I'm not sure how you're going to use so-called "judgment" to decide which students get in out of hundreds of thousands of applications in a reasonable amount of time.