| I'm mostly on board with the critique of university rankings, but I really don't buy this specific critique. It's a strawman. Here's what they did: they took the top research universities and the top liberal arts colleges. They then observed that the rankings can't distinguish between the top institutions within each class, and that you can also flip which of those two classes is preferred. But, in what sense is it even meaningful to compare Amherst College and Harvard [1]? Those are just enormously different types of institutions. It'd be like creating a ranking of "best cars" that includes the top 5 sedans and the top 5 trucks, and then observing that you can jiggle the rankings to get anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question. US News and World Report, for all its problems, does at least get this much right! they break down institutions by "type" and then rank within type. Additionally, just because rankings are noisy and easy to game locally doesn't mean they are inaccurate or easy to game globally. Two institutions within 10-20 slots of one another are probably pretty similar and rankings aren't particularly helpful / are easy to game. But the #100 liberal arts college is probably not as good as Amherst, and the #78 National University is probably not as good as Harvard, and no amount of gaming is going to change that. Rankings are indeed noisy and inaccurate and easy to game. But this particular article is not a compelling demonstration of that fact. [1] For non-US readers: Amherst college (not to be confused with UMass Amherst) is in a class of peculiar institutions that are fairly unique to the USA as described here: https://en.wikipedia.org/wiki/Liberal_arts_college TL;DR: they're basically the diametric opposite of super-charged research universities like Harvard. |
I don't think the paper gives any hints on us dividing universities into classes and comparing. Would love to know how you reached that conclusion.
At the same time, yes, we could have divided universities into different classes: research vs liberal arts, this vs. that state, big vs. small size, etc. These are all trivial groupings but none of these would change the conclusions of this paper. For all practical purposes, we could easily have replaced the university names with labels like U1, U2, ... and still the conclusions would not change. What matters is how a weight-based composite index can be gamed and the paper does show that in multiple ways. Pls review the ILP formulations yourself and run them on the dataset of your choice.