[TangoTrotFox]

Why do you think they don't measure knowledge and ability? They're not perfect of course, but I mean asking you to demonstrate your competence in a series of problems is going to give at least a reasonable ballpark of somebody's competence.

Yeah, things like preparation classes can kind of turn it all into gaming the system, but on the other hand I'm quite curious what percent of people that scored e.g. perfectly on the math on the SAT actually took these classes. I did, and I never did any preparation - but I've always enjoyed and done well at math, and I think my performance was reflected in this. And I don't think somebody who had no ability or enjoyment of math is suddenly going to start scoring very well just because of some prep classes.

[truculation]

The way I see it. Exams can check recall of facts, names, definitions. Also skill at identifying narrow classes of problems (which resemble facts) and extracting them from unrealistic settings. e.g. 'There are 49 dogs signed up to compete in the dog show. There are 36 more small dogs than large dogs signed up to compete. How many small dogs are signed up to compete?' and other such trick questions.

The former can be crammed and quickly forgotten. Not knowledge. The latter skill isn't knowledge of the relevant field.

In the less rigorous, more arty fields (shall we say) the trick is usually to flatter the examiners by firing their own opinions back at them in original ways. This can be highly skillful and requires awareness of the academic milieu. But not knowledge of the relevant field.

What counts is depth, and depth depends on semantic connections, including connections to other fields. These are all differentiated and can't be meaningfully added to yield a number, as if we were counting eggs or measuring a distance.