Hacker News new | ask | show | jobs
by gjm11 2821 days ago
I don't think your reasoning is valid. A test might be unable to discriminate accurately at the extremes not because it's uniformly too inaccurate but because it doesn't have enough range.

Toy model: the test consists of one question that everyone in the top 1% can do and no one in the bottom 99% can do; one question that everyone in the top 2% can do and no one in the bottom 98% can do; ... one question that everyone in the top 99% can do and no one in the bottom 1% can do. This test discriminates very nicely and accurately throughout its range of applicability, but it will do no good at all from distinguishing a top-0.01% person from a merely top-1% person.

(Just as a tape measure 2m long will let you compare people very accurately by height provided they're no taller than 2m, but will be much less useful for people taller than that.)

1 comments

Modern tests are delivered by computer and typically are adaptive. This means that as you answer questions correctly you get asked increasingly difficult questions until you get some wrong.

This means that you aren't limited to asking the same questions to everyone so you can have appropriate discrimination through the range.