[CRConrad]

> Most research in Dunning-Kruger related experiments makes a glaring assumption that results on a test are evenly distributed enough...

Not only evenly distributed; isn't the very first underlying assumption they make, so fundamental that they never even mention it, that the tests are more accurate than the self-evaluation? Sure, over time and across a population they probably are, but that's not (as I understood it) what they measured here.

Haven't we all been there sometimes -- took a test on something we actually know pretty well, but got questions on the one sub-area we know less about (or just had a bad day), so we got a worse test result than what actually reflects our knowledge? Or the other way, took a test on something we don't know as well as we should, but lucked out with the questions hitting exactly what little we know (or got in some lucky guesses), so the test result is better than we actually deserve? I sure have.

That's another source of uncertainty, and directly relevant to what they're trying to investigate, so it feels like a big minus that they just totally ignore it.