[austin-cheney]

The problem in software is not that Dunning-Kruger exists, but the frequency with which it exists and how that frequency corresponds to Dunning-Kruger related research.

Most research in Dunning-Kruger related experiments makes a glaring assumption that results on a test are evenly distributed enough to divide those results into quartiles of equal numbers and the resulting population groups are both evenly sized and evenly distributed within a margin of error.

That is fine for some experiment, but what happens in the real world when those assumptions no longer hold? For example what happens when there is a large sample size and 80% of the tested population fails the evaluation criteria? The resulting quartiles are three different levels of failure and 1 segment of acceptable performance. There is no way to account for the negative correlation demonstrated by high performers and the performance difference between the three failing quartiles is largely irrelevant.

Fortunately, software leadership is already aware of this problem and has happily solved it by simply redefining the tasks required to do work and employing heavy use of external abstractions. In other words simply rewrite the given Dunning-Kruger evaluation criteria until enough people pass. The problem there is that it entirely ignores the conclusions of Dunning-Kruger. If almost everybody can now pass the test then suddenly the population is majority over-confident.

[Joel_Mckay]

"software leadership is already aware of this problem"

What makes you so sure? In general, most security certifications HR gets excited about aren't worth the paper they are printed on.

Process people by their very nature are an unsustainable part of a poisoned business model.

The other misconception is a group of persistent well-funded knuckle-dragging troglodytes are somehow less likely to discover something Einstein overlooked.

https://en.wikipedia.org/wiki/Illusion_of_control#By_proxy

[CRConrad]

> Most research in Dunning-Kruger related experiments makes a glaring assumption that results on a test are evenly distributed enough...

Not only evenly distributed; isn't the very first underlying assumption they make, so fundamental that they never even mention it, that the tests are more accurate than the self-evaluation? Sure, over time and across a population they probably are, but that's not (as I understood it) what they measured here.

Haven't we all been there sometimes -- took a test on something we actually know pretty well, but got questions on the one sub-area we know less about (or just had a bad day), so we got a worse test result than what actually reflects our knowledge? Or the other way, took a test on something we don't know as well as we should, but lucked out with the questions hitting exactly what little we know (or got in some lucky guesses), so the test result is better than we actually deserve? I sure have.

That's another source of uncertainty, and directly relevant to what they're trying to investigate, so it feels like a big minus that they just totally ignore it.