[Groxx]

Not much on "why", just that it exists. The "why" is essentially a re-statement of the situation: illusory superiority and illusory inferiority.

A common "why" theory that I personally subscribe to: the less competent don't know enough to know what they don't know.

[stcredzero]

This happens often in music, where subjectivity comes into play. What's not immediately obvious: programmers use subjective measures of quality all the time. We have to do so to get our work done in a timely fashion.

Those who want to be truly great should be humble. They should also check themselves as much as possible with objective measures and feedback from trusted and respected fellow practitioners.

Come to think of it, I suspect this phenomenon affects the implementation of programming languages. A lot of second tier CS talents implement languages because they have enough ego to get through the ordeal.

[sunkencity]

Yep it's whether you are an active information-seeker or not: If you never question your current model of thought it is a lot harder to learn, you'll mostly learn things that match what you know and the ideas and perspectives you can apply to a problem will be fairly limited.

Contrast with a competent person, that have a knack for picking up new stuff, they are used to learning stuff so they know that they don't know everything.

[sinc]

You are right, I should not have put the "why" there (and can't change it anymore). If someone with editing abilities sees this, could you please change the title to "The Dunning–Kruger effect", or what you think is more appropriate. Thanks.