The Dunning-Kruger effect comes out of research conducted by David Dunning and Justin Kruger of Cornell University since 1999. I have not seen any diaries here about it, but it is surprising how often it is a propos to our discussions here. I hope to make it a major tool in our explanatory toolkit.
Over the years, Dunning and Kruger have conducted studies over a wide range of cognitive tasks. Following the skill assessment, subjects are shown their results, and then asked to rate their performance on a percentile basis. What they have found is that those performing well — say, 65th %ile and higher — exhibit a quite accurate assessment of where they placed, while those who do more poorly are unaware of their own poor performance. In fact, those who do quite badly, say at the 15th %ile, tend to overestimate their relative skill by 50% or more.
A way to summarize these results is to say that those who are truly stupid are too stupid to realize that they are stupid. As Dunning has put it, slightly less colorfully:
If you’re incompetent, you can’t know you’re incompetent.… [T]he skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is.