Human reasoning can fail in many different ways. One of the most pernicious common things we get wrong is overestimating our own abilities.
As a great illustration of this, Michael Mauboussin and Dan Callahan of Credit Suisse Global Financial Strategies performed an ad hoc experiment. They wrote a trivia quiz, which you can take online here, with an interesting twist: In addition to answering 30 true or false questions, respondents were asked how confident they were in their answer for each question.
A respondent who had no idea what the right answer was would say they were 50% sure in their response: They were basically guessing. A respondent who was certain that they had the right answer would say they were 100% sure. If a respondent was somewhere in the middle, they could say that they were 60%, 70%, 80%, or 90% sure of their answer depending on their subjective assessment.
Based on the responses of 1,985 anonymous quiz takers, Mauboussin and Callahan found that most respondents were overconfident in their knowledge. The more confident people were in their answers, the further away their perceived probability of being right was from their actual probability of being right.
As Mauboussin and Callahan wrote (emphasis ours),
"However, as the assigned probability of correctness rose, the subjects became less calibrated. For instance, when the subjects selected 100 percent, they were only correct 77 percent of the time. At 90 percent, they were only correct 65 percent of the time. Overestimation of ability was greatest at the high levels of assigned probability of correctness."
Mauboussin and Callahan included this chart comparing the test-takers' self-assessed probabilities of being right along the horizontal axis with the actual probabilities of being right. While respondents who said they were guessing did actually get about half of the questions right, there's a big gap between perceived ability and actual correct answers as test-takers felt more confident:
