How to CRUSH Anecdotal Evidence

Sky Blue
3 min readDec 24, 2021
Coghill via Scienceornot

Do you find it hard to argue when a person pulls out the “it happened to someone I know” card? Well, turns out there is a quick way to shut down that argument.

Meet the Da Silvas, a Brazilian family with many members having 6 fingers. If I knew no one outside the family, and argued by anecdote saying “well, humans have six fingers, because I know many people who have six fingers”, then I would probably be laughed at.

The tricky part is to recognize that arguing, “well, I have five fingers, and so does my family, so humans have five fingers” is just as incorrect. Even though my conclusion happens to be mostly correct, I cannot generalize from a single tiny sample to an entire population. The Da Silvas are the living counter-example. Do not fall into the trap of assuming the logic is correct just because the final conclusion is.

From C. Mehring

Single anecdotes say nothing

So anybody who tells you “the vaccine works; I got COVID and it was so mild” is not arguing rigorously. Anybody who tells you “the vaccine does not work; I know someone who got it and died” is also not arguing rigorously. It doesn’t matter who is right and who is wrong; if someone tries using that argument for either side, you shut them down immediately. There will be people for whom the vaccine works, and those for whom it does not. An anecdote says one thing and one thing only: that both ways are possible. Nothing more, nothing less. To determine HOW probable requires statistical studies.

But are multiple anecdotes reliable to determine likelihood? No.

Multiple anecdotes also say nothing

In the same way that one anecdote does not work, 10 do not work either. Someone who says they know many people who got severe COVID after being vaccinated would not be reliable. The person is more likely to hear about their contacts that developed severe COVID, rather than the milder ones. This is called survivor bias, in which case one outcome is more represented because it is more visible. Similarly, a young person who claims that all their contacts only got mild COVID because of the vaccine would not be painting an accurate picture because the mildness could very much be due to age rather than anything. This is data bias where the stories come from the same type of people, and results may not apply to the general population.

There are two problems with anecdotes:

  • They tell us WHAT it is possible, and not HOW probable
  • We are not statisticians. The stories we collect around us are never an objective portrayal of reality.

--

--

Sky Blue
0 Followers

Common sense should be common ground.