Personal anecdotes are often the primary ammunition of those who deny science. If you ask anyone in the alternative medicine or anti-vaccine movements for their evidence, you will almost certainly get flooded with anecdotes. A quick internet search will reveal countless people who are insisting that totally worthless treatments like homeopathy work because they took them and then felt better. These accounts are often accompanied by emotional stories about how they “tried everything but only [insert nonsense miracle cure] worked.”  Similarly, I frequently encounter people who are adamant that detox solutions aren’t scams or that organic food is better than GMOs because “they just feel healthier when they eat organic/use the detox supplement.”

Anti-vaccers are probably the worst group for using anecdotes. They use personal anecdotes to blame vaccines for every ailment imaginable, but they don’t just stop there. For them, collections of reported symptoms such as the vaccine package inserts, VAERS, and cases from the NVICP are the gold standards of evidence that vaccines are bad. Those sources are, however, really just collections anecdotes. Similarly, even when anti-vaccers attempt to use the scientific literature, they often end up accumulating case reports, which are essentially glorified anecdotes.

All of this would be fine if anecdotes were actually useful pieces of evidence, but they aren’t. As I will explain in this post, they are worthless, and if your argument is built on anecdotes, then your argument should be rejected.

Before I begin, I want to clarify what I mean when I say that anecdotes are worthless. They are worthless as evidence, and you cannot use them to establish causal relationships. You can’t, for example, say “Bob took X, then got better; therefore, X works.” You can, however, say “Bob took X, then got better; therefore, X might be an interesting topic for future research.” In other words, anecdotes can be useful in helping researchers decide what topics to study, what potential drugs to investigate, etc. However, in the absence of those large, carefully controlled studies, you cannot jump to the conclusion that a causal relationship exists. In other words, you can’t assume that X works until X has been properly tested, and, perhaps most importantly, if the tests disagree with the anecdotes, you must reject the anecdote, not the tests.

There are also a few other situations in which anecdotes can potentially be useful (e.g., if a patient is dying and a doctor has exhausted all science-based options, then and only then would it be appropriate to try a treatment which has only anecdotal evidence to support it). For the purpose of this post, however, I am just going to focus on why they are completely and totally invalid as evidence for causal relationships.

1). If you are using anecdotes, you are committing a logical fallacy
Anytime that someone uses an anecdote to argue that X causes Y, they are committing a logical fallacy known as post hoc ergo propter hoc (often abbreviated as simply post hoc). The Latin translates to “after this, therefore because of this,” and it occurs whenever an argument takes the following form:

• X happened before Y
• Therefore, X caused Y

The astute reader will quickly notice that the vast majority of personal anecdotes are identical to that syllogism. For example, if you say, “I took this supplement, then I felt better; therefore, the supplement works” you are committing a logical fallacy. Similarly, if you say, “I vaccinated my child, then he developed autism; therefore, vaccines cause autism” you are committing a logical fallacy. Also, if you say, “I switched to an organic diet, then I started feeling better; therefore, an organic diet is healthier” you are committing a logical fallacy. Am I making my point clear? Using personal anecdotes as evidence of causation is logically invalid, and the rules of logic tell us that any argument that contains a logical fallacy is unreliable and must be rejected.

The reason that post hoc arguments are invalid should be obvious: the fact that Y happened after X does not mean that X caused Y. Let’s say, for example, that you fill your vehicle with fuel from a reputable gas station, and your car breaks down just a few miles later. Can you conclude that the bad gas killed your car? No. It is certainly possible that bad gas was at fault, but it is also possible your car died from something totally unrelated to the gas, and getting gas was just a coincidence. Even so, the fact that you got better after taking X does not mean that X worked because there are many other factors that could have caused your recovery.

It is worth noting, that you can use the order of events to make a legitimate argument if you are making a probabilistic argument, and if a causal relationship has already been established. In other words, if you know based on actual evidence (not anecdotes) that X can cause Y, then if Y happens after X, it is not unreasonable to conclude that X probably caused Y. So, you can say,

• Item X is known to cause Y
• I took X, then Y happened
• Therefore, X probably caused Y

There is nothing wrong with that if and only if there is actually valid, scientific evidence that X can in fact cause Y. Also, the strength of the argument will depend on the strength of the relationship between X and Y (e.g., if X causes Y in 99% of cases, then it is a very strong argument, but if X only causes Y in 0.0000001% of cases, then it’s not a good argument because X almost never causes Y).

2). Anecdotes aren’t representative
Another major problem with anecdotes is that they don’t give you a proper representation of either the effects of X or the causes of Y. Let’s say, for example, that you are interested in miracle cure X, and when you get online, you find several people claiming that it worked for them. That doesn’t actually tell you much because it doesn’t tell you how many people X didn’t work for, nor does it tell you how many people recovered without X.

To give another example, anti-vaccers love to cite anecdotes of a symptom that followed a vaccine, but for every anecdote that they supply, I can supply anecdotes of people (like me) who received the full recommended vaccine schedule and are perfectly fine. Neither set of anecdotes is actually meaningful, because neither set is representative. To actually know whether or not X caused Y, we need the actual rates of Y relative to X, not just scattered reports. In other words, we need to know how many times Y followed X, how many times Y occurred without X occurring, and how many times X occurred but was not followed by Y (in some situations you may only need one of the later two, but you have to have at least one).

3). Anecdotes aren’t controlled
The third major problem with anecdotal evidence is that fact that they don’t control all possible factors. In other words, you can’t say, “I took X, then got better; therefore X works” because there may be something other than X that caused you to get better. In many cases, people simply get better on their own. For example, I often see people take a “remedy” for the common cold, continue to be sick for a day (or often several days), then get better, but after recovering, they insist that the remedy worked. The problem is, of course, that people normally get over colds in a few days. Therefore, it is utterly impossible (based on that anecdote) to determine if the remedy worked, or if their body simply took care of itself. As I explained in #2, this is why it is so important to know the actual rates of event Y relative X.

The placebo effect is another huge confounder. The placebo effect is often misunderstood and misrepresented (you can find good explanations/discussions here and here), but it is true that in many situations, people will report feeling better if they think that they are taking something that will help them, even if the treatment is totally worthless. This is especially true with highly subjective measurements like pain. So in some cases, people may report feeling better even if the treatment itself didn’t actually do anything.

There are many other potential factors that people fail to account for. Alternative medicine, for example, is famous for recommending a whole slew of treatments, then picking one as the responsible party. For example, I often hear people say things like, “I know X works, because my naturopath told my to exercise more, eat more vegetables, and take X, and I feel great now.” It seems rather silly to give X the credit if you you also started exercising more and eating healthier (both of which are actually supported by scientific evidence). Another one that I often encounter is, “my naturopath told me to do A, B, C, and eat less gluten, and I feel much better now, so gluten must be bad for you.” Again, how do you know that it was gluten and not A, B, or C? Those two examples contain pretty obvious confounding factors, but confounding factors may be much more subtle, and you may not even be aware of them. So, even if, to your knowledge, X is the only thing that has changed, there may be some other change that you haven’t thought about or just aren’t aware of.

Finally, it’s worth noting that the fact that, in some situations, we cannot identify the actual cause of an event does not mean that you can assume that it was X. In other words, if you say, “X causes Y, because I took X and Y happened,” and someone calls you out for using an anecdote, you can’t respond with, “Well if it wasn’t X, then what was what? Unless you can prove that it was something else, it must have been X.” That argument is actually another logical fallacy. Specifically, it is an argument from ignorance fallacy. The fact that I don’t know what caused Y doesn’t mean that it was X, and it’s not logically valid for you to jump to that conclusion. To put this another way, by claiming that X causes Y, you are placing the burden of proof on you, and it is your job to provide actual evidence that X causes Y. It’s not my job to provide evidence that X doesn’t cause Y.

4). An anecdote is a sample size of N=1
The importance of sample size is one of the most fundamental concepts in statistics. The larger your sample size, the more power that you have and the more confident you can be in your results. An anecdote, however, is simply a single observation, and extrapolating from a single observation to a general trend is an absurd thing to do. Imaging, for example, that you want to know whether or not a coin is biased, so you flip it twice and it lands on heads both times. Should you conclude that the coin is biased? Of course not. A sample size that small is meaningless because it is entirely possible (even likely) that you got a biased result just by chance. The same thing is true with anecdotes. Saying, “I vaccinated my kid, then he developed autism; therefore, vaccines cause autism” isn’t substantially different (as far as sample size) from saying, “I flipped the coin twice and got heads both times; therefore, the coin is biased.” Tiny sample sizes simply aren’t reliable.

5). Anecdotes aren’t collected systematically
Following my argument in #4, you may be thinking, “but I have met lots of people on the internet with identical anecdotes, so my sample size is much larger than just one.” The problem with that argument is that the anecdotes were not collected in a systematic way. This is really an overarching problem which overlaps substantially with points 2, 3, and 4, but it is important enough that I want to talk about it separately.

One of the hallmarks of science is being systematic. Real research is done in a careful, planned, controlled, repeatable fashion, and that systematic approach is a big part of why science is such a powerful tool for understanding the universe. For example, when we want to answer a question like, “do vaccines cause autism?” we don’t just haphazardly find someone on the internet. Rather, we carefully select a representative study population, control for confounding factors, use large sample sizes, and measure the actual rates of autism in both children with and without vaccines (for example, Taylor et al. 2014). That approach and that approach alone allows us to overcome the problems described in #2-4 and actually achieve a reliable answer. Anecdotes, on the other hand, are in no way systematic, which makes them exceedingly unreliable and unscientific.

Conclusion
In summary, using anecdotes as evidence of causation commits a logical fallacy, which means that anecdotal arguments must be rejected. Further, anecdotes don’t give you a fair representation of the effects of X on Y, nor do they account for potential confounding factors. Therefore, anecdotes are worthless as evidence. They simply cannot demonstrate causal relationships. As I often say on this blog, if you want to know whether or not X causes Y, the one and only way to do it is by conducting large, properly controlled studies that account for confounding variables. Nothing else will suffice. It doesn’t matter if you have “seen it work,” it doesn’t matter if something has been used for centuries, and it doesn’t matter if a symptom has been reported in a database like VAERS or printed on a package insert. Unless proper scientific testing has shown that X causes Y, you cannot conclude that there is a causal relationship between the two.