Personal anecdotes are often the primary ammunition of those who deny science. If you ask anyone in the alternative medicine or anti-vaccine movements for their evidence, you will almost certainly get flooded with anecdotes. A quick internet search will reveal countless people who are insisting that totally worthless treatments like homeopathy work because they took them and then felt better. These accounts are often accompanied by emotional stories about how they “tried everything but only [insert nonsense miracle cure] worked.” Similarly, I frequently encounter people who are adamant that detox solutions aren’t scams or that organic food is better than GMOs because “they just feel healthier when they eat organic/use the detox supplement.”
Anti-vaccers are probably the worst group for using anecdotes. They use personal anecdotes to blame vaccines for every ailment imaginable, but they don’t just stop there. For them, collections of reported symptoms such as the vaccine package inserts, VAERS, and cases from the NVICP are the gold standards of evidence that vaccines are bad. Those sources are, however, really just collections anecdotes. Similarly, even when anti-vaccers attempt to use the scientific literature, they often end up accumulating case reports, which are essentially glorified anecdotes.
All of this would be fine if anecdotes were actually useful pieces of evidence, but they aren’t. As I will explain in this post, they are worthless, and if your argument is built on anecdotes, then your argument should be rejected.
Before I begin, I want to clarify what I mean when I say that anecdotes are worthless. They are worthless as evidence, and you cannot use them to establish causal relationships. You can’t, for example, say “Bob took X, then got better; therefore, X works.” You can, however, say “Bob took X, then got better; therefore, X might be an interesting topic for future research.” In other words, anecdotes can be useful in helping researchers decide what topics to study, what potential drugs to investigate, etc. However, in the absence of those large, carefully controlled studies, you cannot jump to the conclusion that a causal relationship exists. In other words, you can’t assume that X works until X has been properly tested, and, perhaps most importantly, if the tests disagree with the anecdotes, you must reject the anecdote, not the tests.
There are also a few other situations in which anecdotes can potentially be useful (e.g., if a patient is dying and a doctor has exhausted all science-based options, then and only then would it be appropriate to try a treatment which has only anecdotal evidence to support it). For the purpose of this post, however, I am just going to focus on why they are completely and totally invalid as evidence for causal relationships.
1). If you are using anecdotes, you are committing a logical fallacy
Anytime that someone uses an anecdote to argue that X causes Y, they are committing a logical fallacy known as post hoc ergo propter hoc (often abbreviated as simply post hoc). The Latin translates to “after this, therefore because of this,” and it occurs whenever an argument takes the following form:
- X happened before Y
- Therefore, X caused Y
The astute reader will quickly notice that the vast majority of personal anecdotes are identical to that syllogism. For example, if you say, “I took this supplement, then I felt better; therefore, the supplement works” you are committing a logical fallacy. Similarly, if you say, “I vaccinated my child, then he developed autism; therefore, vaccines cause autism” you are committing a logical fallacy. Also, if you say, “I switched to an organic diet, then I started feeling better; therefore, an organic diet is healthier” you are committing a logical fallacy. Am I making my point clear? Using personal anecdotes as evidence of causation is logically invalid, and the rules of logic tell us that any argument that contains a logical fallacy is unreliable and must be rejected.
The reason that post hoc arguments are invalid should be obvious: the fact that Y happened after X does not mean that X caused Y. Let’s say, for example, that you fill your vehicle with fuel from a reputable gas station, and your car breaks down just a few miles later. Can you conclude that the bad gas killed your car? No. It is certainly possible that bad gas was at fault, but it is also possible your car died from something totally unrelated to the gas, and getting gas was just a coincidence. Even so, the fact that you got better after taking X does not mean that X worked because there are many other factors that could have caused your recovery.
It is worth noting, that you can use the order of events to make a legitimate argument if you are making a probabilistic argument, and if a causal relationship has already been established. In other words, if you know based on actual evidence (not anecdotes) that X can cause Y, then if Y happens after X, it is not unreasonable to conclude that X probably caused Y. So, you can say,
- Item X is known to cause Y
- I took X, then Y happened
- Therefore, X probably caused Y
There is nothing wrong with that if and only if there is actually valid, scientific evidence that X can in fact cause Y. Also, the strength of the argument will depend on the strength of the relationship between X and Y (e.g., if X causes Y in 99% of cases, then it is a very strong argument, but if X only causes Y in 0.0000001% of cases, then it’s not a good argument because X almost never causes Y).
2). Anecdotes aren’t representative
Another major problem with anecdotes is that they don’t give you a proper representation of either the effects of X or the causes of Y. Let’s say, for example, that you are interested in miracle cure X, and when you get online, you find several people claiming that it worked for them. That doesn’t actually tell you much because it doesn’t tell you how many people X didn’t work for, nor does it tell you how many people recovered without X.
To give another example, anti-vaccers love to cite anecdotes of a symptom that followed a vaccine, but for every anecdote that they supply, I can supply anecdotes of people (like me) who received the full recommended vaccine schedule and are perfectly fine. Neither set of anecdotes is actually meaningful, because neither set is representative. To actually know whether or not X caused Y, we need the actual rates of Y relative to X, not just scattered reports. In other words, we need to know how many times Y followed X, how many times Y occurred without X occurring, and how many times X occurred but was not followed by Y (in some situations you may only need one of the later two, but you have to have at least one).
3). Anecdotes aren’t controlled
The third major problem with anecdotal evidence is that fact that they don’t control all possible factors. In other words, you can’t say, “I took X, then got better; therefore X works” because there may be something other than X that caused you to get better. In many cases, people simply get better on their own. For example, I often see people take a “remedy” for the common cold, continue to be sick for a day (or often several days), then get better, but after recovering, they insist that the remedy worked. The problem is, of course, that people normally get over colds in a few days. Therefore, it is utterly impossible (based on that anecdote) to determine if the remedy worked, or if their body simply took care of itself. As I explained in #2, this is why it is so important to know the actual rates of event Y relative X.
The placebo effect is another huge confounder. The placebo effect is often misunderstood and misrepresented (you can find good explanations/discussions here and here), but it is true that in many situations, people will report feeling better if they think that they are taking something that will help them, even if the treatment is totally worthless. This is especially true with highly subjective measurements like pain. So in some cases, people may report feeling better even if the treatment itself didn’t actually do anything.
There are many other potential factors that people fail to account for. Alternative medicine, for example, is famous for recommending a whole slew of treatments, then picking one as the responsible party. For example, I often hear people say things like, “I know X works, because my naturopath told my to exercise more, eat more vegetables, and take X, and I feel great now.” It seems rather silly to give X the credit if you you also started exercising more and eating healthier (both of which are actually supported by scientific evidence). Another one that I often encounter is, “my naturopath told me to do A, B, C, and eat less gluten, and I feel much better now, so gluten must be bad for you.” Again, how do you know that it was gluten and not A, B, or C? Those two examples contain pretty obvious confounding factors, but confounding factors may be much more subtle, and you may not even be aware of them. So, even if, to your knowledge, X is the only thing that has changed, there may be some other change that you haven’t thought about or just aren’t aware of.
Finally, it’s worth noting that the fact that, in some situations, we cannot identify the actual cause of an event does not mean that you can assume that it was X. In other words, if you say, “X causes Y, because I took X and Y happened,” and someone calls you out for using an anecdote, you can’t respond with, “Well if it wasn’t X, then what was what? Unless you can prove that it was something else, it must have been X.” That argument is actually another logical fallacy. Specifically, it is an argument from ignorance fallacy. The fact that I don’t know what caused Y doesn’t mean that it was X, and it’s not logically valid for you to jump to that conclusion. To put this another way, by claiming that X causes Y, you are placing the burden of proof on you, and it is your job to provide actual evidence that X causes Y. It’s not my job to provide evidence that X doesn’t cause Y.
4). An anecdote is a sample size of N=1
The importance of sample size is one of the most fundamental concepts in statistics. The larger your sample size, the more power that you have and the more confident you can be in your results. An anecdote, however, is simply a single observation, and extrapolating from a single observation to a general trend is an absurd thing to do. Imaging, for example, that you want to know whether or not a coin is biased, so you flip it twice and it lands on heads both times. Should you conclude that the coin is biased? Of course not. A sample size that small is meaningless because it is entirely possible (even likely) that you got a biased result just by chance. The same thing is true with anecdotes. Saying, “I vaccinated my kid, then he developed autism; therefore, vaccines cause autism” isn’t substantially different (as far as sample size) from saying, “I flipped the coin twice and got heads both times; therefore, the coin is biased.” Tiny sample sizes simply aren’t reliable.
5). Anecdotes aren’t collected systematically
Following my argument in #4, you may be thinking, “but I have met lots of people on the internet with identical anecdotes, so my sample size is much larger than just one.” The problem with that argument is that the anecdotes were not collected in a systematic way. This is really an overarching problem which overlaps substantially with points 2, 3, and 4, but it is important enough that I want to talk about it separately.
One of the hallmarks of science is being systematic. Real research is done in a careful, planned, controlled, repeatable fashion, and that systematic approach is a big part of why science is such a powerful tool for understanding the universe. For example, when we want to answer a question like, “do vaccines cause autism?” we don’t just haphazardly find someone on the internet. Rather, we carefully select a representative study population, control for confounding factors, use large sample sizes, and measure the actual rates of autism in both children with and without vaccines (for example, Taylor et al. 2014). That approach and that approach alone allows us to overcome the problems described in #2-4 and actually achieve a reliable answer. Anecdotes, on the other hand, are in no way systematic, which makes them exceedingly unreliable and unscientific.
In summary, using anecdotes as evidence of causation commits a logical fallacy, which means that anecdotal arguments must be rejected. Further, anecdotes don’t give you a fair representation of the effects of X on Y, nor do they account for potential confounding factors. Therefore, anecdotes are worthless as evidence. They simply cannot demonstrate causal relationships. As I often say on this blog, if you want to know whether or not X causes Y, the one and only way to do it is by conducting large, properly controlled studies that account for confounding variables. Nothing else will suffice. It doesn’t matter if you have “seen it work,” it doesn’t matter if something has been used for centuries, and it doesn’t matter if a symptom has been reported in a database like VAERS or printed on a package insert. Unless proper scientific testing has shown that X causes Y, you cannot conclude that there is a causal relationship between the two.
Now don’t be naughty Fallacy Man, you know you are going waaay beyond your evidence base in saying anecdotes are worthless and we have been over this
1. Phase 4 “clinical trials”: collection of anecdotes
2. Anecdotes might not be biased and hence usable as data.
3. Phenology is worthless?
4. Do doctors just ignore verbal description of pain? After all just one subjective patient giving their opinion of their symptoms.
5. Jurisprudence? Causality is inferred from anecdotes.
6. Exceptions are interesting: why has the 105 year old smoker survived?
7. People publish science based on anecdotes including me and one former editor of Current Biology known to me.
Personal experience counts for something and the scientists/experiments could be wrong (it is unlikely in any given case but…). Anecdotes are not worthless they are just potentially very bad misleading evidence. For inferring causality clearly worse than experiments and even observational studies but the pie I ate last night might have given me food poisoning and I am not going to construct a trial to prove it. I am not going to say to a mother “you cannot show from a clinical trial little Henry is allergic to peanuts so let him eat a Snickers.” Rational people use anecdotes to infer causation. We have to. So by all means say anecdotes are weak evidence the worst we have but not necessarily worthless nor even unscientific if used with caveats. Plus ultra.
LikeLiked by 3 people
Somehow, I thought that you might comment 🙂 I actually tried to avoid this by clearly stating that anecdotes do have value in some areas (especially for providing topics for future research). They just aren’t useful as evidence of causation. In other words, you can use them to say that there might be a causal relationship, but you cannot use them to say that there is a causal relationship (this covers points 1, 6, and 7).
I also said that you can reach causal conclusions if a previous causal relationship has already been established. For example, when a doctor is diagnosing a patient, and the patient reports Y, he applies that report to the current knowledge of what causes Y. In other words, if the patient says, “I did X and then Y happens,” the doctor will only conclude that X caused Y in this case if X is already known to cause Y. That is a very different thing from what using anecdotes to establish causation. This covers 4 and 5.
As far as point 1, sure, a particular anecdote might be correct, but without proper controls, sample size, etc. we have no way of knowing that it is correct. That’s my whole point.
For point 3, I totally disagree that phenology is based on anecdotes. Studies often begin with anecdotes (just as medical studies often begin case reports), but the actual studies themselves are very systematic, often try to replicate results over several years, account for other potential causes, control by observing patterns during multiple parts of the year, etc. Obviously they aren’t randomized controlled trials (because those don’t work in ecology), but they are far from being anecdotes.
In short, I agree with you that there are areas where anecdotes have value (and I said that in the post), but none of your objections deal with the problem that I was dealing with in this post.
LikeLiked by 1 person
Let me put this one other way for clarity (since apparently I struggle with that). All that I am saying is that without other evidence to show that X can cause Y, you cannot use an anecdote (i.e., the fact that Y happened after X) to conclude that X causes Y. Doing so is a logical fallacy. That is what I am arguing in this post, and I don’t see anything in your comment that disagrees with that. Your comment seems to fall under the category of “There are also a few other situations in which anecdotes can potentially be useful.” I really did try to make it clear what I meant when I said “totally worthless,” but obviously I did not succeed.
LikeLiked by 1 person
Can’t begin to tell you how many times I piss off family and friends because my brain follows my formal science training and experience. I don’t mean to. It just happens. And what they don’t see is that I’m not being contrarian but rather concerned.
LikeLiked by 1 person
A good friend of mine uses anecdotal evidence all the time and he is almost always wrong. Speaks volumes about anecdotes.
LikeLiked by 1 person
Saw what you did there. 😉
LikeLiked by 2 people
Arrrrrrgh. I see this all the time, especially in medical applications. Hopefully better and more general medical care will cut down on this, but government oversight of supplements really needs to happen. Half of my family hates me because I just cannot help pointing out the bs.
LikeLiked by 1 person
My only comment refers to your placebo discussion. Rather than say that the “placebo didn’t do anything”, it would be better to say that the placebo mechanism of action may be based on the subjective belief of the patient rather than on an objective alteration of some physical parameter.
LikeLiked by 1 person
Science, senses have obvious limits. You are correct but it comes across in whole as going too far as many new scientific discoveries are found not by the scientific method but by possibilities unseen. You said it but in an either or world, I wonder if it was heard.
LikeLiked by 1 person
I recently had a frustrating conversation on Facebook with someone who believed that eating GMO’s had caused her Crohn’s Disease. Every serious medical authority website I checked, made it clear that Crohn’s was not caused by diet, but my friend’s line of reasoning was pretty much…. “Well I got Crohn’s after I ate GMO’s, therefore that MUST be the cause!”
In other words almost a textbook example of “Proof By Anecdote.”
In this particular case I just had to “agree to disagree” with my friend… Mainly because it’s quite difficult to argue with someone’s personal account of how they contracted a painful and disabling illness, without accidentally coming across as an arrogant and insensitive “know it all.”
LikeLiked by 1 person
Person A: my feet hurt because I ate GMO s
Person B: you are standing barefoot on broken glass
Person A: Don’t invalidate my feelings, I KNOW what happened
Person B: No, seriously, your feet are bleeding…
Person A: yes, I know. Because of GMO s
Person B: What is wrong with you? Broken glass, barefoot
Person A: I bet you are being paid by Big Shoe
LikeLiked by 2 people
Pah, person’s B’s observation that the person A was standing on glass is a mere single observation (=anecdote) and thus equally worthless. We can only infer why Person A feet are bleeding by putting his feet though multiple trials to prove causation.
Hoisted by your own petard.
LikeLiked by 1 person
I think an important issue here is what do we mean by “anecdote” and does it relate to data. Is it
a) The narrative of a detached incident, or of a single event, told as being in itself interesting or striking.(from OED)
b) Secret, private, or hitherto unpublished narratives or details of history (from OED, humanities so not relevant but still evidence 😉 )
c) a second or third-hand tale (similar to a) but less reliable)
d) an amusing story (OK well I think we can ditch this definition for our purposes)
e) personal experience (similar to a)
f) something else (I cannot think of anything but maybe others can)
These definitions are not necessarily mutually exclusive.
Now a) can be viewed as a single datum, hence such cases could be collected and the question then becomes is the collection unbiased relative to what you want to study. If they are biased there is a problem but if not then they are usable and could be used to infer relationships and maybe causation.
Here are some papers that use anecdotes in different ways, some more dangerous than others. But note they still call them anecdotes even though they are using them (sort of) as data.
Byrne R.W. (1997) What’s the use of anecdotes? Attempts to distinguish psychological mechanisms in primate tactical deception. In R. Mitchell, Nicholas S. Thompson & H. L. Miles (eds.), Anthropomorphism, Anecdotes, and Animals. Suny Press 134–150 (1997) .
Al-Abdulrazzak D, Naidoo R, Palomares MLD, Pauly D (2012) Gaining Perspective on What We’ve Lost: The Reliability of Encoded Anecdotes in Historical Ecology. PLoS ONE 7(8): e43386. doi: 10.1371/journal.pone.0043386
Parsons, D. M., Morrison, M. A., MacDiarmid, A. B., Stirling, B., Cleaver, P., Smith, I. W. G., and Butcher, M. (2009). Risks of shifting baselines highlighted by anecdotal accounts of New Zealand’s snapper (Pagrus auratus) fishery. New Zealand Journal of Marine and Freshwater Research 43, 965–983.
Clearly part of the difference between me is that I would call them “anecdotes that are also data” and hence conclude some anecdotes are worthwhile whilst you would say “they are anecdotes that have become data”, so anecdotes are worthless.
That is a semantic difference so let’s agree to differ but your usage seems strange to me.
A single incident might be useful in another way in just establishing that something can happen may be in animal behaviour, psychology or meteorology or similar. I think you would agree with that to.
And perhaps if one has no other info maybe one can/must use a single anecdote to infer causation. This I suspect is where we still differ. I would dispute that my point 5 above (anecdotes in law), that the evidential value of a single anecdote would depend on whether it existed into some pre-existing theory. And I would add that clearly we use single anecdotes day to day. Now that isn’t science but it does mean anecdotes are not worthless.
The other issue is the value of observational studies in inferring causation. In practise of course scientists infer causation from observational studies all the time (how could astronomy work by experiment?). It is less certain than experiment but we do do it.
LikeLiked by 1 person
I have to share this xkcd: https://xkcd.com/552/
LikeLiked by 2 people