Is the peer-review system broken? A look at the PLoS ONE paper on a hand designed by “the Creator”

abstract from PLoS ONE paper god creator hand design

This is the abstract from the paper with the relevant phrase highlighted.

The internet has recently gone nuts over a scientific paper published in PLoS ONE (a generally respectable journal) which contained several lines suggesting that the human hand was designed by “the Creator.” The paper was quickly retracted, but the brouhaha continues, so I want to briefly talk about the controversy and what it can teach us about the scientific process and the peer-review system.

What was the paper?
The paper in question is “Biomechanical characteristics of hand coordination in grasping activities of daily living” by Lui, Xiong, and Huang. The paper talks about how the hand’s structure and biomechanics allow it to function in a highly versatile way, and the science seems okay, except that several places make really bizarre jumps to divine conclusions. For example, “Hand coordination should indicate the mystery of the Creator’s invention.” When I read lines like that, my instant thought was, “this has to be a translation error.” The paper was clearly not written by native English speakers, and the references to the “Creator” were so jarring and out of place that it seemed like surely it was a mistake. Indeed, the authors have now stated that it was.

We are sorry for drawing the debates about creationism. Our study has no relationship with creationism. English is not our native language. Our understanding of the word Creator was not actually as a native English speaker expected. Now we realized that we had misunderstood the word Creator. What we would like to express is that the biomechanical characteristic of tendious connective architecture between muscles and articulations is a proper design by the NATURE (result of evolution) to perform a multitude of daily grasping tasks. We will change the Creator to nature in the revised manuscript. We apologize for any troubles may have caused by this misunderstanding. We have spent seven months doing the experiments, analysis, and write up. I hope this paper will not be discriminated only because of this misunderstanding of the word. Please could you read the paper before making a decision.

Nevertheless, because of the uproar of the scientific community, PLoS ONE has decided to retract the paper. Personally, I think that full retraction is unnecessary. Anyone who reads this blog knows that I am not in anyway a creationist, and I expect science to be held to a high standard. Further, I do think that this error represents a serious mistake on the part of the editors and reviewers. Nevertheless, this does appear to be an honest mistake by the authors, and the science of the paper seems sound. So my personal opinion is that the paper should be revised to correct the translation issues, sent back out to new reviewers who will double check the science, then a decision should be made about rejecting or accepting it based on those reviews. I know how much work goes into doing research and writing a paper, so it seems like a shame to have that go to waste just because of a translation issue. Further, given that the authors are not native speakers, I think that most of the blame for this lies with the reviewers and editors, not the authors.  Nevertheless, let’s continue to examine the issue further.

What’s the big deal?
Some people may be wondering why this is even important. So what if a paper mentioned “the Creator”? Further, I can easily see many Christians responding with outrage and insisting that this is evidence that scientists are all closed minded atheists who are setting out to prove that God doesn’t exist (this is a common an irrelevant attack that they use to dismiss science).

The reality is that this actually has nothing to do with scientists being atheists (many aren’t); rather, it has to do with the nature of science itself. One of the fundamental guiding principles of science is that we live in a natural world that is governed by natural laws. Let me clear, this is not an inherently atheistic viewpoint. Rather, it is a necessary starting point for scientific inquiry. You see, if we don’t view the world as a natural system that is governed by natural laws, then studying it becomes futile because we can invoke God anytime that we want for anything that we want. Anytime that we have an unexplained phenomena, we can just plug God in as the answer and move on. Indeed, if nature is being directly governed by God rather than the laws of physics, then there is really no point in even studying it, because we will invariably get to a point where there is no scientific answer.

This leads to the next major problem. Namely, science is inherently incapable of answering questions about the supernatural. The type of conclusion that the authors’ mistake suggested is a type of conclusion that science simply can’t arrive at. Science is, by definition, the study of the physical universe, whereas God (if he exists) would inherently by supernatural, and science can’t address questions about the supernatural. Thus, the conclusion, “God did it” is a conclusion that science isn’t capable of reaching.

Therefore, the strong reaction of scientists isn’t evidence that all scientists are atheists who are angry at the concept of God. Rather, scientists reacted strongly because this type of reasoning is a fundamental assault on the very nature of science.

Is peer-review fundamentally flawed?
The peer-review system is not perfect. Let me say that again, the peer-review process is an imperfect system and bad papers do sometimes get through. Ultimately, the system relies on humans, and humans are flawed. Thus, flawed papers sometimes get published, but that doesn’t mean that the system is worthless.

In this particular case, I’m truly dumbfounded about how this paper got through, because one of the references to the “Creator” is in the abstract. Here’s the important catch though: it’s easy to point out examples where the system clearly failed, but you also have to consider how many times the system worked. In other words, we see the glaring examples of flawed papers that made it through the review process, but what we don’t see are the countless thousands of papers that didn’t make it through.

To put this another way, given the millions of scientific papers that get submitted to journals, it is inevitable that a few bad ones will get through; however, it would be a huge mistake to tout those examples as evidence that the system doesn’t work, because that would ignore all of the times that the system did work. Editors and reviewers get overworked, and sometimes they get lazy,  but that’s not the norm. Most editors and reviewers do a very thorough job of reviewing papers. I have personally sent multiple papers through the review system, and I usually get back an extremely lengthy list of comments that critiqued every aspect of my paper, and a mistake like this would have been caught before the editor even sent the paper out for reviews. So you should not view this type of mistake as being normal.

The system worked!
If you try to use this paper as evidence that the peer-review system doesn’t work, then you are missing the fact that the system actually did work! As I’ve often argued, peer-review doesn’t end when a paper is published. Rather, the paper will be critiqued by other researchers who read it, and if serious mistakes are found, they will contact the editors and the paper will be retracted. That is exactly what happened here. Other scientists spotted the mistakes that the editors/reviewers missed, they brought attention to the issue, and the paper was retracted. So, rather than providing evidence that the peer-review system is fundamentally broken, this is a great example of the system working. Yes, there was a very serious mistake at one stage the process, but the other stages corrected that mistake. That is one of the best things about science: it is self correcting.

Conclusion
In short, yes, this was a serious mistake. It is a huge indictment on PLoS One, and they should take a very long hard look at their editorial staff and policies. However, this example and others like it do not prove that they system as a whole is broken beyond repair. Yes, the system is imperfect, and yes, mistakes do happen, but overall, the system works and it both prevents many bad papers from being published and often removes the bad papers that slipped through the formal review process. The great thing about science is that if you make a mistake, there are thousands of other researchers you are ready and willing to tear your work to shreds.

 

 

Posted in Nature of Science | Tagged | 2 Comments

8 lessons that MythBusters taught us about science and skepticism

mythbustersThis is a sad week for me, because this week I must bid farewell to one of my all time favorite TV shows: MythBusters. In a world where educational television has degraded to the point that it consists largely of extreme fishing, people buying and selling old junk, idiots looking for gold, challenging driving, and fake documentaries about mermaids and extinct sharks, MythBusters has stood almost alone in maintaining a high educational standard while still being immensely enjoyable. If you look beyond the explosions and comical personalities displayed on MythBusters, there actually are some extremely good core lessons. Therefore, as the show draws to a close, I want to celebrate it by talking about all of the things that it got right…as well as some things that it didn’t get right.

To be clear, I’m not going to nit pick specific episodes, nor am I going to argue that the show’s value lies in the specific myths that they tested. Rather, I am going to talk about the overarching lessons and themes from this extraordinary show.

Adam Savage internet minefield information

Via MythBusters Episode 187 “Bubble Pack Plunge”

Lesson 1: Question everything
As a skeptic, I think that the single most valuable lesson from MythBusters was simply that we should question everything. We should always demand evidence before believing that something is true, and this show illustrated that brilliantly. Throughout the show’s history, they debunked numerous viral videos, newspaper stories, internet rumors, wives’ tales, etc. Time and time again, things that people believed and thought were true utterly failed once they were tested.

The importance of this lesson cannot be overstated. Everyday on this blog, I deal with people who deny scientific results, and for the most part, they aren’t unintelligent people. Rather, they simply haven’t learned to demand good evidence. They believe things based on hearsay and what some random person wrote on the internet, rather than actually fact checking. MythBusters  did an extraordinary job of demonstrating why that is so foolhardy. This, more than anything else, is why I think that MythBusters did a tremendous public service. It showed the importance of fact checking and looking for good evidence, and it did so in a way that was enjoyable and accessible to everyone.

Lesson 2: Intuition is unreliable
Closely related to the first lesson is the fact that intuition and gut instincts are highly unreliable. In nearly every episode, the MythBusters made predictions about what would happen based on their gut feelings and past experiences, and they were very often wrong. Further, if you’re like me, you probably made predictions at the beginning of each episode as well, and I willing to bet that you were also wrong…a lot (I certainly was).

This is extremely important because people base decisions and views on gut instincts all the time. I constantly encounter people who, “just know in their gut” that pharmaceutical X is dangerous or miracle cure Y works. MythBusters demonstrated why that way of thinking is flawed, and they did so in a very visually engaging way that makes sense to most people.

Lesson 3: Being wrong is exciting/”Failure is always an option”
One of my favorite things about the MythBusters was watching their excitement when they were wrong (especially Adam’s). You could tell that many of their favorite moments were times when they were convinced that X would happen, but Y happened instead. That type of excitement about being wrong exists in real science as well. Sure, there are plenty of times when being wrong is disappointing, but it is very often the case the being wrong is far more exciting that being right. The solutions that nature arrived at are generally far more interesting than the solutions that we’ve come up with (at least in my opinion), and as a scientist, finding something that you didn’t expect is exhilarating. It means that there is more to learn and understand about the system that you are working with. Science is often a process of finding things that don’t work, and you frequently learn far more from being wrong than you do from being right.

Perhaps of more practical importance for most people, you should never be afraid to be wrong. Imagine how boring and annoying the show would have been if the MythBusters always insisted that their original predictions were right, even when their tests said otherwise. Nevertheless, many people go through life that way. They “know” that they are right, and nothing will ever convince them otherwise. That’s a really sad and boring way to view the world. You should always embrace the possibility that you might be wrong rather than running from it.

The best embodiment of the MythBusters’ willingness to be wrong is probably all of the episode revisits. Fans would write in and critique their tests, and they would listen. Rather than stubbornly saying, “no, you’re wrong, we know we’re right” they actually took the comments seriously and tried the fans’ suggestions (and sometimes the fans were right). That’s how everyone should take criticism. It’s not always an easy thing to do, but we should always be willing to accept the possibility that we are wrong, and we should consider contrary evidence when presented with it.

Lesson 4: Other people know more than you do
There is a pervasive and unfortunate tendency for people to downplay the importance of experts. People would rather trust an unqualified blogger than a licensed doctor, experienced scientist, etc. Further, many people go as far as accusing real experts of being arrogant or elitist for having the audacity to think that their years of training and experience have gifted them with more knowledge than could be acquired through a few hours with Google. The MythBusters, however, were more than happy to consult with experts. They constantly got input from qualified individuals and they incorporated that knowledge and experience into their tests. You never saw them saying, “well the experts said to do X, but we read Y on the internet, so the experts must be wrong.”

This is how real science works as well. Look at most scientific publications and you will see a whole string of authors. Topics like science and medicine are extremely complex and most researchers can only claim expertise on a very narrow sub-discipline. As a result, we work with other scientists constantly. We forge collaborations with people who know more than us about a particular area, rather than plowing forward with our ignorance.

To be clear, I’m not suggesting that you blindly accept something just because an expert said it (that would be an appeal to authority fallacy), but you should recognize and acknowledge that experts do, on average, know a lot more about their area of study than someone who has never worked in that field. As a result, you should approach topics on which you have no training or experience with a great deal of humility, and you should be extremely cautious about concluding that you have found something which hundreds of experts have missed.

This quote was obviously made in jest, but it is nevertheless true that science involves a tremendous amount of note taking.

This quote was obviously made in jest, but it is nevertheless true that science involves a tremendous amount of note taking.

Lesson 5: Basics of the scientific method
Some episodes were more scientific than others, and there were plenty of times where I don’t think that they used proper controls, but overall, I think that the show did a very good job of introducing people to the basic concepts of the scientific method. To be clear, there is no one almighty scientific method that everyone religiously follows, but there are some overarching concepts which are nearly always applied.

First, science always goes from data to a conclusion, rather than starting with a conclusion, then trying to make the data fit, and MythBusters illustrated this nicely. They started with the rationale for the myth, then they tested the myth, then they drew a conclusion. Again, imagine how annoying the show would have been if they started with a conclusion, then tried to manipulate the test to make sure that the outcome fit their conclusion. Nevertheless, many pseudoscientific disciplines do exactly that, and you should be wary of them.

Second, MythBusters always at least tried to have a good control group. Controls are vital if you want to assign causation. If you want to know whether or not X causes Y, you need to know how often Y happens without X happening. Thus, scientific tests that are designed to infer causation involve an experimental group that receives the treatment of interest and a control group which is handled the same, but doesn’t get the actual treatment.  MythBusters generally illustrated this well. Not only did they have a control group, but they usually took time to explain it and make relevant comparisons to it. It would often have been very easy to gloss over the control instead of properly explaining it, so I really appreciate the fact that they highlighted it.

To be clear, some of their controls were better than others. For example, on several occasions they would do something like having Jamie drive an experimental car, while Adam drove a control car. That is a big problem because driver becomes a confounding factor. In other words, the control group and treatment group need to be identical in every way except the treatment, but when you have two different drivers, then differences between the groups may have been from the drivers, not the treatment being tested. Overall though, I think that they illustrated the concept of a control nicely, and, let’s be honest, professional scientists don’t always get controls right either.

Finally, the MythBusters constantly made testable predictions. Prior to most experiments, one of them would make a statement to the effect of, “If the myth is true, then we should see X” or “If we see Y, then this one is busted.” Then, after the test, they would draw conclusions based on whether or not the prediction came true. This is, once again, very much the way that real science works. In fact, testable, falsifiable predictions are central to modern science. So, I think that they did a very good job of illustrating that concept for the public.

Addendum (2-3-16): Although I hinted at this in the original post, I should have directly stated it. In science, you always should be asking open ended questions rather than setting out to demonstrate something, and that’s what the MythBusters did as well. They did not set out with the goal of proving a myth right or wrong. Rather, they tested each myth. They gave each myth the best possible chance of succeeding in order to find out whether or not it actually was true. Similarly, scientists don’t set out to prove that drug X works or chemical Y is dangerous. Rather, we test them to find out whether or not they work, are dangerous, etc.

Lesson 6: Critical thinking
Critical thinking skills are generally refined and honed through practice, and MythBusters provided a venue for fostering the development of those skills. Most of the people I know who watch MythBusters don’t do so passively. Rather, they actively scrutinize every minutia of how the MythBusters designed their tests, and they’ll often debate with their friends (or random people on the internet) about whether or not the MythBusters got it right (the extremely active fan site is great evidence of this). This type of critical thinking and analysis is fantastic and is, in fact, a huge part of science.

Scientists generally don’t read papers passively. Rather, we pick them apart and critically analyze the experimental designs and results. Indeed, during my graduate training, I have heard several professors give lectures on analyzing published papers and critiquing other researchers’ results. So it makes me extremely happy that MythBusters succeeded at getting fans to engage with the material and think about things like whether or not the control group was appropriate. I think it is an enormous testament to the educational value of the show.

Lesson 7: Start with small experiments
Most episodes began with “small scale” experiments where the MythBusters dissected and tested each individual component of a myth. Then, at the end, they put all of the pieces together for a full scale test, and science often works in very much the same way. In medical research, for example, we often start with “small scale” experiments like animal trials. Similarly, we use in vitro studies to look at individual components of biochemical pathways. Then, if those preliminary trials yield promising results, we put all of the pieces together in a “full scale” test such as a randomized controlled trial.

Importantly, in both science and myth-busting, the small scale tests and the large scale tests often don’t agree, and when that happens, you default to the full scale tests. It was often the case on MythBusters that individual components would work on the small tests, but once they scaled it up, there were other complexities or interactions that weren’t evident in the small scale, which ultimately resulted in the full scale experiment falsifying the myth. The same is true for scientific tests. For example, in vitro studies are great for looking at how particular cells respond to specific chemicals, but the human body is far more complex than a few cells in a petri dish. Thus, you often have drugs that are very promising in the small scale in vitro tests but fail during the full scale randomized controlled trials. Similarly, just as the MythBusters’ small models were imperfect representations of the full systems, animals are imperfect models of humans, and drugs often act differently in animals than in humans. When that happens, however, you should generally default to the full scale tests, not the animal models or in vitro studies.

Lesson 8: Lessons in physics
This one isn’t really a single lesson, but rather a whole set of lessons about physics (and to a lesser extent chemistry and biology). Multiple episodes were devoted entirely to physics, and these were often my favorites. As an undergraduate, I had to study physics, so I know the math for things like, “if you fire a bullet and simultaneously drop one from the same height, they will hit the ground at the same time,” but actually seeing those classic physics examples demonstrated and visualized was truly delightful. Further, they provided wonderful, memorable illustrations for people who haven’t studied physics and aren’t familiar with the math.

Additionally, even when the MythBusters weren’t directly testing classic physics examples, the episodes tended to be packed with real science. Yes, there were a lot of explosions and silliness, but there were also outstanding explanations and demonstrations of stoichiometry, pressure waves, fulcrums, masses, forces, etc. They explained the science behind everything that they did, and that was truly wonderful to see. Plus, I think that people tend to remember scientific concepts much better when they are used to blow something up, rather than simply being described in text books.

What they got wrong: Sample sizes and statistics
Finally, I want to talk about the one major thing that the MythBusters got wrong: namely, the small sample sizes and lack of statistics. Most tests were only replicated about three times, which simply isn’t enough to give you a reliable answer. Further, in almost every case, they simply looked at the averages and went with whichever one was larger, even if the difference was very small. This is extremely problematic because that difference may simply be a chance result, and even if you have a large sample size, you need to do statistics in order to determine how likely you are to get that difference just by chance.

Many of the fuel mileage myths illustrate this well. They would drive the car under one condition three times, then drive it under a different condition three times, then compare the average fuel consumption and draw a conclusion. Often, there would be a very slight difference between those results, yet they would still call the myth based on those tiny sample sizes and lack of statistics. That’s a big problem because there is probably a lot of variation from slight differences in how fast they drove, how straight they kept the wheel, how steady they were on the gas pedal, wind gusts, etc. All of those factors create statistical noise which make it extremely difficult to distinguish between an actual difference and a false difference that arose because of chance variation. This is why scientists use statistics and large sample sizes. As your sample size increases, you have more power to cut through the statistical noise and detect true differences.

Although this is a real problem, it is ultimately something that I can forgive the MythBusters for, because I understand that it was necessary for the show. I get that there has to be a balance between education and entertainment, and doing each test 30 times would be really boring. Further, if they tried to go into details of the statistics, I find it extremely likely that many people would start looking for something else to watch. So, as much as I would have loved to have seen them use proper statistics, I realize that doing so would probably have shortened the longevity of the show (though they did do a brilliant job of illustrating the Monty Hall paradox, so maybe they could have succeeded at making math fun for most people).

Note: This criticism only applies to myths where they were comparing two things. Many of the myths were simply, “can you use X to do Y” (e.g., can you use one gram of sodium to blow a man-sized hole in a brick wall?). That type of question differs greatly from the type of questions that most professional scientists today tackle, and for that type of question, simply using X to do Y once is enough to say that it can be done. Conversely, in many cases it is possible to pretty conclusively show that something cannot happen, even without a large sample size. 

Conclusion
When it’s all said and done, I think that MythBusters did an extraordinary job of making science exciting and teaching scientific concepts to the general public. It had its faults and some tests were better than others, but I still contend that it taught many valuable lessons and truly lived up to the title, “educational television.” So, to Jamie, Adam, Kari, Tory, Grant, and everyone else involved with this tremendous show, thank you for all of the science, laughs, and explosions. You will be missed.

Posted in Nature of Science | Tagged | 11 Comments

6 reasons why anti-vaccers are wrong that “being sick is good”

Is being sick a good thing? The answer to that question should be an obvious and resounding, “no!” Nevertheless, it is an extremely common claim made by anti-vaccers. They frequently argue that vaccine preventable diseases like measles are actually good for you, and it is a good thing to get sick (for example, in this hilarious video, you can see Tenpenny claiming that it was good that she was so sick as a child that she missed the entire 3rd grade). Therefore, I am going to use simple examples, rudimentary logic, and basic science to explain why this claim is utterly absurd.

Before I begin, I want to clarify that this post is not about vaccines. Although the claim that “being sick is good” is used to argue that we should not use vaccines, the claim itself must be either true or false on its own merits. In other words, the safety and effectiveness of vaccines has no bearing on whether or not it is true that being sick is actually good for you. So although the argument has implications for vaccines, the argument itself is not about vaccines.

1). A simple thought experiment
Let me begin this post with a simple thought experiment that (I think) will clearly demonstrate that the people making this claim are disingenuous. Let’s imagine that we were in the pre-vaccine era when measles was an extremely common and prevalent disease. Now, let’s imagine that someone discovered an herb that was free, 100% natural, 100% safe, and 100% effective at preventing measles (for the sake of example, assume that everyone agreed on all of those points). Would you use the herb or would you allow your children to suffer and potentially die? I find it extremely difficult to accept that parents would actually watch their children suffer through a miserable disease that kills 1 out of every 1000 patients rather than using a preventative measure that they know is 100% safe and 100% effective. I can, of course, make many hypothetical examples like this. For example, if there was a medicine that was 100% safe and 100% effective at preventing colds, I would absolutely take it (and I’m betting you would too).

Now, you may be thinking, “but vaccines aren’t 100% safe or 100% effective.” In which case, you are missing the point. Once again, this is not an argument about vaccines. Rather, it is an argument about whether or not it is good to be sick, and if in the hypothetical situations that I have proposed, you would use the herb to avoid being sick, then you have just affirmed that you do in fact think that being sick is a bad thing. That is what the rules of consistent reasoning dictate.

Note: you can find additional information about why the “vaccines aren’t 100% safe/effective” argument is flawed here.

2). People die from illnesses
Next, I want to bring up what is probably the most obvious problem with the claim that being sick is a good thing. Namely, the fact that many people die from being sick. The diseases that anti-vaccers tout as being “good” are actually deadly. For example, the WHO estimates that measles killed 114,900 in 2014 and 145,700 in 2013. Similarly, every year the flu kills anywhere from 250,000-500,000 people, with over 1,000 deaths in the US (Thompson et al. 2010). Further, in 2008 alone, the WHO estimated that for children under 5 years old, haemophilus influenza type B killed 199,000, pertussis killed 195,000, measles killed 118,000, neonatal tetanus killed 59,000, non-neonatal tetanus killed 2,000, pneumococcal disease killed 476,000, and rotvirus killed 453,000. Indeed, even illnesses that are usually mild can be fatal. For example, prior to vaccines, chicken pox hospitalized 10,500 people annually and killed 100-150. So, being sick is clearly not a good thing for all of the people who die from being sick.

3). It’s hypocritical to claim that being sick is good
My thought experiment (#1) has already established that it is inconsistent to say that being sick is a good thing, but let’s examine the hypocrisy further. I have never once encountered someone who actually lives as if being sick is a good thing (despite their claims to the contrary). Take hand washing, for example. Why do you do it if being sick is a good thing? Similarly, anti-vaccers love to rant about how wonderful modern sanitation is and how it is supposedly the reason for the decline in disease rates (it’s not), but if being sick is good, then sanitation must be bad. In other words, if it is good to be sick, then something that does nothing other than preventing you from being sick must be bad. Am I making my point clear? If someone actually thought that being sick was good, then they would never wash their hands, they would encourage sick people to cough and sneeze in their face, they would make their kids play with feces, etc. They don’t do that, however, because everyone knows that being sick is not a good thing, even if they make claims to the contrary.

4). Getting sick is a terrible way to avoid getting sick
When asked why they think that being sick is good for you, anti-vaxxers typically respond with something to the effect of, “Being sick builds the immune system, and once you get a disease, you’re protected from it for life.” The “logic” of this claim is so outrageously horrible that it makes my head hurt. In its simplest form we can set the argument up using the following syllogism:

  1. Getting a disease will prevent you from getting it again
  2. Therefore, getting a disease is a good thing

This syllogism is obviously problematic for numerous reasons. Perhaps most importantly, it’s missing a premise. You see, being protected from a disease is only a good thing if getting the disease is a bad thing. In other words, the argument has to be structured like this:

  1. Getting a disease will prevent you from getting it again
  2. Getting a disease is bad
  3. Therefore, getting a disease is good

Is the problem with this argument clear now? Lifetime protection from a disease is only a good thing if getting the disease is a bad thing, but if getting the disease is a bad thing, then getting the disease cannot simultaneously be a good thing. It is utterly idiotic to think that it is good to get a disease so that you won’t get the disease. It’s no different from saying, “My friends want me to go snowboarding, but I’m afraid that if I do I will break my leg. Therefore, I am going to break my leg before the trip, that way I have an excuse for not going snowboarding and will be protected from breaking my leg a second time.”

5). Getting sick is a bad way to build the immune system
At this point, you may be thinking, “Fine, getting sick is a bad way to protect yourself from a specific disease, but doesn’t getting sick build your overall immune system?” The answer to that question is a bit complex and multifaceted, but the short answer is, “not really.”

First, there is little evidence (at least to my knowledge) of childhood infections actually strengthening your immune system. In other words, if, as an adult, you get exposed to something like the flu virus, the way that your immune system reacts will not be dependent on whether or not you previous had childhood diseases like measles. Granted, it is true that for many diseases your body will become immune to them after recovering from an infection (if you recover), but that does not impact your body’s ability to fight other infections. Each pathogen contains specific antigens (surface recognition molecules) which your body uses to distinguish friend from foe, and you become immune by producing antibodies and immune cells that are specific for particular antigens. Thus, when you get an infection, your body creates immune cells that are specific for that infection. So, being sick only “builds your immune system” in that it prevents you from getting the same strain of a given disease twice, and we have already established that getting sick to avoid getting sick is idiotic (see #4).

The next important topic is the hygiene hypothesis. In its simplest terms, this states that childhood infections train our immune system, and a lack of early infections is the cause for increases in the rates of autoimmune problems like allergies and asthma. There are several things to note about this. First, although this is a plausible hypothesis, we aren’t really sure if it is correct. There is a lot of support for it, but the immune system is amazingly complex and there is still a lot that we don’t know. So although the hygiene hypothesis seems very likely, it’s not the only possibility, and the true answer is probably the combination of several hypotheses (Rook 2011).

Second, there is growing evidence that it is not the actual infections that are responsible for training our immune systems; rather, it is beneficial helminths and microorganisms (Gaurner et al. 2006; this is a modification of the hygiene hypothesis known as the “old friends hypothesis”). Our bodies are hosts to untold legions of beneficial microorganisms, and our current sanitation standards are likely preventing us from coming into contact with many of the species that we coexisted with historically. Thus, it is likely that rising rates of allergies, asthma, etc. are from a deficiency of beneficial bacteria, rather than a lack of childhood infections.

Third, under the hygiene/old friends hypothesis, microorganisms are not helping to “build” the immune system as much as “control” the immune system. The autoimmune disorders that they prevent are usually situations where the body over-reacts and basically attacks itself. During an allergy attack like hay-fever, for example, your body over-reacts to harmless hay antigens and mounts an unnecessarily strong response. It is the histamines and other chemicals that your body releases that make you feel like crap. So, rather than building a more robust immune system, microorganisms actually teach your immune system to tone things down and not over-react (again, I’m describing things in absurdly simplistic terms, but I’m afraid that I will lose people if I start talking about cytokines, Th1 cells, Th2 cells, etc.).

Finally, even if a lack of childhood illness, like measles infections, was responsible for the increase in autoimmune diseases, that would still not make being sick a good thing, because the argument would still be, “it’s good to be sick, because it prevents you from being sick.” Granted, this time you are getting one disease to avoid getting a different disease, but those first diseases are often horrible (see #2), and we are much better off with the later category. If you don’t believe me, just look at the data for life expectancy, infant moralities, etc. As diseases have been eliminated, child mortality rates have plummeted and life expectancies have steadily climbed. So I, for one, do not long for the good old days where no one had allergies, but 1 out of every 10 infants died (and I suffer from really bad allergies, btw).

6). Measles infections weaken your immune system
Finally, I want to talk about a recent study which found that getting a measles infection actually harms your immune system. This study (Mina et al. 2015) looked at the long term effects of measles infections, and it found that measles is so devastating to your immune system that it takes two to three years for your immune system to return to normal functional levels. In other words, for up to three years after a measles infection, your body is at a greater risk of additional infections. This is extremely clear evidence that being sick is most definitely not a good thing, because measles infections actually weaken your immune system.

Conclusion
In summary, arguing that being sick is a good thing is hypocritical because everyone actively attempts to avoid being sick, and any rational person would use a completely safe and effective preventative measure. Further, being sick only builds the immune system in that it prevents you from getting a particular strain of a particular disease a second time, and getting sick to avoid getting sick makes no sense whatsoever. Finally, some diseases actually result in a decreased immune response for months or even years after an infection. Therefore, being sick is clearly a bad thing and should be avoided (duh).

Posted in Vaccines/Alternative Medicine | Tagged | 3 Comments

Evolutionary mechanisms part 5: Sexual selection

As Charles Darwin sailed on his epic voyage, he noticed something which initially troubled him. In many species, the males had traits which seemed disadvantageous. In birds, for example, the females tended to be dull and camouflaged, whereas the males were often bright and garish. He was particularly impressed with extreme examples, such as the peacock. How, he wondered, is it possible that nature would select for peacocks to have such clearly disadvantageous traits like absurd colors and impossibly long tails? Fortunately, Darwin was a smart man, and he soon uncovered the answer: an evolutionary mechanism known as sexual selection.

I honestly found this post very hard to write, because I find sexual selection to be utterly fascinating and mesmerizing. There are so many cool facets to it and so many amazing examples that I wanted to share with you that writing a single, condensed post seemed nearly impossible. As a result, I have been forced to leave out a ton of great examples, and this post won’t be much more than a Cliff Notes introduction. I would encourage you, however, to study it more on your own. Also, if you want to see a bunch of neat examples, I cannot recommend David Attenborough’s documentaries strongly enough. They are fantastic.

What is sexual selection?
Sexual selection is really best understood as a type of natural selection because it operates off of the same three requirements:
1). The trait is heritable
2). The trait is variable
3). The variation affects individuals’ ability to pass genetic material on to the next generation

Sexual selection also has an additional requirement, however. It deals specifically with the traits that are directly involved with obtaining a mate and forming a zygote (egg+sperm). Exactly what traits fall under the realm of sexual selection is somewhat of a grey area, but it is generally applied predominantly to sexual dimorphisms (i.e., anatomical differences between males and females), especially secondary sexual dimorphisms (i.e., dimorphisms that are involved in obtaining a mate, but not in the actual act of mating). This covers traits such as ornaments and colors that help many males to attract females, as well as features that help in conflict within the sexes (e.g. sexual selection is generally given credit for driving the evolution of antlers in the males of many deer species, because the antlers are used by the males to fight over females).

Sexual selection also differs from classical natural selection in one other important way. In classical natural selection, there isn’t actually an agent doing the selecting. Rather, it is simply a numbers game where the individuals who pass on the most genes are “selected” by simple virtue of the fact that they passed more genetic material into the next generation than their rivals did. Remember, evolution is simply a change in the allele frequencies of a population over time. So, getting a disproportionate number of your alleles into the next generation causes the allele frequencies to shift in your favour. In sexual selection, however, there often is an agent who is actually selecting traits. In many species, one sex (usually females) directly chooses who to mate with, which means that females are actually selecting which traits will become predominant in the population. This is fascinating because, as I will explain, it gives females the opportunity to guide selection, and it often results in them being real jerks (if you’ll forgive me for anthropomorphising).

Note: for most of this post, I am going to act as if females are the ones doing the selecting, but there are a few exceptions which I will discuss at the end.

Females often select for seemingly arbitrary traits
Sometimes, the traits that females select make good sense. For example, females of many insect species demand a “nuptial gift” from their suitors. This is generally something edible, such as another insect, salt crystal, or even sperm (there is some debate about how beneficial these actually are for females; see Gwynne. 2008 for a review). For example, if you have ever seen a group of butterflies congregating around a drying mud puddle, many of those are actually males who are collecting salt crystals to present to potential mates.

Similarly, some birds select for sensible traits like nest construction. In other words, females are choosing their mates based on who makes the best nest. This makes good sense since a well- constructed nest increases the chance of her offspring surviving. In many other species, it is a male’s ability to defend a territory that interests females, and the females choose to mate with males who have large territories with adequate resources for rearing her young.

This bower belongs to a great bowerbird (Chlamydera nuchalis) that lives on my university's campus. You'll notice that in addition to constructing the bower itself (the columns of sticks) he has amassed a larger collecting of green, white, and grey objects. Some of his treasures are natural like rocks, dead coral, and shells, but he also has a lot of human trash like old nails, broken glass, and plastic.

This bower belongs to a great bowerbird (Chlamydera nuchalis) that lives on my university’s campus. You’ll notice that in addition to constructing the bower itself (the columns of sticks) he has amassed a large collection of green, white, and grey objects. Some of his treasures are natural like rocks, dead coral, and shells, but he also has a lot of human trash like old nails, broken glass, and plastic.

For many species, however, the traits that females select initially appear arbitrary. Take color and call, for example. Why do females of so many species prefer their males to be brightly colored, and why do they care if the males make loud, long, complex calls? Other species go far beyond simply wanting calls and colors and demand complex dance routines or even decorations. The bowerbirds are one of my favorite examples of this (Attenborough has some great documentaries on these guys and you can see a clip from one here). In these species, the males build elaborate and often enormous structures called bowers, and they decorate them with all manner of trinkets. Exactly what gets used depends on both the species and the individual. Some species/individuals have a fascination with particular colors and may collect predominantly one color. Others are more eclectic and gather a diversity of objects. Regardless of the specifics, however, they all tend to be a bit OCD, and the males devote themselves to meticulously maintaining their collection. The females then fly around to different bowers and assess the males’ collections as well as their construction abilities, and they use those factors to decide who to mate with. Importantly, after mating, females do not stay at the bower. Rather, they fly off to make a nest and rear the young, while the males waits for more females. In other words, the bower is not a nest or a foraging area, and the females get nothing out of it except sperm. So, once again, why? Why do females care how large the males’ collection is?

Partial answers for these questions do, of course, exist, and, as I will explain, the selection for traits is not as arbitrary as it may initially seem.

The handicap principle and runaway selection
Before I explain why females choose the traits that they do, I need to introduce two other pieces of the puzzle. First, let’s talk about the handicap principle. In short, this states that females generally choose traits that screw the males over. In other words, they are selecting traits that are disadvantageous for males. Think, once again, about colors and calls in birds. It is clearly not in the males’ best interest (as far as survivorship) to be brightly colored or to loudly announce your position to predators, yet the females want the loud, brightly colored males. Similarly, in species like bowerbirds, the males invest an incredible amount of energy and resources into building and maintaining a bower, and they could be spending that time and energy finding food.

The second principle is runaway selection. This states that there is essentially no limit to the females’ obsessive desire for improved traits. In other words, if a bright male is good, a brighter male is even better. The classic experiment on this was conducted on long-tailed widowbirds (Euplectes progne; Andersson 1982). Male widowbirds live in the African savannas and display for females by jumping and flying above the tall grass to display their absurdly long tails. Scientists suspected that tail length was important in mate choice. So, Andersson selected a group of males with similar tails and similar mating success. Then, he cut the tail feathers off of several males and glued them on to the tails of some of the other males. Thus, he had shortened tails, normal tails, and tails that were roughly twice their normal length. What he found, was that females weren’t thrilled about the short tails, would still mate with the normal tails, and were really turned on by the long tails (i.e., the birds with the extra-long tails got most of the mates).

Here, we have a situation with both the handicap principle and runaway selection. Having long tails is not good for the males because it makes it harder for them to fly, and it makes it easier for predators to pick them off. Nevertheless, despite (or perhaps because of) the disadvantages to the males, there doesn’t seem to be a limit on the females’ desire for long tails (the longer the better).

Why do females choose traits that are bad for males?
At this point, we finally have enough pieces of the puzzle to start to put it together. There are several different hypotheses about why females choose the way that they do, but I will just discuss the two predominant ones. First up, we have the “sexy sons hypothesis” (yes, that is its actual name). This basically states that females select for traits that will maximize the reproductive potential of their offspring. In other words, if a female mates with a very attractive male, then her offspring will also be very attractive, which will allow them to get lots of mates (remember, if you don’t continue to get genes into the next generation, then you are evolutionarily dead).

At a quick glance, the sexy sons hypothesis makes good sense, but I am personally not a big fan because I don’t think that it actually answers anything. Consider the following.

1). A female is attracted to red
2). Therefore, she mates with a red male so that her offspring will also be red
3). Other females are also attracted to red
4). Therefore, her offspring will enjoy lots of mates

Do you see the problem? Why were females attracted to red in the first place? The sexy sons hypothesis basically says, “females are attracted to X because other females are attracted to X.” It’s circular, and I don’t like circular logic. To be clear, I’m not saying that the sexy sons hypothesis is worthless. I think that it may help to explain runaway selection, but I don’t think that it is an adequate ultimate explanation for why females choose the traits that they do.

A more compelling explanation (in my opinion) can be found in the “good genes hypothesis.” This proposes that females are using traits like colors and calls to judge whether or not the males have good genes. For example, a male that is able to produce brightly colored feathers and call loudly while still being able to avoid predators and forage successfully probably has good genes, which means that offspring from that male will also have good genes. This may sound superficially like the sexy sons hypothesis, but there is an important difference. In the sexy sons hypothesis, females are selecting a trait that they are attracted to because their sons will also get that attractive trait; whereas, in the good genes hypothesis, the females are simply using that trait as a way to assess whether or not a given male will produce high quality offspring.

The good genes hypothesis provides satisfying explanations for both the handicap principle and runaway selection. It proposes that females choose traits that are disadvantageous for males because that is the best way to actually judge the male. In other words, if females selected a trait that didn’t affect males one way or the other, that wouldn’t let them judge the quality of the males’ genes. In contrast, selecting a trait that is harmful for the males lets them judge the males, because only a male with good genes would be able to have the harmful trait and still survive. Further, it makes sense that a very exaggerated trait will be a better judge of a male’s quality than a minor trait. Thus, judging males on exaggerated traits could drive runaway selection (think back to the long-tailed widowbird experiment: the longer your tail, the harder it is to survive/avoid predators, but the more the females like you). All of this is, of course, predicated on the notion of “honest signalling.” In other words, this only works if the traits that females are selecting do actually reflect the males’ quality.

At this point, you may notice that there is still a problem. I have offered an explanation for why females choose disadvantageous traits, but I have not explained why they choose the particular disadvantageous traits that they do (e.g., why do female Northern Cardinals like red instead of a different color, like blue?). In short, we don’t really know, and there are probably lots of different factors that affect different species. In some cases, it may even be simply a random result of genetic drift, but other possibilities exist.

One really interesting possibility is what’s called a pre-existing sensory bias. For example, let’s say that a certain species eats mostly red berries. This could give them a pre-existing sensory bias for the color red (i.e. they like the color red). Then, after many generations of eating red berries, a male hatches with a mutation that causes him to have red feathers. Because of the pre-existing bias, the females will find that mutation very attractive, and he will get lots of mates. As a result, the mutation will become more common in the next generation and his offspring will benefit greatly from it. Thus, in each generation the mutation becomes more and more common, until eventually all of the males are red.

Sexual conflict
It should by now be clear that there is a conflict between sexual selection and natural selection which ultimately results in a battle of the sexes. On the one hand, female choice and sexual selection are driving males towards increasingly elaborate and disadvantageous traits. Meanwhile, classical natural selection is driving males away from those elaborate traits because they result in dead males. This results in a balance or equilibrium state between the two forces.

Let’s use a bird with a long tail as an example, and let’s say that the normal equilibrium tail length is 10cm, and individuals with 10cm tails have 5 offspring on average. Females would choose longer tails if they were available, but individuals with longer tails have such high predation rates that they don’t live long enough to mate often. As a result, individuals with tails >10cm actually only have four or fewer offspring before dying. Conversely, males will live longer if they have tails that are <10cm, but they won’t get as many mates. So, once again, they will have four or fewer offspring. This is an equilibrium state because longer tails get selected against because of low survivorship, and shorter tails get selected against because of low interest from the females.

There are many different factors that influence the equilibrium point, and it can change with the environment. For example, if a new predator gets introduced, that may cause the point to shift towards slightly shorter tails because survivorship becomes more important than female choice. Nevertheless, it is obvious that in many cases, the equilibrium point was reached in favor of the females, because as far as male survivorship is concerned, it would be best for the males to be just as plain and boring as the females. The reason that the balance is usually shifted towards female choice is the simple fact the if you don’t mate, you don’t get selected. Remember, selection is all about passing on your genetic material, and surviving is only important in that it gives you more time to reproduce. If you live forever, but never have any offspring, then you’re evolutionarily dead.

Why females are the choosy sex
The next important issue to address is why females are usually the ones who choose. The typical answer is, “Eggs are expensive, and sperm are cheap.” In other words, females invest more heavily in the offspring; therefore, they are the ones who choose the mate. To put this another way, females are physiologically limited in the number of offspring that they can produce, whereas males are only limited by the number of mates that they can obtain. If you think about humans for a second, females can have, at most, about one offspring a year (excluding twins, triplets, etc.). In contrast, males could, in theory, produce several hundred children a year because sperm is cheap and easy to produce. This means that females have a much greater investment in each offspring.

Think for a minute about an extremely polygamous, randomly mating bird (i.e., one where males and females both mate with many partners and females mate randomly). If a clutch of eggs is lost, the loss to the female is enormous because of the time, resources, etc. that went into producing those eggs. In contrast, the loss to the male is much less because all that he invested was sperm, and he has plenty more sperm to knock up other females with. By carefully selecting her mates, however, the female can maximize the chance that her offspring survive. Thus, it is in her best interest to make sure that her offspring get the best genes available. In contrast, it is in the male’s best interest to mate with as many females as possible.

Sex role reversal
Throughout this post, I have been acting as if it is always the female that chooses, but that’s not actually correct. There are several species of insect, bird, amphibian, mammal, and fish in which the male chooses (there are also probably some in other taxonomic groups). Based on what I just explained about female vs. male investment, it should not surprise you to learn that in these species, males do most or all of the parental care. In other words, they are the ones with the biggest investment in the offspring, which means that they are the ones who choose. Phalarope species (a variety of shorebird much like a plover) are a common example of this (Delehanty et al. 1998). In these species, the female is brighter than the males (though still fairly dull) and the female displays for the males. After mating and laying her eggs; however, she goes off to find another mate while the males take care of the eggs/young.

A comb-crested jacana (Irediparra gallinacea) near where I am currently living.

A comb-crested jacana (Irediparra gallinacea) near where I am currently living.

Jacanas are another good example (Haf et al. 2003). These super cool birds live on top of the lilies in tropical rivers and swamps, and the females (who are much larger than the males) control a territory with a harem of males. A female will mate with each of her males, but she doesn’t do any of the parental care. Rather, she leaves that to the males. Other than size, males and females look very much alike, so it is likely that they are selecting based on the territory that the females hold.

Summary
In short, sexual selection is simply a type of natural selection that acts on the traits responsible for obtaining a mate. Females are generally the sex that chooses because females have a greater investment in the offspring than the males. Also, females often choose exaggerated traits that are disadvantageous for the males because they use those traits to judge whether or not the male will produce high quality offspring. This results in a conflict where sexual selection is driving the evolution of elaborate features, while classical natural selection is driving the evolution of traits that maximize survivorship. Finally, although females are generally the choosy sex, there are exceptions, and in these exceptions, it is usually the males that do the selecting.

This post has only scratched the surface and I described most things using very broad brush strokes. So, if you found this interesting, I would encourage you to do some more reading (or at least watch Attenborough) because there are tons of great topics that I didn’t get a chance to talk about (sperm competition, dishonest signals, sneaker males, the effects of mating systems on sexual selection, sexual selection in humans, etc.).

Other posts on evolutionary mechanisms

References
Andersson. 1982. Female choice selects for extreme tail length in a widowbird. Nature 299:818–820.

Delehanty et al. 1998. Sex-role reversal and the absence of extra-pair fertilization in Wilson’s phalaropes. Animal Behavior 55:995–1002.

Gwynne. 2008. Sexual Conflict over Nuptial Gifts in Insects. Annual Review of Entomology 53: 83–101.

Haf et al. 2003. Parentage and relatedness in polyandrous comb-crested jacanas using ISSRs. Journal of Heredity 94:302–309.

Posted in Science of Evolution | Tagged , | 8 Comments

5 reasons why anecdotes are totally worthless

anecdotal evidence anti-sciencePersonal anecdotes are often the primary ammunition of those who deny science. If you ask anyone in the alternative medicine or anti-vaccine movements for their evidence, you will almost certainly get flooded with anecdotes. A quick internet search will reveal countless people who are insisting that totally worthless treatments like homeopathy work because they took them and then felt better. These accounts are often accompanied by emotional stories about how they “tried everything but only [insert nonsense miracle cure] worked.”  Similarly, I frequently encounter people who are adamant that detox solutions aren’t scams or that organic food is better than GMOs because “they just feel healthier when they eat organic/use the detox supplement.”

Anti-vaccers are probably the worst group for using anecdotes. They use personal anecdotes to blame vaccines for every ailment imaginable, but they don’t just stop there. For them, collections of reported symptoms such as the vaccine package inserts, VAERS, and cases from the NVICP are the gold standards of evidence that vaccines are bad. Those sources are, however, really just collections anecdotes. Similarly, even when anti-vaccers attempt to use the scientific literature, they often end up accumulating case reports, which are essentially glorified anecdotes.

All of this would be fine if anecdotes were actually useful pieces of evidence, but they aren’t. As I will explain in this post, they are worthless, and if your argument is built on anecdotes, then your argument should be rejected.

Before I begin, I want to clarify what I mean when I say that anecdotes are worthless. They are worthless as evidence, and you cannot use them to establish causal relationships. You can’t, for example, say “Bob took X, then got better; therefore, X works.” You can, however, say “Bob took X, then got better; therefore, X might be an interesting topic for future research.” In other words, anecdotes can be useful in helping researchers decide what topics to study, what potential drugs to investigate, etc. However, in the absence of those large, carefully controlled studies, you cannot jump to the conclusion that a causal relationship exists. In other words, you can’t assume that X works until X has been properly tested, and, perhaps most importantly, if the tests disagree with the anecdotes, you must reject the anecdote, not the tests.

There are also a few other situations in which anecdotes can potentially be useful (e.g., if a patient is dying and a doctor has exhausted all science-based options, then and only then would it be appropriate to try a treatment which has only anecdotal evidence to support it). For the purpose of this post, however, I am just going to focus on why they are completely and totally invalid as evidence for causal relationships.

1). If you are using anecdotes, you are committing a logical fallacy
Anytime that someone uses an anecdote to argue that X causes Y, they are committing a logical fallacy known as post hoc ergo propter hoc (often abbreviated as simply post hoc). The Latin translates to “after this, therefore because of this,” and it occurs whenever an argument takes the following form:

  • X happened before Y
  • Therefore, X caused Y

The astute reader will quickly notice that the vast majority of personal anecdotes are identical to that syllogism. For example, if you say, “I took this supplement, then I felt better; therefore, the supplement works” you are committing a logical fallacy. Similarly, if you say, “I vaccinated my child, then he developed autism; therefore, vaccines cause autism” you are committing a logical fallacy. Also, if you say, “I switched to an organic diet, then I started feeling better; therefore, an organic diet is healthier” you are committing a logical fallacy. Am I making my point clear? Using personal anecdotes as evidence of causation is logically invalid, and the rules of logic tell us that any argument that contains a logical fallacy is unreliable and must be rejected.

The reason that post hoc arguments are invalid should be obvious: the fact that Y happened after X does not mean that X caused Y. Let’s say, for example, that you fill your vehicle with fuel from a reputable gas station, and your car breaks down just a few miles later. Can you conclude that the bad gas killed your car? No. It is certainly possible that bad gas was at fault, but it is also possible your car died from something totally unrelated to the gas, and getting gas was just a coincidence. Even so, the fact that you got better after taking X does not mean that X worked because there are many other factors that could have caused your recovery.

It is worth noting, that you can use the order of events to make a legitimate argument if you are making a probabilistic argument, and if a causal relationship has already been established. In other words, if you know based on actual evidence (not anecdotes) that X can cause Y, then if Y happens after X, it is not unreasonable to conclude that X probably caused Y. So, you can say,

  • Item X is known to cause Y
  • I took X, then Y happened
  • Therefore, X probably caused Y

There is nothing wrong with that if and only if there is actually valid, scientific evidence that X can in fact cause Y. Also, the strength of the argument will depend on the strength of the relationship between X and Y (e.g., if X causes Y in 99% of cases, then it is a very strong argument, but if X only causes Y in 0.0000001% of cases, then it’s not a good argument because X almost never causes Y).

2). Anecdotes aren’t representative
Another major problem with anecdotes is that they don’t give you a proper representation of either the effects of X or the causes of Y. Let’s say, for example, that you are interested in miracle cure X, and when you get online, you find several people claiming that it worked for them. That doesn’t actually tell you much because it doesn’t tell you how many people X didn’t work for, nor does it tell you how many people recovered without X.

To give another example, anti-vaccers love to cite anecdotes of a symptom that followed a vaccine, but for every anecdote that they supply, I can supply anecdotes of people (like me) who received the full recommended vaccine schedule and are perfectly fine. Neither set of anecdotes is actually meaningful, because neither set is representative. To actually know whether or not X caused Y, we need the actual rates of Y relative to X, not just scattered reports. In other words, we need to know how many times Y followed X, how many times Y occurred without X occurring, and how many times X occurred but was not followed by Y (in some situations you may only need one of the later two, but you have to have at least one).

3). Anecdotes aren’t controlled
The third major problem with anecdotal evidence is that fact that they don’t control all possible factors. In other words, you can’t say, “I took X, then got better; therefore X works” because there may be something other than X that caused you to get better. In many cases, people simply get better on their own. For example, I often see people take a “remedy” for the common cold, continue to be sick for a day (or often several days), then get better, but after recovering, they insist that the remedy worked. The problem is, of course, that people normally get over colds in a few days. Therefore, it is utterly impossible (based on that anecdote) to determine if the remedy worked, or if their body simply took care of itself. As I explained in #2, this is why it is so important to know the actual rates of event Y relative X.

The placebo effect is another huge confounder. The placebo effect is often misunderstood and misrepresented (you can find good explanations/discussions here and here), but it is true that in many situations, people will report feeling better if they think that they are taking something that will help them, even if the treatment is totally worthless. This is especially true with highly subjective measurements like pain. So in some cases, people may report feeling better even if the treatment itself didn’t actually do anything.

There are many other potential factors that people fail to account for. Alternative medicine, for example, is famous for recommending a whole slew of treatments, then picking one as the responsible party. For example, I often hear people say things like, “I know X works, because my naturopath told my to exercise more, eat more vegetables, and take X, and I feel great now.” It seems rather silly to give X the credit if you you also started exercising more and eating healthier (both of which are actually supported by scientific evidence). Another one that I often encounter is, “my naturopath told me to do A, B, C, and eat less gluten, and I feel much better now, so gluten must be bad for you.” Again, how do you know that it was gluten and not A, B, or C? Those two examples contain pretty obvious confounding factors, but confounding factors may be much more subtle, and you may not even be aware of them. So, even if, to your knowledge, X is the only thing that has changed, there may be some other change that you haven’t thought about or just aren’t aware of.

Finally, it’s worth noting that the fact that, in some situations, we cannot identify the actual cause of an event does not mean that you can assume that it was X. In other words, if you say, “X causes Y, because I took X and Y happened,” and someone calls you out for using an anecdote, you can’t respond with, “Well if it wasn’t X, then what was what? Unless you can prove that it was something else, it must have been X.” That argument is actually another logical fallacy. Specifically, it is an argument from ignorance fallacy. The fact that I don’t know what caused Y doesn’t mean that it was X, and it’s not logically valid for you to jump to that conclusion. To put this another way, by claiming that X causes Y, you are placing the burden of proof on you, and it is your job to provide actual evidence that X causes Y. It’s not my job to provide evidence that X doesn’t cause Y.

4). An anecdote is a sample size of N=1
The importance of sample size is one of the most fundamental concepts in statistics. The larger your sample size, the more power that you have and the more confident you can be in your results. An anecdote, however, is simply a single observation, and extrapolating from a single observation to a general trend is an absurd thing to do. Imaging, for example, that you want to know whether or not a coin is biased, so you flip it twice and it lands on heads both times. Should you conclude that the coin is biased? Of course not. A sample size that small is meaningless because it is entirely possible (even likely) that you got a biased result just by chance. The same thing is true with anecdotes. Saying, “I vaccinated my kid, then he developed autism; therefore, vaccines cause autism” isn’t substantially different (as far as sample size) from saying, “I flipped the coin twice and got heads both times; therefore, the coin is biased.” Tiny sample sizes simply aren’t reliable.

5). Anecdotes aren’t collected systematically
Following my argument in #4, you may be thinking, “but I have met lots of people on the internet with identical anecdotes, so my sample size is much larger than just one.” The problem with that argument is that the anecdotes were not collected in a systematic way. This is really an overarching problem which overlaps substantially with points 2, 3, and 4, but it is important enough that I want to talk about it separately.

One of the hallmarks of science is being systematic. Real research is done in a careful, planned, controlled, repeatable fashion, and that systematic approach is a big part of why science is such a powerful tool for understanding the universe. For example, when we want to answer a question like, “do vaccines cause autism?” we don’t just haphazardly find someone on the internet. Rather, we carefully select a representative study population, control for confounding factors, use large sample sizes, and measure the actual rates of autism in both children with and without vaccines (for example, Taylor et al. 2014). That approach and that approach alone allows us to overcome the problems described in #2-4 and actually achieve a reliable answer. Anecdotes, on the other hand, are in no way systematic, which makes them exceedingly unreliable and unscientific.

Conclusion
In summary, using anecdotes as evidence of causation commits a logical fallacy, which means that anecdotal arguments must be rejected. Further, anecdotes don’t give you a fair representation of the effects of X on Y, nor do they account for potential confounding factors. Therefore, anecdotes are worthless as evidence. They simply cannot demonstrate causal relationships. As I often say on this blog, if you want to know whether or not X causes Y, the one and only way to do it is by conducting large, properly controlled studies that account for confounding variables. Nothing else will suffice. It doesn’t matter if you have “seen it work,” it doesn’t matter if something has been used for centuries, and it doesn’t matter if a symptom has been reported in a database like VAERS or printed on a package insert. Unless proper scientific testing has shown that X causes Y, you cannot conclude that there is a causal relationship between the two.

 

Posted in GMO, Nature of Science, Rules of Logic, Vaccines/Alternative Medicine | Tagged , , , , , , , , , , | 14 Comments