Basic Statistics Part 3: The Dangers of Large Data Sets: A Tale of P values, Error Rates, and Bonferroni Corrections

Posted on April 7, 2015 by Fallacy Man

In previous posts, I have explained the importance of having lots of data, but what I failed to mention was the dangers of analyzing these large data sets. You see, all real data has variation in it, and when you have a very large data set, you can usually subset it enough that eventually you find a subset that, just by chance, fits your preconceived view. Sometimes, these erroneous results arise as a deliberate form of academic dishonesty, but other times they come from honest mistakes. Regardless of their origin, they present a very serious problem because to the untrained eye (and sometimes even to the trained eye), they seem to show scientific evidence for truly absurd positions, and an enormous number of the studies and “facts” that anti-scientists cite are actually the result of this illegitimate sub-setting of large data sets. Therefore, I want to explain how and why these erroneous results arise, how scientists deal with them, and how you can watch for them so that you are not duped by research which appears to carry all the hallmarks of good science.

Note: I have split this post up into two sections. The first just explains the phenomena without going into the actual math or the gritty details of what is going on. The second half, called “Technical Notes” explains the math behind this problem. I encourage everyone to read both sections, but you can get the basic idea just by reading the first section.

I want to illustrate how this works using a totally fictitious data set. To set up this example, let’s suppose that I was working on a particular species of frog, and I wanted to know how body size effected clutch size (i.e., the number of eggs that a female lays). I examined this by measuring female frogs and their clutch sizes at 30 populations (assume that I used proper controls so that any correlation would actually show causation). Now, remember that this is a fictional data set, so to actually generate it, I used the statistical program R to generate 30 sets of random data. Each set contained the body size and clutch size for 50 individuals. For body size, the computer randomly selected a number between 1 and 1,000 (inclusive), and for clutch size it selected a random number between 1 and 500 (inclusive). So I got “measurements” from a total of 1,500 “individuals.” When I put all of those individuals together, I got the figure below.

Figure 1: A comparison of body size and clutch size for my fictional data set (data were randomly selected). As expected, there is no relationship between our variables. You can see this in the flat trend line.

Figure 1: A comparison of body size and clutch size for my fictional data set (data were randomly selected). As expected, there is no relationship between our variables. You can see this in the flat trend line.

As we would expect, there is no relationship between body size and the number of eggs that a female laid. This is what we should find since the data came from a random number generator. In this case, it is pretty obvious to just look at the trend line and see that there are no relationships, but scientists like to actually test things statistically because statistics give us an objective way to tell determine whether or not there are any relationships. For these data, the appropriate test is what is known as a Spearman rank test. Running this test produces a P value of 0.7615. I explain P values more in the Technical Notes section, but for now, just realize that for most scientific studies, anything less than 0.05 is considered significant. So when you see a scientific study state that it found a “significant difference” or “significant relationship” it usually means that the P value was less than 0.05.

So far, everything is as it should be: there is no significant relationship. However, watch what happens when I subset the data. This time, I’m going to display each population separately.

Figure 2: These are the same data as figure 1, but the data have been subset by population so that each population is now shown separately (note: the data are fictional and were randomly generated).

Figure 2: These are the same data as figure 1, but the data have been subset by population so that each population is now shown separately (note: the data are fictional and were randomly generated).

That’s obviously a bit of a mess to look at, but a few of those lines appear to be showing a relationship, and when we run a Spearman rank test on each subset, we find that populations 2, 19, and 23 all had a significant positive relationship between frog size and clutch size!

Figure 3: This just shows the data for populations 2, 19, and 23. All three of them had significant positive relationships. (note: the data are fictional and were randomly generated)

Figure 3: This just shows the data for populations 2, 19, and 23. All three of them had significant positive relationships. (note: the data are fictional and were randomly generated)

illustration of a type one error 1

Table 1: P values for all 30 fictional populations. Anything less than 0.05 was considered statistically significant. The three significant populations have been highlighted.

Unless you already have a good understanding of probability theory, this should surprise, shock, and disturb you. We know that there is not actually a relationship between these two variables because I randomly selected the numbers, yet three of our populations are showing significant relationships! What is going on here? Are all statistics part of a massive lie by the government/big pharma/Monsanto designed to deceive us into allowing them to poison the food supply, fill our children with toxins, neuter our pets, and inject us with mind controlling chemicals!? Hardly.

The reality is that this is a well-known statistical fluke known as a type 1 error. This type of error occurs anytime that you incorrectly conclude that there is either a relationship between variables or a difference between groups (see Technical Notes for details). This is simply an unavoidable byproduct of the law of large numbers. Because of natural variation in data, you will inevitably get erroneous results from time to time. Fortunately, there are ways to control this type of error when you are working with subgroups of large data sets (see Technical Notes). The problem is that sometimes studies that did not use these controls manage to make it through the peer-review process, and the media and general public have a panic over “results” that are actually just statistical outliers.
The breeding habits of frogs rarely make the news, so to illustrate this problem, imagine instead that my data set was looking at the side effects of a vaccine, or, perhaps it was looking for relationships between GMO consumption and neurological problems among different age groups. You can no doubt envision the headlines, “GMOs are dangerous for 10-15 year olds,” “New study proves that GMOs are dangerous,” etc. The problem is that, in this example, we know that the results are spurious. They are just from random chance, but to the untrained eye, they appear to show significant relationships for certain subgroups.

My data set is fictional, but this happens with real data and it causes all manner of problems. A great example of this occurred last year in a highly publicized paper called, “Measles-mumps-rubella vaccination timing and autism among young African American boys: a reanalysis of CDC data.” The headlines from the media and anti-vaccers were predictable, “New Study Finds A 340 Percent Increased Risk Of Autism In Boys Who Receive MMR Vaccine On Time” (from Collective-evolution), “MMR vaccines cause 340% increased risk of autism in African American infants” (from Natural News), etc. When we look at the actual study though, we find that it is fraught with problems. The one that I want to focus on is sub-setting. The authors took a data set of several hundred children, then subset it by race, then subset those subsets by sex, then subset those subsets by age at injection. So we now have lots of subsets, and out of all of those subsets only the group of 36 month old African American males was significant, and the authors failed to control their type 1 error rate. Is the problem with this approach obvious? It should be. This is exactly the same thing that happened with my fictional frog data. Once you subset the data enough, you will eventually find some tiny subset that just by chance appears to support your position.

Further, even if this result wasn’t simply a statistical anomaly, the “news” articles about it are still a clear example of a sharpshooter fallacy, because 36 month old African American boys was the only group that showed a significant relationship. The headlines should have said, “MMR vaccine safe for everyone except 36 month old African American males.” Males of all other races = no relationship. African American males of other age groups = no relationship. African American girls of any age group = no relationship, etc. This one, tiny subgroup is the only group with a significant difference, making this is a classic type 1 error. Fortunately, I was not the only one who could spot the problem with this paper, and multiple scientists quickly pointed out its numerous flaws, ultimately resulting in a speedy retraction by the journal.

Now that you understand how this works, you should be able to spot this problem in lots of pseudoscientific papers. For example, many of the papers that I have read on homeopathy used lots of different measurements for the exact same treatment without controlling the type 1 error rate, and out of the 20 types of measurements that they used, one happened to show a significant improvement. When you get results like that, you shouldn’t conclude that homeopathy works. Rather, you must acknowledge that 19 out of 20 measurements showed no improvement, therefore, the one that did show an improvement was probably a statistical outlier.

Before delving into some fun mathematical details, I want to discuss one last case where this occurs. Everything that I have talked about so far has been either a deliberate manipulation of the data, or a mistake by the researchers in which they did not use the correct statistical methods. There is, however, another way that you get these spurious results without any mistakes or deception by the researchers. Let’s go back to my frog populations again, but this time, imagine that instead of studying all 30 populations, I just studied population #2 (assume that I got the same data as I did in my simulation). There was no dishonest or biased reason that I chose that population. It was just the one that was available for me to examine. Studying that population would, however, give me an erroneous, significant result, but because that was the only population that I studied, I would have no way of knowing that the result was incorrect. Even if I did everything correctly, designed a perfect project, and used the appropriate statistics, I would still get and publish a false result without ever knowing that I was doing so.

It’s important to understand this because this problem happens in real research. Papers get published by good, honest researchers saying that a drug works or a treatment is dangerous when, in fact, the results were simply from statistical anomalies. This is one of the key reasons that scientists try to replicate each others research. Suppose, for example, that 29 other researchers decided to try to replicate my results by studying the other 29 frog populations. Now, we would have 30 papers, 3 of which say that body size affects clutch size and 27 that say that there is no relationship. This lets us do something called a meta-analysis. This is an extremely powerful tool where you take the data from several studies and run your analyses across the whole data set. You can think of this like taking all the individual populations of frogs (Figure 2) and combining them into one massive population (Figure 1; the math is actually significantly more complicated than that, but that’s the basic idea). This method is great because it lets us see the central trends, rather than the statistical outliers. Also, remember that the law of large numbers tells us that the calculated value of large data sets should be very close to the true value. In other words, large data sets should give us very accurate results.

One of my favorite meta-analyses was published last year, and looked at the relationship between the MMR vaccine and neurological problems like autism. I cite this study a lot on this blog, but that is because it provides such powerful evidence that vaccines are safe. It combined the results of 10 different safety trials, giving it an enormous sample size of over 1.2 million children (feel free to rub that in the face of any anti-vaccer who says that the safety of vaccines hasn’t been tested). With that large of a sample size we expect to see true results, not statistical outliers. So the fact that it found no relationships between vaccines and autism is extremely powerful evidence that vaccines do not cause autism.

You can find these meta-analyses for lots of different topics, and they are crucially important because they have such large sample sizes. You can, for example, find scattered papers from individual trails that found significant improvements using homeopathy, but you can also find plenty of papers showing that homeopathy did not work for many trials, and it can be difficult (especially as a layperson) to wade through that flood of information and determine what is actually going on and which studies you should trust. That’s the beauty of meta-analyses, they comb through the literature for you and give you the central trends of the data. To be clear, I am not advocating that you blindly accept the results of meta-analyses. You still need to carefully and rationally examine them, just like you do every other paper, but, they often do a much better job of presenting true results.

In conclusion, I would like to give several pieces of advice to those of you who are interested in truly informing yourself about scientific topics (as I hope you all are). First, take the time to learn at least the basics of statistics. Learn how the tests work, when to use them, and how to interpret their results. I personally recommend studying the Handbook of Biological Statistics. It’s a great website that does a fantastic job of introducing a lot of statistical tests in a way that most people can understand. Second, watch out for these type 1 errors. Learn the methods that we use to control them, and when you’re reading the literature, make sure that the authors used those controls. Finally, look for the meta-analyses. They are one of our most powerful tools, and they can really help you sort through the tangled mess that is academic literature.

Technical Notes
P-values and alphas
In statistics, you are generally testing two hypotheses: the null hypothesis and the alternative hypothesis. The null hypothesis usually states either that there is no association between the variables you are testing or that there are no differences between your groups. The alternative hypothesis states the opposite. It states that there is a significant relationship/difference. The P value is the probability of getting the results that you got if the null hypothesis is actually true. So if you are comparing the average value of two groups (group 1 mean = 120, group 2 mean = 130), and you get a P value of 0.8, that means that if there is no significant difference between those groups (i.e., the null hypothesis is true) then, just by chance, you should get the result that you got 80% of the time. So most likely, your “difference” of an average of 10 is just a result of chance variation in the data. In contrast, if you get a P value of 0.0001, that means that if the null hypothesis is actually true, you should only get your results 0.01% of the time. This makes it very likely that your difference is a true difference, rather than just a statistical anomaly. Thus, the smaller your P value, the more confidence you have in your results.

It’s important to note that you can never prove anything with statistics. You can show that there is only a 0.0000000000000000000000000000000000000000000000000001% chance of getting your result if the null is true, but you can never prove anything with 100% certainty.

To objectively determine whether or not there is a significant relationship/difference, you compare your P value to a predetermined significance value called alpha. Generally speaking, you use an alpha of 0.05, but sometimes other alphas are used, especially 0.01. So, if your calculated P value is less than your alpha, you reject the null hypothesis. In other words, you conclude that your result is statistically significant. Whereas if your P value is equal to or greater than your alpha, you fail to reject the null hypothesis (this is not the same thing as accepting the null, but that’s a conversation for another post). Importantly, your alpha must be determined ahead of time, and you are not allowed to cheat. If your alpha is 0.05 and your P value is 0.051, you cannot claim significance or talk about your results as if they were significant. This is, however, another case where meta-analyses come in handy. Sometimes, there is a significant result, but your sample size was too small to detect it, so by combing your data with the data from other studies, you can boost the sample size and reveal a trend that was not visible just by looking at your data.

Type 1 and type 2 errors
A type 1 error occurs anytime that you reject the null hypothesis when the null hypothesis was actually correct. Remember, P values are the probability of getting your results if the null hypothesis is actually true. So, if you have an alpha of 0.05, then a P value of 0.049 will be significant, but you should get that result just by chance 4.9% of the time. If you think back to my fictional frog data, this is why some of the populations were significant. Because I had 30 populations, I expected that just by chance some of them would happen to have body sizes that produced a P value less than 0.05.

Now, you may be thinking, “well why not just make the alpha really tiny, that way you almost never have a type 1 error.” The problem is that then you get a type 2 error. This occurs when you fail to reject the null hypothesis, but the null hypothesis was actually incorrect. In other words, there was a significant relationship, but you didn’t detect it. So if, for example, we used an alpha of 0.00001, we would have an extremely low type 1 error rate, but we would almost never have any significant results. In other words, almost all of our studies would be type 2 errors. So the alpha level is a balance between type 1 and type 2 errors. If it’s too large, then you have too many type 1 errors, but if it is too small, then you have too many type 2 errors.

Family-wise type 1 error rates
The family-wise type one error rate is basically what this whole post has been about. When you are testing the same thing multiple times, your actual type 1 error rate is not the standard alpha. Think about it this way, if you did 100 identical experiments and the null hypothesis was true, we would expect, just by chance, that five of the experiments would have a P value of 0.05. This is, again, what happened with the frog data. So you need a new alpha level that accounts for the fact that you are doing multiple tests on the same question. This new alpha is your family wise type 1 error rate.

There are several ways to calculate the modified error rates, but one of the most popular is the Bonferroni correction. To do this, you take the alpha that you actually want (usually 0.05) and divide it by the number of tests you are doing. This results in a 5% chance that any of your results will have a P value of 0.05 if the null hypothesis is correct. So, if we apply this to my fictional frog data, I had 30 populations, so my corrected alpha is 0.05/30 = 0.00167. So, for any of my populations to show a significant relationship, they must have a P value less than 0.00167. My lowest P value was, however, 0.0174. So, now that done the correct analysis and have properly controlled the family-wise error rate, we can see that there are no significant relationships between body size and clutch size, which is, of course, what we should see since these data were randomly generated. This is why it is so important to understand these statistics: if you don’t control your family-wise error rate, you are going to get erroneous results.

It’s worth mentioning that the standard Bonferroni correction tends to slightly inflate the type 2 error rate, so I (and many other researchers) prefer the sequential Bonferroni correction. This works the same basic way but with one important difference: rather than comparing all of your P values to alpha/# of tests, you compare the lowest P value to alpha/# of tests, the second lowest P value to alpha/(# of tests – 1), the third lowest P value to alpha/(# of tests – 2), etc. You keep doing this until you reach a P value is that is greater than or equal to the alpha you are comparing it to. At that point, you conclude that that test and all subsequent tests are not significant. So, for my frog data, we are done after the first comparison because even my lowest P value was greater than 0.00167, but let’s suppose instead that my four lowest P values were: 0.0001, 0.0002, 0.0016, and 0.003. First, we compare 0.0001 to 0.05/30 = 0.00167. Since 0.0001 is less than 0.00167, we reject the null. Next, we compare 0.0002 to 0.05/29 = 0.00172. Again, reject the null. Next, we compare 0.0016 to 0.05/28 = 0.00179. Again, reject the null. Then, we compare 0.003 to 0.05/27 = 0.00185. Now, our P value is greater than our alpha, so we fail to reject the null for this test and for all 26 other tests that produced larger P values.

A final way to deal with this problem of multiple comparisons (and really the best way) is to design your experiment such that you can use a statistical test that incorporates your subsets into the model and accounts for the fact that you are making several comparisons. Entire books have been written on this topic, so I won’t go into details, but to put it simply, some tests (like the ever popular ANOVA) allow you to enter different groups as factors in your analysis. Thus, the test makes the comparisons while controlling the error rate for you. Again, I want to impress upon you that if you want to be able to understand scientific results, you need to at least learn the most common statistical tests and how and when they are used. This is fundamental to understanding modern science. If you don’t understand statistics, you’re not going to be able to understand scientific papers, at least not in a way that lets you objectively assess the authors’ conclusions.

Other posts on statistics:

 

Posted in Nature of Science | Tagged , , , | 2 Comments

What does it mean to be a skeptic?

Posted on April 2, 2015 by Fallacy Man

It is good to be skeptical about everything that you hear and read. In fact, skepticism is one of the defining characteristics of a scientist. Nevertheless, terms like “skeptic” and “open-minded” are often misappropriated by people in the anti-science movement, and many of the most biased people on the planet are under the delusion that they are skeptical. Climate change deniers, for example, often refer to themselves as, “climate change skeptics,” and it is rare to have a conversation with anti-vaccers without them referring to pro-vaccers as “sheeple.” Therefore, I want to briefly examine what it actually means to be a skeptic.

First, I want to clear up a common misconception. Many people seem to be under the impression that being a skeptic means going against the mainstream view. Thus, anti-vaccers consider themselves to be “thinking parents,” while viewing everyone else as “sheeple.” There is, however, nothing in the definition of “skeptic” that requires you to reject a scientific consensus. You are welcome to accept a consensus as long as that consensus formed as a result of strong, scientific evidence.

Having cleared up that misconception, let’s move on to the definition of a skeptic. There are basically two parts to being a skeptic. First, a true skeptic does not accept something or commit to a position unless there is sufficient evidence for that position. In other words, a skeptic questions what he/she is told and doesn’t accept anything until they have carefully studied the issue and examined the available evidence. Importantly, you must use good sources when fact checking. So, for example, reading Natural News does not constitute examining the evidence. Rather, you need to look at the original, peer-reviewed research.

This first requirement of skepticism may sound simple, but it is something that most people struggle with (including people who strongly support science). It is very easy and tempting to quickly latch onto some new study that seems to support your position, but it is crucially important that you avoid this trap. You must always carefully examine the evidence regardless of whether or not it supports your position. This is, in fact, one of the most important things that students of science get taught in graduate school. The peer-review system works well, but it is far from perfect, and sometimes bad research does get published. Therefore, you can never assume that something is true, and you must rigorously and carefully examine everything before accepting or rejecting it.

The second prerequisite for skepticism is being open-minded. This simply means that you are willing to change your position if you are shown evidence to the contrary. The term, “open-minded” has, however, been stolen and perverted by the anti-science movement. On numerous occasions, I have had people tell me that I need “open my mind about alternative medicines.” The reality is that am completely open to the possibility that alternative medicines work, but I’m not going to accept that they work until they have passed rigorous scientific testing. That’s not being close-minded, that’s being skeptical. Similarly, I have had multiple anti-vaccers tell me that my training in the sciences has made me, “close-minded.” When I pressed these people for what they meant , they explained that I was being close-minded by demanding scientific studies and refusing to accept anecdotal evidence. Think about this for a second. According to them, those of us who argue in favor of science are close-minded because we demand scientific evidence for a debate about science. Further, these same people will usually proudly proclaim that nothing will ever change their minds, which is, of course, the very definition of close-minded.

Accepting something without sufficient evidence is not being open-minded, it’s being gullible. It is not, for example, open-minded to use anecdotal evidence to arrive at the conclusion that vaccines cause autism. Rather, someone who is open-minded would reject those anecdotal reports in favor of the large, carefully controlled studies which clearly show that vaccines do not cause autism. It is important to note, however, that nothing in science is ever 100% certain. Thus, being truly open-minded means that you are always willing to consider the possibility that you might be wrong no matter how clear the data currently seem. So, for example, if in the future a large, well designed, carefully controlled study is published showing that vaccines do cause autism, and the results of that study are replicated by other researchers, I promise you that skeptics around the world (myself included) will write and prominently display posts admitting that we were wrong about the relationship between vaccines and autism. That is what it truly means to be open-minded. It means that you are willing o change your view when presented with good evidence. It does not mean that you are willing to blindly accept something despite a lack of evidence.

In summary, a skeptic is simply someone who demands good evidence before accepting something and is willing to change their view when it conflicts with the evidence. These requirements are easy to say, but often hard to follow. Nevertheless, everyone should strive to be a skeptic. You should use good sources, question your assumptions, demand evidence, beware of cognitive biases, and above all else, never hold any position so dearly that you are not willing to challenge it.

Posted in Uncategorized | Tagged , | Comments Off on What does it mean to be a skeptic?

Evolution is blind

Posted on March 29, 2015 by Fallacy Man
Note: the notion that evolution is blind has nothing to do with Darwin's eyesight. I just thought this was an amusing image.

The notion that evolution is blind has nothing to do with Darwin’s eyesight. I just thought this was an amusing image.

One of the central tenets of evolutionary biology is the concept that evolution is blind. In other words, it has not foresight or goal. This principle is extremely important for understanding how evolution works, but it’s a concept that is often misunderstood, even among people who accept evolution. Further, this lack of understanding often leads to a number of erroneous creationist arguments such as the claim that whales defy evolutionary theory. Similarly, the popular irreducible complexity argument is easily defeated once we understand that evolution is blind.

Note: in this post, I am going to use “evolution” to refer specifically to evolution by natural selection. Remember that evolution is simply a change in allele frequencies over time, and natural selection is one among several mechanisms that causes allele frequencies to change (others include things like gene flow and genetic drift). The concept of blind evolution is, however, most germane to evolution by natural selection; therefore, that is what I will be describing in this post.

Before I can explain what is meant by, “evolution is blind,” I have to give a brief primer on how evolution by natural selection works. Natural selection requires two things: heritable variation for a trait and selection for that trait. Both of these are nearly always met in real populations. In other words, in any population there is variation for a trait (e.g., not all individuals are the same height) and that variation is nearly always heritable (e.g., tall individuals tend to produce tall offspring). Further, that variation affects fitness (e.g., tall individuals may be able to get more food, which gives them more energy, which lets them produce more offspring), so certain traits get “selected” by virtue of the fact that the individuals with those traits produce more offspring than individuals without those traits. As a result, the genes for the beneficial trait will be more common in the next generation. Thus, the population will evolve because its allele frequencies will change.

There are several important clarifications to be made here. First, everyone accepts natural selection. It’s a simple mathematical certainty (even young earth creationist organizations accept it, they just place arbitrary and logically invalid limits on it). Second, evolution only acts on populations not individuals. Individuals simply cannot evolve. Third, in evolutionary terms, “fitness” refers to the number of genes that you get into the next generation, not physical strength. Generally speaking, being physically fit does give you a higher evolutionary fitness, but not always. Thus, the phrase, “survival of the fittest” is something of a misnomer. Survival is important only in that it gives you more time to produce offspring, and there are plenty of short lived species that have a high evolutionary fitness. For example, think about species like many octopuses where the females die after laying their first and only clutch of eggs. They have a lower survivorship, but a high fitness (fun fact: “octopi” = several within a species, “octopuses” = several species). Thus, natural selection only acts on traits that affect your reproductive potential. These may be traits that directly influence your survivorship, such as antipredatory behaviors, but they can also be traits like foraging ability (more food = more energy = more offspring) and the ability to attract a mate (i.e., sexual selection).

It may seem like I have digressed from my thesis, but this is all important groundwork for understanding blind evolution. They key here is that evolution has no foresight or direction. In other words, it has no goal or endpoint in mind. In each generation, it simply adapts populations to their current environment, but if that environment changes, then an adaptation that has been useful for thousands of generations can suddenly be detrimental. Let’s say, for example, that we have a group of birds who eat very small seeds that are held in little folds of the plants. Thus, they need fairly small, skinny beaks to get to the seeds. Therefore, in each generation, the birds with the beaks that are most well suited to reaching into the folds get the most food and produce the most offspring. So for many generations, evolution has been shaping the birds beaks to fit in these folds. However, one year there is a massive drought, and all of the plants that the birds usually get seeds from die, but another species with large thick seeds survives. Now, the small, petite beaks that have been so useful are suddenly detrimental, and large thick beaks are useful. This means that the birds who previously would have had a very high fitness are now going to have a very low fitness, and the thick billed birds that would previously have had a low fitness are suddenly going to have a high fitness. This is what we mean by, “evolution is blind.” It cannot anticipate the future needs of an organism. All it does is adapt a population to its present environment.

Hopefully, at this point, the problem with creationists’ whale argument is clear. For those who aren’t familiar with this argument, creationists often claim that whales are a problem for evolution because evolutionary history tells us that all land organisms evolved from a marine ancestor, but whales would have had to evolve from a terrestrial ancestor. Thus, creationists claim that whales had to evolve “backwards” or “de-evolve” because they went back to water. Similar arguments are made about species like flightless birds.

The problem with these arguments is simply that they ignore this concept of evolution being blind. There is no “forwards” or “backwards.” At one point in time, for a certain population of marine organisms, it was beneficial to be able to come out on land. Therefore traits that allowed individuals to come out on land were selected for. Then, millions of years later, for a certain population of land-dwelling mammals, it was beneficial to be able to go into the water. Therefore, nature selected the traits that allowed individuals to enter the water. This is in no way shape or form a problem for evolution because in both cases, populations were evolving to match their current environment.

It’s also worth noting that there is really no such thing as being “more evolved” because evolution has no direction. Chimps are not, for example, more evolved than a single celled bacteria living in a hot spring. Chimps are certainly more complex, and they certainly have accumulated more genetic changes, but they are not “more evolved” because that suggests that evolution is directional. Both chimps and bacteria are well adapted to their current environments, and that is all that evolution does: it adapts populations to their present environments. Think about it this way, a chimp would die in the hot springs where many bacteria thrive, and the specialized bacteria would die in the chimp’s rainforest. They are both highly adapted for their environments, but neither one is more evolved than the other. To put this another way, you can say that chimps are more evolved for a life in the forest, but you cannot make a broad comparison between them and bacteria because it is equally fair to say that certain bacteria are more evolved for a life in the hot springs.

It’s also important to realize that because evolution is blind, we don’t expect it to produce perfect organisms. Rather, we expect organisms to be a hodgepodge of former traits. In other words, we expect them to have a large number of evolutionary leftovers. We call these leftovers vestigial traits. I plan on devoting an entire post to them in the future because they provide extremely strong evidence for evolution, but to describe them briefly, these are traits that have no function or a very limited function in the current organism, but they would have been fully functional in that organism’s ancestors. Blind cave fish are the classic example of vestigial structures. These are fish that have eyes, but the eyes are no longer functional and often have a layer of skin growing over them because the fish spends its whole life inside a pitch black cave. So, at one point in time there was a population of fish living outside of a cave that had functional eyes. Then, for one reason or another, the fish ended up inside of the cave where the eyes were no longer useful. Thus, nature stopped selecting for vision, and the eyes slowly accumulated mutations to the point that they are now useless. Animals are full of examples of these structures. For example, baleen whales and certain snakes retain non-functional pelvic bones. Humans also have many vestigials. Our tail bones, goose bumps, wisdom teeth, and multiple other features are all vestiges of our evolutionary history. They are evolutionary leftovers that were beneficial in previous environments and situations but are no longer beneficial today.

Finally, I want to briefly discredit irreducible complexity. I addressed this topic in detail here, but to put it in its simplest terms, irreducible complexity states that certain biological systems are highly complex to the point that removing any one part prevents the system from function. For example, irreducible complexity claims that the bacterial flagellum couldn’t have evolved because it requires 42 proteins to function (for most species), and if any one protein is removed, it no longer functions as a flagellum. Thus, the argument is that it couldn’t have evolved because no one protein would be useful unless all of the other proteins were already in place. The problem is that this argument sets up the flagellum as some ultimate endpoint that evolution is working towards, but as you now know, that’s not the way that evolution works. Each protein doesn’t need to function as a flagellum, it just has to function. In other words, if the protein does anything useful, it will be selected for. You see, the function of a trait can change in response to new environments or as a result of new mutations. In fact, we know that all of the proteins in flagella are used for other things in the cell, and often several of them work together to perform a function. It is not hard to imagine a series of mutations that brings these functions together. In fact, we have a hypothetical pathway that would allow a flagellum to evolve step by step with each step being useful. Only the final step functions as a flagellum, but that doesn’t matter because each step still functions, and that is all that evolution needs because it is a completely blind process that acts without any forethought or anticipation of future needs.

Summary
Evolution by natural selection simply adapts populations to their current environments. It cannot anticipate future environments or needs. As a result, a trait may be selected for in one generation, and selected against in a later generation after the environment changes. Therefore, it is incorrect to describe evolution as having a “direction” because it is simply responding to the current conditions.

Related posts:

Posted in Science of Evolution | Tagged , , , , | Comments Off on Evolution is blind

7 easy ways to lose a debate

Posted on March 27, 2015 by Fallacy Man

One of the saddest statistics about my life is the amount of time that I spend pointlessly debating anti-scientists. Having devoted so much time to this endeavor has, however, allowed me to observe certain patterns and trends in their debate tactics. Specifically, there are several flawed debate strategies that anti-scientists frequently employ. In fact, I have rarely been in a debate where my opponent did not at some point commit one of these. So in this post, I am going to describe seven fundamentally flawed, yet extremely common debate tactics that you should watch out for and avoid at all cost. If you commit one of these, then you have automatically lost the debate.

When I say that you lost, I am obviously being somewhat facetious because the fact that you used a bad argument doesn’t inherently mean that your position itself is flawed (that would be a fallacy fallacy). Nevertheless, these strategies generally arise as a result of a fundamental weakness in the position that is being defended. As such, their use does generally indicate a critical problem that must be taken seriously. Finally, even if the position you are arguing for is actually correct, these strategies are so flawed that I would still contend that using them automatically costs you the debate itself (it is technically possible to lose a debate, but still be right).

 

#1. Accusing your opponent of being a shill
This tactic, often called the shill gambit, is perhaps the most common flawed strategy. Many other commentators on science have explained the problems with it, so I will try not to belabor the point too much. To describe it briefly, this occurs whenever you simply accuse your opponent of being paid off rather than actually engaging with their arguments. So, for example, when I explain the science of vaccines to anti-vaccers, they frequently respond by saying, “you’re just supporting vaccines because you’re a shill for Big Pharma.”

The problems with this response are numerous. First, it’s nothing more than an ad hominem fallacy. It attacks your opponent rather than attacking his/her position. Second, it’s an ad hoc fallacy and places the burden of proof on you. In other words, you cannot accuse someone of being a shill unless you have actual evidence that they are in fact being paid off. The fact that someone would dare to disagree with you does not automatically mean that they have a financial motive for doing so. Finally, it is often a red herring fallacy because it conveniently dodges any contrary evidence that has been present against your position (more on that later).

 

#2. Accusing your opponent’s sources of being shills
This tactic fails for the same basic reasons that the shill gambit fails, but its structure is somewhat different so I want to talk about it separately. In this case, you don’t accuse the actual person that you dealing with of being paid off. Rather, you blindly reject all of their sources by accusing the authors of being shills. For example, I frequently present anti-vaccers with papers that discredit their arguments, and when I ask them why they refuse to accept those papers, they almost invariably respond that, “the study was funded by Big Pharma” or “those authors were paid off” or “they are just in it for the money.” As with the normal shill gambit, this is nothing more than an ad hominem assault. It confidently ignores the results of the paper by attacking its authors. Also, scientific papers always included funding sources and author affiliations. So you can check and see whether or not the authors had a financial conflict of interest. When I point this out to anti-vaccers, they usually claim that the money is being distributed under the table and isn’t officially reported, but at that point, we are back to logically invalid ad hoc reasoning, and, once again, the burden of proof is on them.

 

conspiracy theory skeptic science expert#3. Inventing a conspiracy
This is closely related to the first two flawed tactics, but it has a much wider scope. So wide, in fact, that it utterly astounds me that anyone would ever use it. For this flawed strategy, you don’t simply accuse your immediate opponent of being a shill, nor do you accuse a handful of scientists of being corrupt. No, for this logical blunder, you accuse the entire scientific (and/or medical) community of being involved in some insanely large conspiracy. The exact nature of the supposed conspiracy varies widely, but the core reason for proposing it is always the same: when faced with the fact that nearly all experts disagree with them, people try to discredit all of those experts in one fell swoop by accusing all of them of being in a massive conspiracy. This approach suffers all of the problems of the shill gambits, but those problems are magnified a thousand times. Now, the burden of proof doesn’t simply require you to prove that one or two people or corrupt; it requires you to prove that hundreds of thousands of people are corrupt! If you have to invent an extensive global conspiracy of astronomical proportions to defend your position, then your position is so fundamentally weak that it is beyond saving.

 

#4. Shifting the goal posts
This problem arises from inconsistencies in your arguments and views. Basically what happens is that when you are faced with contrary evidence, you simply make an ad hoc modification to your view rather than rejecting it. This most clearly occurs when you have stated a clear challenge or dogmatic thesis. For example, many anti-vaccers are adamant that thimerosal in vaccines is causing autism. One of the key problems with this claim is the fact that thimerosal hasn’t been in childhood vaccines for over a decade, yet autism rates haven’t dropped. When presented with this fact, anti-vaccers generally change their thesis and begin to adamantly proclaim that it is the aluminum in vaccines that is causing autism. At its core, the problem here is that this line of reasoning is ad hoc. Anti-vaccers have decided beforehand that vaccines cause autism, so when shown evidence to the contrary, they simply modify their explanation of how vaccines are causing autism. The difficulty with this is that nothing will ever convince them that vaccines don’t cause autism. No matter what data they are presented, they will keep shifting their goal posts so that the conclusion is always that vaccines cause autism (this often involves question begging fallacies).

Global warming deniers not only commit this blunder, but they generally do so in a predictable pattern. When I make the mistake of engaging them, they generally start with the position that the planet isn’t even warming. With no small amount of effort, I can usually at least convince them that the planet is in fact warming, at which point, they simply change tune and claim that it’s just a natural warming and it’s not our fault. Using satellite data (Harries et al. 2001; Griggs and Harries 2004; Chen et al. 2007) and isotope ratios (Bohm et al. 2002; Ghosh and Brand 2003;Wei et al. 2009), I can sometimes persuade them that we are actually the cause, but if I am successful in that endeavor, they inevitably retort that it won’t actually be that bad, so we don’t need to do anything about it. Sadly, when shown evidence to the contrary, they usually either resort to shifting back to one of their previous positions or they just go full conspiracy theorist. The problem is, again, that they have already decided what is correct, so nothing will ever make them reject their view. Rather, they will simply continue to twist, contort, and modify it indefinitely.

 

#5. Playing red herring
The red herring fallacy occurs when you simply ignore one of your opponent’s arguments. This can be very obvious, such as playing the shill card and constructing a conspiracy theory, but it is often more subtle. People often commit this fallacy by responding in a way that gives the illusion that they answered the criticism when in fact they addressed a totally separate issue (politicians are masters at this, watch for it in the next political debate).

I want to focus on the more blunt version of this because it is the one that I see most frequently in internet debates. One of my favorite debate tactics is either to present a logical proof for my position or ask a question that reveals a fundamental inconsistency in my opponent’s reasoning. For example, I might present a logical proof for anthropogenic climate change, or I might ask young earth creationists why they use science to interpret Joshua but not Genesis. Both of these illuminate fundamental problems with my opponents’ positions that must be addressed in order for those positions to be valid. When faced with these problems, my opponents nearly always play red herring. They simply ignore my challenge and move on to some other point. This is not a logically valid strategy. You must deal with serious criticisms of your position, and you cannot simply ignore any arguments that discredit your view. I generally refuse to let people get away with this and continually come back to my challenge until they finally deal with it (or usually just commit one of the other blunders that I have already discussed).

 

#6. Citing Natural News, Mercola, Whale.to, etc.batman slap meme natural news valid source
The internet is full of terrible, misleading, and downright dishonest sources, and if your argument is based on those sources, then you have a very serious problem. Similarly, when faced with information that opposes your position, you cannot cite these illegitimate sources. You must defend your position using valid sources, and in science, that means using the peer-reviewed literature. If you cannot find support for your position there, then support for your position doesn’t currently exist.

 

#7. Stating that nothing will change your mind
This should be an obvious problem, but for some reason I frequently encounter people who proclaim this with pride as if the strength of their resolve correlates with the strength of their position. I wrote about this problem in detail here, but to briefly summarize, you must always be willing to challenge your views. If you start off by ardently proclaiming that you are correct, then you will always find a way to twist the facts so that they seem to fit your preconceived views. The fundamental problem is that at this point it is impossible to disprove your position no matter how clearly flawed it is. In other words, even if your position is the stupidest thing that anyone has ever come up, nothing will ever convince you of that because you have already decided that you are right and, therefore, you won’t even look at the evidence to the contrary. This is an extremely serious problem, and it is a cognitive trap that you should avoid at all cost. Never hold any position so dear that you are not willing to admit the possibility that you might be wrong.

Posted in Global Warming, Rules of Logic, Science of Evolution, Vaccines/Alternative Medicine | Tagged , , , , , , , , , | Comments Off on 7 easy ways to lose a debate

2 biggest lies of the anti-vaccine movement

Posted on March 22, 2015 by Fallacy Man
measles isn't harmless meme anual deaths

In 2013, 145,700 people died from measles. Given numbers like that, it is downright dishonest to characterize measles as a mild illness.

It’s no great secret that the anti-vaccine movement is rife with scientific inaccuracies and logical fallacies, but a few of their claims are so extraordinarily erroneous and demonstrably false that I have difficulty calling them anything other lies. There are several of these falsehoods that I considered writing about, such as the blatantly false claim that vaccines contain mercury and aborted fetal cells, but I ultimately decided to focus on just two dangerous and pervasive assertions: the claim that the safety of vaccines hasn’t been tested, and the claim that measles isn’t a dangerous disease.

 

The safety of vaccines hasn’t been tested
This claim is something of a rallying cry for the anti-vaccine movement, and I repeatedly see it on anti-vaccer blogs and memes. If it was true, it would indeed be a serious problem, but it is demonstrably false. There are literally hundreds of papers on the safety of vaccines, and they are easy to find. Get on Google Scholar or PubMed and search for papers on the safety of vaccines. In mere seconds, you will be rewarded with numerous safety trials. That is why I consider this claim to be a lie. It’s not a debatable claim about the toxicity of a chemical, experimental results, etc. Rather, it is a simple factual statement. Either these papers exist or they don’t, and they clearly and undeniably exist. Therefore, anyone who tells you that the safety of vaccines hasn’t been tested is either lying or willfully ignorant. Either way, they clearly aren’t a trustworthy source of information.

Now, some people will object that I am being too general. They’ll say that although a few safety trials have been done, these studies are limited and don’t address all possibilities. This claim is usually followed by an example of a specific type of study which they believe has not been conducted. I clearly can’t make a blanket statement that none of these claims are true, but the vast majority aren’t, and it takes only a few seconds to discredit them. So, again, the people making these claims are either dishonest or willfully ignorant.

Here are just a few examples of studies that supposedly don’t exist.

Claim: There are no double blind, placebo controlled studies on the safety of vaccines
Reality: Yes there are. I entered, “vaccine safety double blind placebo controlled” on Goggle Scholar and got 62,000 results. Granted, not all of those are going to actually be relevant, but many of them are. For example, Zhu et al. 2010 and Cutts et al. 2005 were among the first three results. Again, the claim that the anti-vaccers are making is absurdly easy to test. It takes mere seconds. So the fact that they don’t bother to test it should be extremely disturbing to people.

Claim: “There are no studies comparing the health of the vaccinated and unvaccinated”
Reality: Yes there are.

Claim: There are no studies on the effects of the large number of antigens in vaccines/the effects of multiple vaccines
Reality: Yes there are.

I could keep going, but hopefully by now you get the point: scientists have very carefully studied the safety of vaccines from multiple angles and the result has consistently been that they are safe. Anyone who tries to tell you otherwise is either deceptive or simply doesn’t know what they are talking about. Either way, you should not be getting medical advice from this person.

 

Measles isn’t a dangerous disease
This claim has been around for a long time, but it gained recently popularity after the Disneyland outbreak. In response to the accusations that they were responsible for the spread of the disease, many anti-vaccers argued that measles is a “harmless childhood disease” and isn’t really that serious. I have seen this claim on innumerable anti-vax blogs and memes, and there is even a truly horrifying children’s book based around this notion. The reality is completely different. During the supposedly harmless Disneyland outbreak, for example, 20% of patients were hospitalized. Anything that requires hospitalization deserves to be called “serious.” You don’t take people to the hospital for harmless childhood diseases. You take them to the hospital for serious medical conditions that require expert intervention. I have never once seen a parent stand beside their child’s hospital bed, shrug their shoulders and say, “it’s just a normal childhood illness.” Further, just because these children recovered doesn’t mean that the illness wasn’t serious. Many people recover from heart attacks, but that doesn’t make cardiac arrest a “minor illness.”

Now, let’s look at some general data rather than data from a specific case. On average, 1 out of every 20 people who get measles will develop pneumonia, which can be deadly in young children. In fact, 1 out of every 1,000 measles infected children will die as a result. Something that kills children clearly should be taken seriously and is not harmless. Further, even if your child doesn’t die from the measles, there is a 1 in 1,000 chance that he/she will develop encephalitis, which is a swelling of the brain that can cause fun side-effects such as seizures, permanent deafness, and mental disabilities. Again, those are not the side effects of a harmless disease.

At this point, you may be thinking, “fine, measles can be dangerous, but for most children it’s not that big of a deal.” That’s all well and good until it’s your child lying on a hospital bed or the cloth padding of a coffin. In my opinion, 1 in 1,000 is still far too many deaths for a disease that we can eliminate. To be clear, I’m not committing an appeal to emotion fallacy here, because hospitalizations and even deaths are very real possibilities from measles infections. For example, although 1 in 1,000 may sound small, realize that thousands of children die from measles annually. In 2013, for example, 145,700 children died from measles. Harmless childhood disease? I think not.

You may be tempted to ignore numbers like 145,700 based on the fact that you live in a luxurious, developed country while most of those deaths came from third-world countries, but do you know why most of those deaths came from developing countries? It’s because they don’t have good access to vaccines! As vaccination rates increase, the death toll decreases. The only reason that we don’t have hundreds of children dying from measles in the US is because we have relatively high vaccination rates. If those rates drop, measles will come back and children will suffer and die. That’s not fear mongering, that’s a scientific fact. For example, low vaccination rates recently caused an outbreak in the Netherlands which resulted in 147 complications (14.4% of cases) including 90 cases of pneumonia and 82 hospitalizations. Fortunately, no one died during that outbreak; however, a recent outbreak in France (which also centered around unvaccinated communities) claimed 10 lives and caused almost 5,000 hospitalizations and over 1,000 cases of “severe pneumonia.” Does that honestly sound like a mild childhood illness?

America USA measles deaths vaccine before after anual

This graph shows the annual number of measles deaths in the USA immediately before and after the introduction of the vaccine. Source: CDC

Finally, let’s back the clock up to 1953, 10 years before the first measles vaccine. By any reasonable standard, American was a devolved country with high health and sanitation standards in 1953. Yet in the absence of a vaccine, 462 people died from measles. Similarly high numbers continued until 1963 when the first measles vaccine was introduced. With the introduction of this vaccine, the number of deaths began to drop, and no year since then has had a death toll that was higher than the average for the 10 year period prior to the vaccine (mean = 440.3 deaths from 1953–1962). Further, in 1968, a more effective live vaccine was introduced, and 1968 set a new record low of only 24 deaths! This clearly demonstrates two things. First, the vaccine clearly works. Our sanitation practices didn’t magically improve in the late 60s. The decline was unambiguously from the vaccine. Second, measles is a serious disease. For the 10 years immediately prior to the vaccine, an average of 440 people in the US died from measles each year. There is no universe in which that is a “harmless childhood disease,” and anyone who tells you that measles isn’t serious is either dishonest or delusional.

A note on appeal to emotion fallacies: It’s worth pointing out that appeals to emotion are only fallacious when all that they are doing is appealing to emotion rather than making a rational argument, or if they make some irrelevant appeal to emotion (they are often associated with straw man fallacies). There is, however, nothing wrong with a logical, factual argument also evoking emotions. In the case of this post, all of the death rates, hospitalizations, etc. are facts, and they are facts that should evoke a strong emotional response, but simply evoking an emotional response does not automatically make them appeal to emotion fallacies.

Let me give an example of an actual appeal to emotion fallacy to show the contrast. Have you ever seen those anti-vaccine pictures where a screaming child is receiving a massive shot that is filled with a sinister looking fluid? Well those images are perfect appeal to emotion fallacies. First, the fact that a child cries from a shot is irrelevant because children also cry when you tell them that they can’t have ice cream for breakfast, tell them to go to bed, etc. So a child crying is clearly not a good indication of whether or not something is ultimately good for them. Second, these images also make an irrelevant and dishonest appeal to fear (this is where the straw man comes in). By showing you a horse needle filled with a colored fluid, they are deliberately playing into your emotions to make you scared of the vaccine. In reality, the tiny needles and clear fluids used in vaccines are far less frighting. Do you see the difference between images like that and simply presenting facts that happen to evoke emotions? One is deliberately misrepresenting the situation for the express purpose of playing on your emotions, whereas the other is presenting an accurate portrayal of the situation, and that situation just happens to be horrible.    

Posted in Vaccines/Alternative Medicine | Tagged , , , , | Comments Off on 2 biggest lies of the anti-vaccine movement