Basic statistics part 4: understanding P values

Posted on December 28, 2015 by Fallacy Man

If you’ve ever read a scientific paper, you’ve probably seen a statement like, “There was a significant difference between the groups (P = 0.002)” or “there was not a significant correlation between the variables (P = 0.1138),” but you may not have known exactly what those numbers actually mean. Despite their prevalence and wide-spread use, many people don’t understand what P values actually are or how to deal with the information that they give you, and understanding P values is vital if you want to be able to comprehend scientific results. Therefore, I am going to give a simple explanation of how P values work and how you should read them. I will try to avoid any complex math so that the basic concepts are easy for everyone to understand even if math isn’t your forte.

Note: for the sake of this post, I am not going to enter into the debate of frequentist (a.k.a. classical) vs. bayesian statistics. Although bayesian methods are becoming increasingly common, classical methods are still extremely prevalent in the literature and it is, therefore, important to understand them.

Hypothesis testing and the types of means
Before I can explain P values, I need to explain hypothesis testing and the difference between a sample mean and a population mean. To do this, I’ll use the example of turtles living in two ponds. Let’s say that I am interested in knowing whether or not the average (mean) size of the turtles in each pond is the same. So, I go to each pond and capture/measure several individuals. Obviously, however, I cannot capture all of the individuals at each pond (assume that there are hundreds in each pond), so instead I just collect 10 individuals from each pond. The mean carapace lengths (carapace = the top part of a turtle’s shell) from my samples are: pond 1 = 20.1 cm, pond 2 = 18.4 cm.

Now, the all important question is: can I conclude that on average, the turtles in pond 1 are larger than the turtles in pond 2? Maybe. The problem is that we don’t know whether or not my sample was actually representative of all of the turtles in the ponds. In other words, those numbers (20.1 cm and 18.4 cm) are sample means. They are the average values of my samples, and the sample means are clearly different from each other, but that doesn’t actually tell us much. It is entirely possible that, just by chance, I happened to capture more large turtles in pond 1 than pond 2.

To put this another way, we are actually interested in the population means. In other words, we want to know whether or not the mean for all of the turtles in pond 1 is the same as the mean of all of the turtles in pond 2, but since we can’t actually measure all of the turtles in each pond, we have to use our sample means to make an inference about the population means. This is where statistical testing and P values come in. The tests are designed to take our sample size, sample means, and sample variances (i.e., how much variation is in each set of samples) and use those numbers to tell us whether or not we should conclude that the difference between our sample means represents an actual difference between the population means.

For these statistical tests, we usually have two hypotheses: a null hypothesis and an alternative hypothesis. The null hypothesis states that there is no difference/relationship. So in our example, the null hypothesis says that the population means are not different from each other. Similarly, if we were looking for correlations between two variables, the null hypothesis would state that the variables are not correlated. In contrast, the alternative hypothesis says that the population means are different, the variables are correlated, etc.

The P value
Now that you understand the difference between the types of means and types of hypotheses, we can talk about the P value itself. For my fictional turtle example, the appropriate statistical test is the Student’s T-test (note: I got the values of 20.1 cm and 18.4 cm by using a statistical program to simulate two identical populations and randomly select 10 individuals from each population). When I ran a T-test on those data, I got P = 0.0597, but what does that mean? This is where things get a bit tricky. Despite what some people will erroneously tell you, the P value is not, “the probability that you are correct” or “the probability that the difference is real.” Rather, the P value is the probability of getting a result of your observed difference/correlation strength or greater if the null hypothesis is actually true. So, in our example, the difference between our sample means was 1.7 cm (20.1-18.4 cm) and the null hypothesis was that the population means are identical. So, a P value of 0.0597 means that if the populations means are identical, we will get a difference of 1.7 cm or great 5.97% of the time (to turn a decimal probability into a percent just multiply by 100). Similarly, for a correlation test, the P value tells you the probability of getting a correlation as strong or stronger than the one that you got if the variables actually aren’t correlated.

normal distribution p values two-tailed t test

Figure 1: these are the results of 10,000 samples from my identical, simulated ponds. For each sample (10 individuals per pond) I subtracted the mean for the pond 2 sample from the mean for the pond 1 sample. The occurrences highlighted in blue had a difference that was equal to or greater than the difference in our first sample (1.7).

To demonstrate that this works, I took the same simulated ponds that I sampled the first time, and I made 10,000 random samples of 10 individuals from each population. For each sample, I calculated the difference between the sample means for pond 1 and pond 2, resulting in Figure 1. Out of 10,000 samples, 525 had a difference of 1.7 or greater. To put that another way, the two population means were identical and just by chance I got our observed difference or greater 5.25% of the time, which is extremely close to the calculated value of 5.97% (because the sampling was random, you wouldn’t expect the numbers to match perfectly).

When you look at Figure 1, you may notice something peculiar: I included both differences of =/> 1.7 and =/< -1.7 in my 525 samples. Why did I include the negatives? The answer is that our initial question was simply “are the turtles in these ponds different?” In other words, our null hypothesis was “there is no difference in the population means” and our alternative was “there is a difference in the population means.” We never specified the direction of the difference (i.e., our question was not, “are turtles in pond 1 larger than turtles in pond 2?”). A non-directional question like that results in a two-tailed test. In other words, because we did not specify the direction of the difference, we were testing a difference of the size 1.7 rather than testing the notion that pond 1 is 1.7 larger than pond 2.

You can do a one-tailed test in which you are only interested in differences in one direction, but there are two important things to note about that. First, your hypotheses are different. If, for example, you want to test the idea that turtles in pond 1 are larger than turtles in pond 2, then your null is, “the population mean of turtles in pond 1 is not larger than the population mean of turtles in pond 2” and your alternative is, “the population mean of turtles in pond 1 is larger than the population mean of turtles in pond 2.” Notice, this does not say anything about the reverse direction. In other words, your null is not that the means are equal, so a result that pond 2 is greater than pond 1 would still be within the null hypothesis and would not be considered statistically significant.

Figure 2: These are the same data as figure 1, but this time only the results where pond 1 was 1.7 or greater than pond 2 are highlighted.

Figure 2: These are the same data as Figure 1, but this time only the results where the sample mean for pond 1 was 1.7 cm or greater than the sample mean for pond 2 are highlighted.

Second, if you are going to do a one-tailed test, you have to decide that you are going to do that before you collect the data. It is completely inappropriate to decide to do a one-tailed test after you have collected your data because it artificially lowers your P value by ignoring one half of the probability distribution. Look at the bell curve in Figure 1 again. You can see that just by chance you expect to get a result of +/-1.7 or greater 5.25% of the time, but if you ignore the differences on the negative side of the distribution (Figure 2), then suddenly you are looking at a probability of 3.95% because if the null hypothesis is true, getting a difference of =/> 1.7 is less likely than getting a difference that is either =/> 1.7 or =/< -1.7 (typically, one-tailed values are half of the two-tailed value, but because of chance variation, this sample came out with a slight negative bias). If you had a good biological reason for thinking that pond 1 would be greater than pond 2 before you started then you could and should use the one-tailed test because it is more powerful, but you can’t decide to use it after looking at your data because that makes your result look more certain that it actually is (this is something to watch out for in pseudoscientific papers).

What does statistical significance mean
At this point, I have explained what P values mean in technical terms, but the question remains, what do they mean in practical terms? In our example, we got a P value of 0.0597, but what does that actually mean? In short, we use various cut off points (known as alpha [α]) to determine whether or not the P value is “statistically significant.” What α you use depends on your field and question, but it always has to be defined before the start of your experiment. In biology, α = 0.05 is standard, but other fields use 0.1 or 0.01. If your P value is less than your α, you reject the null hypothesis, and if your P value is equal to or greater than your α, you fail to reject the null. In other words, if your α = 0.05 and your P value is less than that, your conclusion would be that the observed difference between your sample means probably represents a true difference between your population means rather than just chance variation in your sampling. Conversely, if your P value was 0.05 or greater, you would conclude that there was insufficient evidence to support the conclusion that the differences between the sample means represented a real difference between the population means. This is not the same thing as concluding that there is no difference between the population means (more on that later).

It should be clear by this point that we are dealing with probabilities, not proofs. In other words, we are reaching conclusions about what is most likely true, not what is definitely true. The astute reader will realize that if a P value of 0.049 means that you will get that result by chance 4.9% of the time if the null hypothesis is actually true, then for every 20 tests with a P value of 0.049, one of them will be a false result (on average). This is what we refer to as a type I error. It occurs when you reject the null, but should have actually failed to the reject the null, and it is the reason that we like to have small α values: the larger the α, the higher the type I error rate (I explained type I errors in far more detail here). This also explains why some published results are wrong, even if the authors did everything correctly, and it once again demonstrates the importance of looking at a body of literature rather than an individual study.

Now, you may be thinking that we should try to make the α values very tiny, that way we rarely get false positives, but that creates the opposite problem. If the α is tiny, then there will be many meaningful differences which get ignored (this is known as a type II error). Thus, the standard α of 0.05 is a balance between type I and type II error rates.

Statistical significance and biological significance are not the same thing
This is an extremely important point that is true regardless of whether or not you got a statistically significant result. For example, let’s say that chemical X is dangerous at a dose of 0.5 mg/kg, and you do a study comparing people who take pharmaceutical Y to people who don’t, and you find that people who take Y have an average of 0.2 mg/kg and people who don’t take Y have an average of 0.1 mg/kg and the difference is statistically significant. That doesn’t in anyway shape or form show that Y is dangerous because the levels of X are still lower than 0.5 mg/kg. In other words, the fact that you got a significant difference does not automatically mean that you found something that is biologically relevant. The different levels of X may actually have no impacts whatsoever on the patients.

Conversely, if you did not get a significant difference, that would not automatically mean that there isn’t a meaningful difference. Statistical power (i.e., the ability to detect significant differences/relationships) is very strongly dependent on sample size. The larger the sample size, the greater the power. Consequently, if the population means are very different from each other, then you can get a significant result with a small sample size, but if the population means are very similar, then you are going to need a very large sample size. This is part of why you fail to reject the null, rather than accepting the null. There may be an actual difference between your population means, but you just didn’t have a large enough sample size to detect it. For example, let’s say that a drug causes a serious reaction in 5 out of every 1000 people and that “reaction” already occurs in 1 out of ever 1000 people (those are the population ratios), but when you test the drug, you only use sample sizes of 1000 people in the control group and 1000 people in the experimental group, resulting in sample ratios of 6/1000 and 1/1000. When you run that through a statistical test (in this case a Fisher’s exact test) you get P = 0.1243. So, you would fail to reject the null hypothesis even though the drug actually does cause the reaction that you were testing. In other words, the drug does cause adverse reactions, but your sample size was too small to detect it. If, however, your sample sizes had included 2000 people in each group, and you had gotten the same ratios, you would have had a significant difference (P = 0.0128) because those extra samples increased the power of your test. This is why scientists place so much weight on large sample sizes and so little weight on small sample sizes. Research that uses tiny sample sizes is extremely unreliable and should always be viewed with caution.

Finally, it is worth noting that the population means of any two groups will nearly always be different, but that difference may not be meaningful. Going back to my turtle example, for any two ponds, if I measured all of the turtles in each pond, it is extremely unlikely that the two means would be identical. There is almost always going to be some natural variation that makes them slightly different, but, with a large enough sample size, you can detect even a very tiny difference. So, for example, if I had two turtle ponds whose population means were  18.01 cm and 18.02 cm, and I had several million turtles from each pond, I could actually find that there is a statistically significant difference between those ponds, even though that actual difference is extremely tiny and is not a meaningful difference between the two ponds. My point is simply that the fact that a study found a statistically significant result does not automatically mean that they found a meaningful result, so you should take a good hard look at their data before drawing any conclusions.

What do you do with a non-significant result?
The question of what to do with non-significant results is a complicated one, and it is probably the area where most people mess up. For various reasons (some of which I discussed above) you’re never supposed to accept the null hypothesis, rather, you fail to reject it. In other words, you simply say that you did not detect a difference rather than saying that there is no difference (in reality there is nearly always a difference, it just might not be a meaningful one). In practice, however, there are many situations in which you have to act as if you are accepting the null hypothesis. For example, let’s say that you are comparing two methods, one of which is well established but expensive, while the other is untried and cheap. You do a large study and you don’t find any significant differences between those two methods. As a result, scientists will begin using the cheap method and they will cite your paper as evidence that it is just as good as the expensive method.

Drug trials present a similar dilemma. Let’s say that we are trying a new drug and we find that there are no significant differences in side effect X between people who take it and don’t take it. The FDA, doctors, and general public will treat that result as, “the drug does not cause X” which is essentially accepting the null.

So how do we solve this problem? Do all drug trials violate a basic rule of statistics? No, the key here is sample size and statistical power. Remember from the section above that nearly all real population means will be different, but the difference may be very slight and not meaningful. So, when we accept that a novel method is as effective as the old, for example, we aren’t actually saying that there are no differences between the two. Rather, we are saying that there are probably no differences at the effect size that we were testing. To put this another way, we would say that the current evidence does not support the conclusion that they are different.

This may seem confusing, but you can think of it like a jury decision. We don’t declare someone “guilty” or “innocent.” Rather, we declare them “guilt” or “not guilty.” The “not guilty” verdict is essentially the same thing as failing to reject the null. It doesn’t mean that the person definitely didn’t commit the crime. Rather, it simply means that we do not have the evidence to conclude that they did commit the crime; therefore, we are going to treat them as if they didn’t.

Jumping back to science, ideally you should do something called a power analysis. This shows you what size difference you would be able to detect given your sample sizes, variance, and the level of power that you are interested in. So, let’s say that during the methods comparison test, anything less than a 0.01 difference between the two methods would be good enough to consider the new one reliable, and you had the statistical power to detect a difference of 0.001. That would mean that although there may be some very tiny difference between the methods, that difference is less than a difference that you would care about, and you had the power to detect meaningful differences. Similarly, if you are doing a drug trial and you have the power to detect a side effect rate of in 1 out of every 10,000 people, then you cannot conclude that the drug doesn’t cause that side effect, but you can say that if it does cause that side effect, it probably does so at a rate of less than 1 in 10,000 (note [added1-1-16]: as a general rule, power analysis should be done before conducting the study in order to determine what sample size will be necessary to detect a desired effect size).

All of this connects back to the importance of sample size. If you have a small sample size, then you won’t be able to detect small differences. Let’s say, for example, that a drug trial found improvements in 40 out of 100 people in the control group and 50 out of 100 people in the experimental group. That would result in a P value of  0.2008, which is not statistically significant, but that test would not have much power. As a consequence, that result is not very helpful. It could be that the drug simply doesn’t work, but it could also be that it does have an important effect, and this study just didn’t have a big enough sample size to detect it. Therefore, I am personally very hesitant to use results like this as evidence one way or other, and I think that when you have results like this, it is best to wait for more evidence before you try to say that there are no meaningful differences.

Some people, however, fall for the opposite pitfall. On several occasions, I have encountered people who look at studies with small samples sizes like this and say, “there isn’t enough power to actually test for differences, therefore we should just go with the raw numbers and assume that there is a difference.” This is completely, 100% invalid. Think back to the very start of this post, the whole reason that we do statistics is because without them we can’t tell if a result is real or just the result of chance variation. So you absolutely cannot blindly assume that the difference is real. If you don’t have enough power to do a meaningful test, then you simply cannot draw any conclusions.

Conclusion
This post ended up being quite a bit longer than planned, so let me try to sum things up with a bullet list of key points (note: for simplicity, I will talk about means, but the same is true for proportions, correlations, etc. so you can replace “no difference between he means” with “no relationship between the variables,” “no difference between the proportions,” etc.).

  • In science, you nearly always sample subsets of the total groups that you are interested in (these are sample means)
  • The means of those subsets will nearly always be different, but you actually want to know whether or not the means of the entire groups are different (these are the population means)
  • You have two hypotheses. The null says that there is no difference between the population means, and the alternative says that there is a difference between the population means
  • The P value is the probability of getting a result where the difference between the sample means is equal to or greater than the difference that you observed, if the null hypothesis is actually true
  • The larger your sample size, the greater your ability to detect differences
  • If you get a statistically significant result, you reject the null hypothesis; whereas, if you don’t get a significant result, you fail to reject the null hypothesis
  • Statistical significance is not the same as biological significance
  • Nearly all population means will be different from each other, but that difference may not be meaningful. Therefore, you cannot conclude that no difference exists, but you can provide evidence that if a difference exists, it is very small (or at least smaller than the effect size that you were testing)

Other posts on statistics:

 



 


 

							

	
			

													
						Posted in Nature of Science					
					|
				
								
					Tagged , 				
				|
				
				3 Comments

							

		


		
	

	
			

			
5 bad arguments against the influenza vaccine


			

				Posted on December 21, 2015 by Fallacy Man			


					

				
influenza, vaccine, anti-vaccer, meme, deaths, deadly disease, minor illness I spend a lot of time on this blog debunking bad anti-vaccine arguments (for example here and here). Nevertheless, logically invalid anti-vaxxer nonsense continues to rear its ugly head. Therefore, in this post I am going to focus specifically on five seriously flawed, yet amazingly common, arguments against the flu vaccine.


Bad argument #1: The flu isn’t a deadly disease

This is probably the most common argument that I hear against the influenza vaccine. It’s also a complete load of crap. People often describe the flu as, “a minor illness” or “just a bad case of the sniffles.” In reality, however, the flu kills roughly 1,000-49,000 people every year in the US alone (mean = 6,309, median = 5,128; Thompson et al. 2010), and globally, it kills 250,000-500,000 people annually (WHO 2014). A disease with a six digit death toll simply cannot be described as a “minor illness.” To be clear, this is not fear mongering. It is a simple fact that the flu kills hundreds of thousands of people annually. Therefore, it is, by definition, a deadly disease.


On several occasions, I have encountered anti-vaccers who respond to these mortality rates by claiming that influenza isn’t actually the killer. They point out that most of the people who “die from the flu” actually died from secondary complications like pneumonia. Technically speaking, this is true, but those secondary complications occurred because of the flu. This response is no different from describing the death of a gunshot victim by saying, “the bullet didn’t kill him, it was the loss of blood.” Fine, perhaps the loss of blood was the proximate cause of death, but the patient lost the blood because of the bullet. The patient would not have died at that particular time if he hadn’t been shot. Even so, yes, pneumonia, heart failure, etc. are often the proximate causes of death in influenza patients, but those problems arise because of the flu and result in the deaths of people who probably would not have died at that particular time if they hadn’t gotten influenza. So if you are going to say that the flu isn’t deadly because it simply causes secondary problems rather than directly killing its victims, you are also going to have to make a lot of other rather bizarre claims. For example, you’re going to have to say that smoking isn’t deadly, because it’s the lung cancer that actually kills people. Similarly, jumping off of a sky skyscraper isn’t deadly, because it’s the impact with the pavement that actually kills you. HIV is also not deadly, because it’s the secondary infections that actually kill you. These are necessary outcomes of using this absurd line of reasoning.


Bad argument #2: Influenza isn’t serious in healthy teenagers/adults

This argument is really a continuation of bad argument #1, but it is common and important enough that I decided to treat it separately. It is an extremely frequent response to the enormous death tolls from influenza, and it’s basically just the classic, “it won’t happen to me” response that anti-vaccers so dearly love.


This argument usually goes as follows, “Influenza is only serious/deadly in the elderly, infants, and people with certain medical conditions, but I am a healthy adult; therefore, I don’t need to be worried.” First, it is true that the death rates are highest in those high-risk groups, but that does not mean that no healthy individuals ever die from it. Further, the list of people who are at a high risk of serious complications is quite extensive. The WHO says the following:


“Yearly influenza epidemics can seriously affect all populations, but the highest risk of complications occur among children younger than age 2 years, adults aged 65 years or older, pregnant women, and people of any age with certain medical conditions, such as chronic heart, lung, kidney, liver, blood or metabolic diseases (such as diabetes), or weakened immune systems.”


I’m not sure about you, but I know quite a few people who fit one of those categories, which brings me to a very important point: vaccination is about more than just your personal safety. Let’s assume for the sake of debate that as a healthy adult, you are impervious to the serious consequences of the flu. That doesn’t change the fact that you can act as a vector and spread the flu to people who are in the high-risk categories. Anti-vaccers like to claim that herd immunity is a myth, but it’s actually a well established fact. Numerous studies have experimental confirmed that it works (Monto et al. 1970; Rudenko et al. 1993; Hurwitz et al. 2000; Reichert et al. 2001; Ramsay et al. 2003), and it’s really just a simple mathematical concept that is easy to simulate. So by getting vaccinated, you are helping to keep people in the high-risk categories safe. Anti-vaccers often act as if vaccines are a matter of personal freedom, but they are actually a matter of public responsibility. Even if you are not personally in a high-risk category, you should get vaccinated for the same reason that you shouldn’t drive drunk: your actions affect other people. It is extremely easy for an otherwise healthy adult to accidentally infect a nursing home, nursery, relative who is pregnant, someone who is fighting cancer, etc. Your action (or inaction) can have dire consequences on those people.


Finally, although healthy adults generally do not experience the worst symptoms, it’s still a miserable disease. It is not just, “a bad case of the sniffles.” For most people, it is several days of fever, cramps, vomiting, and feeling like utter crap. One study estimated that each year in the US, influenza results in 3.1 million days of hospitalization and 31.4 million outpatient visits (Molinari et al. 2007). Further, a study which specifically tested the effectiveness of the vaccine on healthy, working adults found that the vaccine significantly reduced instances of respiratory infections, days of missed work, and visits to a physician (Nichol et al. 1995).  To be fair, a different study (Bridges et al. 2000) did find that the effectiveness of the vaccine at preventing missed work days depended on how closely the vaccine matches the circulating viral strain, but that is not a valid reason for avoiding the vaccine (see bad argument #3).


In short, yes, the flu probably wouldn’t be life threatening for a healthy adult like me, but the vaccine is extremely safe and will only result in a sore arm and perhaps a day of feeling slightly unwell, and it will help to ensure that I don’t infect people who are at-risk, and it will help me avoid spending a week curled up in a ball, hugging the toilet, and wishing for the sweet release of death. Deadly or not, I’d much rather have a day with a sore arm than a week of abject misery.


Bad argument #3: The vaccine was only 23% effective in 2014-2015, so what’s the point?

There are several important things to note here. First, the effectiveness of the vaccine varies from one year to the next. The 2014-2015 season was a particularly bad one, but the effectiveness is often much higher. There are several reasons why the flu vaccine isn’t nearly as effective as most vaccines. A big part of it is due to the fact that the flu strain changes from one year to the next, and it’s impossible to vaccinate against all of the strains. So the strain that was active last year may not be the dominant strain this year. Thus, vaccine engineers do their best to predict which strains will be circulating in a given year, and they design the vaccine accordingly, but if their predictions are wrong, then the vaccine may not be particularly effective (you can find more on how this works here).


A second reason for low effectiveness is low herd immunity due to poor vaccination rates (see bad argument #2). Even when vaccines cause the body to produce the correct antibodies, they don’t make you impervious to the disease. They make you resistant to a minor dose (a dose that still would have caused an infection without the vaccine), but if you are constantly exposed to large doses of the pathogen, it’s going to be more than you’re circulating antibodies can handle. So, vaccines are most effective when most people are vaccinated, and by not vaccinating, you are actually making the vaccine less effective for everyone else.


Finally, even if 23% effectiveness was the norm, it would still be a good idea to get vaccinated, because your odds of getting influenza would still be reduced. Do you know what’s worse than 23% effectiveness? O% effectiveness, and that’s what you get without the vaccine. Let me put it this way, if seat belts were only 23% effective, would it still be a good idea to wear them? Yes, of course it would, because they lower your odds of having a serious injury in a car accident. Even so, even if the vaccine only prevented a few illnesses and deaths each year, it would still be a good thing because it would still save lives and prevent unnecessary suffering.


Bad argument #4: I’ve never been vaccinated and never had the flu/my uncle was vaccinated and still got the flu

Anecdotes spew forth from the mouths of anti-vaccers like water from Niagara Falls. The problem is, of course, than anecdotes are totally worthless in situations like this (see the comments).  I’m sure that you know someone who received the vaccine and still got the flu, as well as someone who didn’t receive the vaccine and didn’t get the flu, but I am equally certain that you also know people who received the vaccine and didn’t get the flu and people who didn’t receive the vaccine and did get the flu. You and I can exchange anecdotes all day and never get anywhere because anecdotes are meaningless. We need proper controls and knowledge of the actual disease rates, not scattered observations. Actual studies, of course, show that influenza rates are lower among the vaccinated (reviewed in Osterholm et al. 2012). Yes, the influenza vaccine is not the most effective in the world (see bad argument #3), and we should definitely be trying to improve it, but in the meantime, some protection is still better than no protection, and anecdotes do absolutely nothing to defeat that fact.


Bad argument #5: You can get the flu from the flu vaccine

No you freaking can’t. The flu vaccine contains either a deactivated virus or no virus at all. It is not biologically possible for you to get the flu from the vaccine because the virus has been shut off. When people say that the vaccine can give you the flu, they are literally proposing a zombie scenario in which something reanimates. We live in the real world, not the Walking Dead, and in the real world, the flu vaccine simply cannot give you the flu.


Note: many people refer to the flu vaccine as having a “killed” or “dead” virus. I personally don’t like that terminology because technically viruses aren’t alive to begin with, and, therefore, can’t be “killed.” However, the fundamental meaning of those terms still applies. It’s like talking about a “dead” battery. Technically speaking, it’s not “dead” because it was never alive, but it’s still totally non-functional and inert. Even so, the virus in the vaccine isn’t technically “dead” because it was never alive, but it’s still non-functional and can’t infect you.


Conclusion

Is the flu vaccine a magic cure all that guarantees that you will never get sick? No, of course not. Is the flu a violent plague that threatens to wipe out humanity? No, but it is undeniable that the flu is a serious disease which kills hundreds of thousands of people each year and causes unfathomable amounts of suffering. It is also undeniable that getting the flu vaccine reduces your chance of getting the flu. So yes, you will probably live without the vaccine, and yes, the vaccine does not guarantee that you won’t get the flu, but when the cost is a few bucks and a sore arm, what do you have to lose? Serious reactions to the flu vaccine are almost unheard of, and getting the vaccine lowers your chance of getting the disease, and it builds herd immunity. Thus, it also helps to protect you and the people who are at a high risk of death or serious complications from the disease. So please, stop reading Natural News, Mercola.com, and other pseudoscience websites and go get the vaccine.


Note: I’m sure that any anti-vaccers reading this will take issue with my claim that serious side effects are extremely rare, so let me curtail your inevitable anecdotes by reminding you that the fact that event A happened before event B does not mean that event A caused event B (that’s a logically fallacy known as post hoc ergo propter hoc). So please, don’t waste my time with your logically invalid anecdotes because they are meaningless and I don’t give a crap about them. Find me a properly controlled, peer-reviewed study with a large sample size, and then we’ll talk.


Literature cited


Bridges et al. 2000. Effectiveness and cost-benefit of influenza vaccination of healthy working adults. Journal of the American Medical Association 284:1655–1663.


Hurwitz et al. 2000. Effectiveness of influenza vaccination of day care children in reducing influenza-related morbidity among household contacts. 284: 1677–1682.


Molinari et al. 2007. The annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine 25:5086–5096.


Monto et al. 1970. Modification of an outbreak of influenza in Tecumseh, Michigan by vaccination of schoolchildren. Journal of Infectious Diseases 122:16–25.


Nichol et al. 1995. The effectiveness of vaccination against influenza in healthy working adults. New England Journal of Medicine 333:889–893.


Osterholm et al. 2012. Efficacy and effectiveness of influenza vaccines: a systematic review and meta-analysis. Lancet Infectious Diseases 12:36–44.


Ramsay et al. 2003. Herd immunity from meningococcal serogroup C conjugate vaccination in England: database analysis. BMJ 7385: 365–366.


Reichert et al. 2001. The Japanese experience with vaccinating schoolchildren against influenza. New England Journal of Medicine 344: 889–896.


Rudenko, et al. 1993. Efficacy of live attenuated and inactivated influenza vaccines in schoolchildren and their unvaccinated contacts in Novgorod, Russia. Journal of Infectious Diseases 168: 881–887.


Thompson et al. 2010. Updated estimates of mortality associated with seasonal influenza through the 2006-2007 influenza season. MMWR 59: 1057–1062.


WHO. 2014. Influenza (Seasonal). Fact Sheet N°211.


 

							

	
			

													
						Posted in Vaccines/Alternative Medicine					
					|
				
								
					Tagged , , 				
				|
				
				5 Comments

							

		


		
	

	
			

			
Do we need more studies on vaccines, GMOs, climate change, etc.?


			

				Posted on December 15, 2015 by Fallacy Man			


					

				
anti-vaccers want more studies but refuse to accept studies memeI frequently encounter people who state that, “I’m not anti-vaccine/GMO, I just think that we need more studies” or “we need more research before we take major action on climate change.” I have, however, noticed that whenever someone declares, “I’m not X” they usually end the statement with some pathetic justification for why they are in fact X, and that is definitely the case in this situation. The cry for more studies on vaccines, GMOs, etc. is nearly always hypocritical and stems from a willful ignorance about just how many studies there actually are. The reality is that topics like vaccines have been so well studied that they have achieved the status of settled science (please actually read the linked post before berating me over that term). So, the problem isn’t that there aren’t enough studies; rather, the problem is that people refuse to read or accept the hundreds of studies that we already have. To be fair, I have occasionally encountered people who asked for more studies out of honest ignorance rather than willful ignorance, and those people quickly retracted their statements once I directed them to the veritable mountain of published literature. That type of ignorance is fine. There is nothing wrong with not knowing something, then updating your view when presented with evidence, but in my experience, those people represent a tiny minority, and most of the people who demand more studies are doing so out of willful ignorance.


Note: Before I present some examples of this flawed excuse for denialism, I want to make it absolutely clear that I am not suggesting that we should be doing less research or should spend less funding on science. I am a scientist, and like most scientists, I spend a minimum of 60 hours a week collecting and analyzing data, writing papers, etc. So obviously I place a high value on scientific research, and I think that we need more of it. The amount that most governments invest in research is pathetic (granted, I’m clearly not objective on that issue). So, I’m not saying that we need less research, but I am saying that there are certain topics that have been so well studied that we should move on and focus our efforts on real questions.


The autism scare is the perfect embodiment of this situation. Anti-vaccine parents continue to insist that we need more research on the link between vaccines and autism, when the reality is that there is no link. Study, after study, after study, after study, after study has failed to find any link between vaccines and autism. In fact, we have dozens of these studies including a massive meta-analysis with over 1.2 million children which failed to find any significant difference in autism rates between the vaccinated and unvaccinated. That is one of the largest sample sizes that I have ever seen, and as a general rule, the larger the sample size, the more certain you can be of the result (I explained the evidence in more detail here).


Now, if you’re an anti-vaccer, you’re probably thinking, “fine, there are studies, but they were all paid for by Big Pharma,” but you’re wrong. Many of those studies (including the meta-analysis) had no conflicts of interest. Further, even a recent study that was funded by an anti-vaccine group failed to find any evidence that vaccines cause autism. Finally, you have to evaluate each paper individually and present actual evidence that they are biased/flawed. You can’t just blindly accuse all of them of being bought off.



In the spirit of openness and honesty, I will acknowledge that if you dig around, you can find a few studies which have suggested that vaccines cause autism, but these are nearly always poorly designed studies that are riddled with problems. Further, most of them are correlation studies that cannot establish causation,  and the few studies that actually made proper comparisons all had tiny sample sizes (you can find details about most of the papers here and here). Sometimes you get false results, just by chance, but the odds of that happening are much lower with larger sample sizes, and when numerous large studies all agree with each other (as is the case for vaccines) it would be absurd to reject them in favor of a handful of tiny, methodologically flawed studies.


My point in all of this is simply that the supposed link between vaccines and autism has been studied so many times by so many researchers that we are extremely confident that the link is imaginary and we should move on. Nevertheless, anti-vaccers continue to insist that we need more studies; therefore, every year more studies on vaccines and autism are conducted, but that’s absurd! We know that vaccines don’t cause autism, but we don’t fully understand what actually causes it (though genetics seem to be important), nor do we know how to cure it or even effectively treat it, not alone prevent it. That is what we should be studying. We should be trying to understand its real cause, and we should be looking for ways to actually help the people that have it rather than pouring money down the toilet looking for an answer that we already have, especially when the group that is demanding the studies is never going to accept the results of those studies. Are we really naive enough to think that study number 100 will convince them when the past 99 haven’t? If a sample size of 1.2 million isn’t enough to persuade you, than nothing will ever be good enough for you. That is why the claim that we need more studies is nearly always hypocritical. You can’t sincerely claim that there aren’t enough studies while simultaneously willfully ignoring all of the studies that we actually have.


Note: I want to be clear that there are many researchers studying the real causes of autism and potential treatments, but my point is that every year money and countless man hours get spent doing yet another study on vaccines and autism, and that time and money would be better spent elsewhere. 


If we expand the situation beyond autism, we find the same pattern across vaccine issues. Anti-vaccers ardently insist that there aren’t enough studies despite the fact that there are literally thousands of studies. We’ve looked for relationships between vaccines and SIDS, (Hoffman et al. 1987; Griffin et al. 1988; Mitchell et al. 1995; Fleming et al. 2001; Vennemann et al. 2007a; Vennemann et al. 2007b), asthma (Kramarz et al. 2000; Offit and Hackett 2003; Grabenhenrich et al. 2014), allergies (Koppen et al. 2004), general health (Schmitz et al. 2011), etc. You name it, we’ve done it. Vaccines are probably the most well studied topic in medical history, and if you claim that they haven’t been well studied, you are simply displaying your own ignorance.


To be clear, I obviously think that any new vaccines should be rigorously tested before being released to the general public (which they are, btw), and I have no problems with doing research on any novel concerns that arise if that there is good justification for them. However, it is a pointless waste of time and money to continue to study topics for which we already have very clear and well established answers. Further, you absolutely cannot justify opposing vaccines on the basis of a lack of studies because in reality, there is a plethora of studies.


Moving beyond vaccines, we find the exact same story for GMOs. In fact, I personally encounter this argument more often for GMOs than for vaccines. People tell me all the time that we need more research before eating or growing GMOs, but is it really true that they haven’t been properly studied? If you are one of the people who thinks that it is, then let me ask you this: how many studies would be enough to convince you? A few dozen? A few hundred? How about 1,700? Would that be enough? Because we have well over that. This review from 2013 examined 1,783 studies and failed to find any evidence that GMOs are dangerous for us or the environment (Nicolia et al. 2013). Say it with me: there are over 1,700 studies on the safety and environmental impacts of GMOs. Further, several hundred more studies have been conducted since that review, and they have consistently found that GMOs are no worse for us or the environment than traditional crops (in some cases they are better). So please, don’t sit there telling me that we need more studies unless you can also give me a logically and scientifically valid reason why you reject all of the 1,700+ studies that we have already done (you should also check out this massive review looking at the health of livestock before and after the introduction of GMO feed). Also, just to be clear, over half of those studies have no ties (financial or otherwise) to agricultural companies (details and sources here).


Finally, let’s look at climate change. On this topic, people are prone to claim that we don’t have enough evidence to warrant action, but that’s once again absurd. We have extremely clear evidence that we are causing it, and thousands of papers from numerous fields of study have confirmed the results. We carefully tested all of the known drivers of climate change, and no combination of natural factors can explain the current warming. The only way to explain the warming is to include our greenhouse gas emissions in the analyses (Meehl, et al. 2004; Wild et al. 2007; Lockwood and Frohlich 2007, 2008; Lean and Rind 2008; Foster and Rahmstorf 2011; Imbers et al. 2014). More details and sources here,  here, and here.


To be fair, it is true that we don’t know exactly how much we are causing it to change, or exactly what will happen in the future, and we should continue to study those topics; however, we have plenty of data to be extremely confident that our actions are causing the bulk of the current climate change, the changes are already having negative impacts for us, and the changes will continue unless we modify our behavior. On those key points, there is no serious debate among scientists. So the claim that we should wait for more data before we take action is misguided and dangerous. Further, in my experience, this claim is nearly always made by people who completely deny climate change and will never accept the results of any study that opposes their preconceived views. So once again, the claim is disingenuous.


But what if…


At this point, inevitably, someone is getting ready to make a comment along the lines of, “but what if there are things that we missed? What if there is a problem with GMOs, vaccines, etc. that we haven’t found yet? Science isn’t perfect and it is arrogant to think that we know everything.”


I agree that science isn’t perfect, and if scientists actually claimed to know everything, I would agree that they were being arrogant, but they don’t claim to know everything. Rather, they simply make conclusions based on all available evidence, and when that body of evidence is extensive, those conclusions can be reached with a high degree of confidence. Look, it is always possible that we missed something. This is true for absolutely any technology that has ever been tested. As a result, it would always be possible to make this argument. No matter how many studies we’ve conducted, it is still possible that we missed something. Therefore, the only rational approach is to study something up to the point that we are as confident as we ever could be in the conclusion, and for things like the safety of vaccines and genetic engineering, we’re there. In fact, we crossed that line long ago.


Let me ask you this, if around 2,000 studies on GMOs isn’t enough, then what would be? How many studies does it take? There has to come a point at which you acknowledge that we have studied a topic so thoroughly that it is exceedingly unlikely that our conclusions are wrong, and if you ask me, that line occurs long before 2,000 studies.


This is especially true for things like vaccines and GMOs where the known benefits are great. We know, for example, that vaccines save millions of lives each year. So given that known benefit, it makes absolutely no sense to oppose them out of the slight possibility that we’ve missed some unknown danger. For things like climate change, the same concept applies, but the situation is somewhat reversed. In other words, all available evidence shows that the consequences of not acting will be dire, so saying that we shouldn’t take action because of the extremely slight chance that we are wrong is incredibly foolhardy.


Conclusion


In summary, do we need more studies on vaccines, GMOs, climate change, etc.? Yes, of course we do, but we need to be studying the things that are actually unknown rather than pandering to people who will never accept any study that disagrees with their biases and preconceptions. Replication is certainly important in science, and we should try to replicate any important results, but once a result has been consistently corroborated over and over again by numerous studies, we should move on. We should be focusing on how to improve vaccines and make vaccines for more diseases rather than producing yet another study on vaccine and autism. We need more research on making GMOs that provide vitamins and economic benefits to developing countries, and less research on whether or not the fundamental technology is safe. We should continue to study the climate, but we shouldn’t wait for future studies before taking action. In short, we should be studying new and marvelous things rather than repeating something that we have already done hundreds of times in the vain hope that people are actually reasonable and will be willing to change their views when presented with one more study.


Literature Cited


    

  • Anders et al. 2003. Association between thimerosal-containing vaccine and autism. JAMA 290:1763–1766.
    • 

  • Anders et al. 2004. Thimerosal exposure in infants and developmental disorders: a retrospective cohort study in the United Kingdom does not support a causal association. Pediatrics 114:584–591.
    • 

  • Destefano et al. 2004. Age at first measles-mumps-rubella vaccination in children with autism and school-matched control subjects: a population-based study in metropolitan Atlanta. Pediatrics 113:259–266.
    • 

  • Fleming et al. 2001. The UK accelerated immunization programme and sudden unexpected death in infancy: case-control study. BMJ 322:822.
    • 

  • Foster and Rahmstorf 2011. Global temperature evolution 1979–2010. Environmental Research Letters 7:011002.
    • 

  • Gadad et al. 2015. Administration of thimerosal-containing vaccines to infant rhesus macaques does not result in autism-like behavior or neuropathology. Proceedings of the National Academy of Science 112:12498–12503.
    • 

  • Grabenhenrich et al. 2014. Early-life determinants of asthma from birth to age 20 years: a German birth cohort study. Journal of Allergy and Clinical Immunology 133:979–988.
    • 

  • Griffin et al. 1988. Risk of sudden infant death syndrome after immunization with the Diphtheria–Tetanus–Pertussis vaccine. New England Journal of Medicine 319:618–623.
    • 

  • Hoffman et al. 1987. Diphtheria-Tetanus-Pertussis immunization and sudden infant death: results of the national institute of child health and human development cooperative epidemiological study of sudden infant death syndrome risk factors. Pediatrics 79:598–611.
    • 

  • Imbers et al. 2014. Sensitivity of climate change detection and attribution to the characterization of internal climate variability. Journal of Climate 27:3477–3491.
    • 

  • Jain et al. 2015. Autism occurrence by MMR vaccine status among US children with older siblings with and without autism. Journal of the American Medical Association 313:1534–1540.
    • 

  • Koppen at al. 2004. No epidemiological evidence for infant vaccinations to cause allergic disease. Vaccine 22:3375–3385.
    • 

  • Kramarz et al. 2000. Does influenza vaccination exacerbate asthma? Archives of Family Medicine 9:617–923.
    • 

  • Lean and Rind. 2008. How natural and anthropogenic influences alter global and regional surface temperatures: 1889 to 2006. Geophysical Research Letters 35:L18701.
    • 

  • Lockwood and Frohlich. 2007. Recently oppositely directed trends in solar climate forcings and the global mean surface air temperature. Proceedings of the National Academy of Sciences 463:2447–2460.
    • 

  • Lockwood and Frohlich. 2008. Recently oppositely directed trends in solar climate forcings and the global mean surface air temperature. II. Different reconstructions of the total solar irradiance variation and dependence on response time scale. Proceedings of the National Academy of Sciences 464:1367–1385.
    • 

  • Madsen et al. 2002. A population-based study of measles, mumps, and rubella vaccination and autism. New England Journal of Medicine 347:1477–1482.
    • 

  • Meehl, et al. 2004. Combinations of natural and anthropogenic forcings in the twentieth-century climate. Journal of Climate 17:3721–3727.
    • 

  • Mitchell et al. 1995. Immunisation and the sudden infant death syndrome. Archives of Disease in Childhood 73:498–501.
    • 

  • Nicolia et al. 2013. An overview of the last 10 years of genetically engineering research. Critical Reviews in Biotechnology 34:77–88.
    • 

  • Offit and Hackett. 2000. Addressing parents’ concerns: do vaccines cause allergic or autoimmune diseases? Pediatrics 111:653–659
    • 

  • Schmitz et al. 2011. Vaccination status and health in children and adolescents. Medicine 108:99–104.
    • 

  • Patz et al. 2005. Impact of regional climate change on human health. Nature 438:310–317.
    • 

  • Taylor et al. 2009. Autism and measles, mumps, and rubella vaccine: no epidemiological evidence for a causal association. Lancet 353: 2026–2029.
    • 

  • Taylor et al. 2014. Vaccines are not associated with autism: and evidence-based meta-analysis of case-control and cohort studies. Elsevier 32:3623–3629.
    • 

  • Van Eenennaam and Young. 2014. Prevalence and impacts of genetically engineered feedstuffs on livestock populations. Journal of Animal Science 92:4255–4278.
    • 

  • Vennemann et al. 2007a. Sudden infant death syndrome: No increased risk after immunisation. Vaccine 25:336–340.
    • 

  • Vennemann et al. 2007b. Do immunisations reduce the risk of SIDS? A meta-analysis. Vaccine 26:4875–4879.
    • 

  • Wild et al. 2007. Impact of global dimming and brightening on global warming. Geophysical Research Letters
    • 


							

	
			

													
						Posted in Global Warming, GMO, Vaccines/Alternative Medicine					
					|
				
								
					Tagged , , , , , 				
				|
				
				8 Comments

							

		


		
	

	
			

			
What movie theories teach us about science vs. pseudoscience


			

				Posted on December 7, 2015 by Fallacy Man			


					

				
Movie theories are lots of fun. I thoroughly enjoy to contemplating and debating novel ideas like the notion that all Pixar movies are connected or that the Joker was actually the hero of Dark Knight, but in addition to being fun, I think that movie theories provide an excellent illustration of the demarcation between science and pseudoscience. Therefore, I am going to use them to try to explain one of the key differences between the two, and by so doing, I will provide you with a vital tool for identifying pseudoscience as well as simultaneously illustrating why science is such a powerful method for understanding our universe. I will conclude this post by applying the lessons from movie theories to one of the prominent pseudosciences of our day: creationism (everything that I am going to talk about also applies to ghost hunters, UFO spotters, Big Foot believers, and just about every other pseudoscience position you can think of).


jar jarThe idea for using movie theories as an illustration for pseudoscience occurred to me while reading the viral theory that Jar Jar Binks was actually the ultimate villain of Star Wars episodes I-III. Therefore, I will use it as my model throughout this post. If you don’t feel like reading the entire theory, it simply argues that Jar Jar was actually a powerful force user who was only playing the part of the fool in order to execute his master plan, and he was at the very least collaborating with Palpatine, and possibly even Palpatine’s master. For this post to make sense, you will probably need to have seen the Star Wars movies, but the vast majority of people have so that shouldn’t be a problem. If you haven’t seen them, what on earth is wrong with you? Go watch them right now, it’s more important than reading this blog post.


Note: when I say “movie theory” I am referring to the alternative explanations that are proposed after a film has been released. I am not referring to guesses about what will be in a movie that has yet to come out.


Use of the word “theory”

I want to briefly point out that movie “theories” are not theories in the scientific sense of the word. In science, a theory is an explanatory framework that has been rigorously tested and has been shown to have an extremely high predictive power (I’ll elaborate on that later). Movie “theories” by contrast are just explanations. There is no testing nor do they make predictions. Therefore, although I will continue to call them “movie theories” I want to be explicitly clear that they are not actually theories in the scientific use of the word.


Movie theories explain facts

The core distinction between science and the type of pseudoscience that is displayed in movie theories is the order in which knowledge is acquired and dealt with. Movie theories are inherently retrospective. People make them after a movie has been released and after all of the data are available. In other words, all that they do is explain the existing facts. In contrast, real science uses the existing data to make predictions about future data (more on that later).


At a quick glance, the ability to explain facts may seem like a good quality, but in isolation it is actually extremely problematic because there are often multiple ways to explain the same facts. For example, the Jar Jar theory explains Jar Jar’s jumping abilities by claiming that he is a force user, but when I watch him do a massive somersault, I explain that fact as simply being part of Gungans’ natural athletic abilities. After all, Gungans seem more closely related to amphibians than anything else, and amphibians are known for jumping abilities. Additionally, there are other non-Jedi species (like Wookies) who have extreme physical abilities.


Similarly, the Jar Jar theory proposes that Jar Jar’s seemingly clumsy moves are actually a form of martial art known as Zui Quan (aka Drunken Fist Wushu); whereas, I think that Lucas simply made a bad call and wrote an awkward, annoying character. The Jar Jar theory can support its position by trying to retroactively match Jar Jar’s moves with specific moves from Zui Quan, and I can support my position by citing numerous other movies that have had a bumbling idiot who accidentally saves the day. I can also bring up Lucas’s other recent blunders (dare I mention Kingdom of the Crystal Skull?).


So, which view is actually correct? I don’t know and neither do you, that’s the problem. Both views can retroactively bolster their position and try to make the available data fit their model. Both positions can offer an explanation for the data, but neither position can demonstrate with a high level of confidence that the other explanation is wrong.


Retroactively explaining data in this fashion also has an additional problem. Namely, in movie theories and pseudoscience, evidence and explanations often become intertwined and confused. Let’s think about the possibility of Jar Jar using Zui Quan again for a moment. Is that evidence for the overarching theory that Jar Jar was actually a force user, or is the theory that he was a force user the explanation for his clumsy behavior? There’s no clear answer to that question, and that is a huge problem, because if your evidence is also your explanation, then you are running a massive risk of a circular reasoning fallacy. In other words, the view is self-reinforcing, but for a good theory, you really want external validation rather than internal support.


There is another problem that is closely related to the last point. Whenever you are retroactively applying an explanation, it is always tempting to stitch together seemingly arbitrary or disconnected facts in order to make them fit your view (conspiracy theories are excellent at this). For example, the Jar Jar theory makes a big deal out of the fact that Palpatine and Jar Jar are both from the same planet and therefore (according to the theory) likely knew each other. That explanation, however, seems like quite a stretch given how large a planet is and the fact that Gungans and humans clearly did not get along or interact with each other. In other words, the fact that they are both from the same planet is a rather minor point which gets conflated into a major topic in order to make it fit with/support the overarching theory.


In the process of overemphasizing minor points, movie theories also have a tendency to break Occam’s razor. For example, at one point, the side of the bridge from which Jar Jar is falling switches, and the theory proposes that he force jumped. The more parsimonious explanation, however, is that it was simply an editing mistake. The Star Wars movies are full of editing mistakes. Therefore, it seems odd to latch onto this one mistake and elevate it as evidence of the Jar Jar theory, but that is exactly what happens when you rely on an overarching explanation rather than falsifiable predictions. This way of thinking causes you to view everything as evidence for your position, which is why it is so dangerous.


yodaFinally, because movie theories are inherently retroactive explanations, it is always possible to explain any evidence that anyone else cites. For example, I could argue that this theory doesn’t work because Jar Jar never would have gotten involved were in not for his chance encounter with Qui-Gon Jinn, but someone who subscribes to this theory could then propose that Jar Jar was a powerful enough force user that he could actually see into the future and not only knew that Qui-Gon would hide on a ship, but also knew which ship he would be on. You could even go as far as citing other instances where Jedi saw future events, as well as Obi-Wan’s comment that, “in my experience there is no such thing as luck.” Technically speaking, that explanation would be boarding on an ad hoc fallacy, but movie theories are by their very nature already ad hoc. In other words, because they are made after all of the data have been collected, they are deliberately designed to account for all of the data. This makes them inherently impossible to defeat because it is always possible to continue making more retrospective explanations in order to justify your original theory. To be clear, those explanations may not be logically valid, but it is still always possible to make them, which actually makes it impossible to disprove the theory (i.e., you can demonstrate that the view is logically invalid, but that doesn’t prove that it’s wrong [that would be a fallacy fallacy]).


In short, movie theories are problematic because all that they do is explain existing facts. When you read them, it is always tempting to say, “this theory is great because it explains everything,” but as I’ll elaborate on in a minute, explanatory power can actually be an extraordinary weakness, rather than a strength.


Science predicts future data

Up until now, I have only been talking about movie theories, but all of the problems that I raised exist within actual views about the real world, and these are problems which scientists and philosophers have wrestled with for a long time. This is, in fact, the very issue that Karl Popper dealt with in his seminal work Science as Falsification. While examining the various “scientific” views of his day, he realized that some of them were extremely problematic. For example, two dominant views in the field of psychology were those of Freud and Adler. Both of them were massive explanations, and both of them conflicted with each other. What Popper astutely realized, however, was that there was no real way to tell which view was correct. No matter what patient came into a psychologist’s office, both schools of thought could give a plausible explanation for the patient’s behavior.


This lead Popper to conclude that explanatory power was not enough to make something a valid scientific view, and in isolation, explanatory power was actually problematic because a view which was designed to be able to explain everything would be inherently untestable. Therefore, Popper proposed that real science should be falsifiable. In other words, real science should make predictions about future data which, if they don’t come true, will falsify the view. Thus, a scientific theory is judged based on its predictive power. In other words, a good theory should make numerous falsifiable predictions, and all of them should come true. The theory of gravity is a good example of this. It predicts that anytime that you drop an object, it should fall. This is a falsifiable prediction because, if you dropped a pen and it floated in mid air, the theory of gravity would be falsified, and we would reject it; however, the fact that objects consistently fall means that gravity’s predictions consistently come true, which means that we can be very confident that it is correct.


Falsifiability is a stark contrast to the logic of movie theories. They generally do not make predictions because they are only designed to explain facts. Further, although they do sometimes make predictions about sequels, they are generally not falsifiable predictions. This is a crucial point: the prediction has to run the risk of falsifying the view. For example, the Jar Jar theory predicts that Jar Jar will appear in Episode VII (perhaps even as Supreme Leader Snoke), but does that make the view falsifiable? No, it doesn’t, because if Jar Jar isn’t there, it could simply be that Abrams decided to go another direction, or that Jar Jar has died in the intervening years (after all, we don’t know the normal lifespan of a Gungan). In other words, you could easily explain his absence without rejecting the theory. Therefore, the prediction is not falsifiable and does not provide an adequate test of the theory.


Applying falsifiability to creationism

In the final section of this post, I want to apply everything that I have been talking about to one of the most blatant and prevalent forms of pseudoscience: creationism. If you read the creationists’ literature, they actually fully acknowledge that they are explaining rather than predicting, but they don’t see it as a problem, and they incorrectly think that real scientists are doing it as well. They seem to be stuck in a pre-Popper era in which science is judged by explanatory power rather than predicting power. They frequently insist that scientists and creationists have the same evidence, but they are just interpreting it differently. Does that sound familiar? In their flawed view, creationism is one explanation and evolution is another, and both camps “interpret” the data to fit their explanation, but neither one can really be demonstrated to be better than the other (just as it is impossible to actually determine which theory of a movie is correct). The reality is, however, quite a bit different.


Evolution is falsifiable, whereas creationism is not (with possible exceptions concerning the flood). You see, evolution does not not simply retroactively “interpret” the data; rather, it predicts the data beforehand. As I have previously explained, evolution predicted that we should find intermediate fossils, and today we have hundreds of them that are exactly like what evolution predicted decades earlier. To be clear, that was a falsifiable prediction. Darwin himself even said that if we never found any intermediates, evolution would be discredited, but we did find them, and that is why evolution is so powerful. A theory which can predict the existence of organisms that we have yet to discover is utterly incredible. Creationism, in contrast, can’t do that. It very clearly predicted that intermediates shouldn’t exist, so, every time that we find them, it simply changes its tune and claims that “those aren’t actually intermediates, God just created them to look exactly like what we would expect intermediates to look like.” Do you see the problem? Creationism isn’t falsifiable because you can always fall back on the “God did it” argument. To be clear, that response isn’t logically valid (it’s an ad hoc fallacy), but it is technically possible. Thus, creationism can’t be falsified.


Further, the predictive power of evolution goes far beyond intermediates. For example, it also predicted that the fossil record would show a clear and orderly progression, it predicted that genetics would agree with the fossil record, and it predicted that biogeographic patterns would match the patterns seen in genetics and the fossil record. All of these are falsifiable predictions, and all of them are predictions that really should only come true if evolution is actually true. If the fossil record turned out to be jumbled, and modern mammals, dinosaurs, and Precambrian invertebrates were all found in the same layers, that would completely shatter evolution. Similarly, if the fossils said that birds and mammals evolved from reptiles, reptiles from amphibians, and amphibians from fish, but the genetics said that birds were most closely related to fish, and mammals were most closely related to amphibians, that would have been devastating to the theory of evolution. The fact that evolution got all of those predictions right, however, allows us to be extraordinarily confident in it. In contrast, creationists are left shrugging their shoulders and saying, “God did it that way.”


Think about the difference between those two for a minute. Evolution made a series of extraordinary and extremely risky predictions in totally different fields, and if any one of those predictions had failed, evolution would have been falsified. In contrast, creationism either made no predictions, or got its predictions wrong, but none of those predictions were falsifiable, so it simply changed its interpretation. Which one of those sounds like a robust and reliable way to understand our universe?


Finally, just like our movie theories, creationism has a long history of latching onto minor points and exalting them as proof of their position. For example, creationists are fond of claiming that dragons were actually dinosaurs and all of the legends of dragons are actually evidence that humans and dinosaurs lived together. Now, to anyone who wasn’t already convinced that creationism was true, that idea sounds laughably ridiculous. Ancient cultures are full of all sorts of legends that we don’t take literally, so why should we do so with dragons? Once again, this is the problem with applying a pre-existing explanation rather than making falsifiable predictions. If you have an explanation already in place, then you will view all of the evidence in a way that supports that explanation, even if that means making some truly imaginative leaps (technically, this argument is a question begging fallacy).


Conclusion

In summary, real science makes testable, falsifiable predictions. It does not simply retroactively apply a pre-existing explanation to the data. Rather, it predicts what the data should be before those data are collected. In contrast pseudoscience simply “interprets” data to fit its preconceived views, which often results in logical fallacies.


 

							

	
			

													
						Posted in Nature of Science					
					|
				
								
					Tagged , , , 				
				|
				
				7 Comments

							

		


		
	

	
			

			
12 bad reasons for rejecting scientific studies


			

				Posted on November 30, 2015 by Fallacy Man			


					

				
quote there is nothing wrong with asking questions but you have to be willing to accept the answers to those questions vaccine safety scientific studyA few days ago, I posted what I thought was a fairly innocuous image (right) onto my blog’s Facebook page. I was, however, sadly mistaken. My page was quickly flooded with comments by people who arrogantly insisted that there was nothing wrong with blindly rejecting all of the thousands of studies showing that vaccines are safe. I probably shouldn’t have been surprised by this, but still, I was astounded by the level of hubris and willful ignorance that was being so proudly displayed. What didn’t surprise me, however, were the attempts at justifying such a baffling position. They included all of the usual tropes about conspiracies, scientists being paid off, government corruption, etc. (I have included screen shots of some of the responses to the meme throughout this post). Most of these responses suffered the same fundamental problem. Namely, they assumed that there was something wrong with the studies rather than actually providing evidence that they were flawed. This is a very common mistake. When faced with a study that disagrees with their preconceptions, people often blindly assert that the authors were paid off, the data were manipulated, there’s a conspiracy afoot, etc., but unless they can actually prove that such unethical behavior occurred, that response is logically illegitimate and is no different from simply saying, “it’s wrong because I say it’s wrong” (in technical terms, it’s an ad hoc fallacy). You cannot assume that a study is flawed just because you disagree with it.


To be clear, I am not suggesting that you blindly accept every scientific paper. Unfortunately, not all papers are of a high quality, and bad, biased research does get published. So you should carefully examine all scientific studies, but you cannot blindly reject them just because they discredit your preconceived views. This is especially true for topics like vaccines for which we have thousands of studies that all agree with each other. It’s one thing to say that one paper is biased or one research group was paid off, but it is something else entirely to assert that essentially every paper on a topic is wrong and every researcher is corrupt.


In short, it is important to carefully examine scientific studies rather than blindly accepting them, but the inverse is also true (i.e., you have to carefully examine the study before rejecting it). I have previously written about good criteria for rejecting a paper; therefore, in this post I want to flip things and instead describe 12 bad arguments for rejecting a paper.




Bad reason #1: Galileo/Columbus


galileoWhen faced with results that they don’t like, many people will invoke Galileo or Columbus and claim that they defied the mainstream view and people thought that they were crazy, but they turned out to be right. I explained this one in detail here, but to be brief, no one thought that Galileo was crazy. He presented facts and careful observations, not conspiracies and conjecture. He did not blindly reject the science of his day, rather he made meticulous observations and presented data that discredited the common views. That is not in any way shape or form the same as arrogantly and ignorantly rejecting a paper just because you disagree with it.


Moving on to Columbus, the debate in his day was about the size of the earth, not its shape, and Columbus was dead wrong. In fact, his stubborn ignorance would probably have killed him and his crews if it hadn’t been for the fortunate “discovery” of the New World. You see, Columbus was the ideological equivalent of a modern day anti-vaccer. He “did his own research” and pompously declared that all of the experts were wrong about the size of the earth, and he thought that the well-accepted calculations were flawed. In reality, the accepted calculations of his day were very close to correct, while Columbus’s numbers were way off.




Bad reason #2: science has been wrong in the past 


Moving beyond the specific examples of Galileo and Columbus, other people often make the broad claim that science shouldn’t be trusted because it has been wrong before. This is another one that I have dealt with at length elsewhere, so I’ll be brief here. First, it is true that science has been wrong, but it has always been other scientists who have figured out that it was wrong. Further, it is logically invalid to blindly assume that it is wrong just because it has been wrong before.


Additionally, although there have been plenty of minor hypotheses which have been discredited, there have been very few core ideas that have been rejected in the past century. In other words, ideas which are supported by thousands of studies have rarely been rejected, and very few central ideas have been overthrown in recent decades. The closest example that you can find is probably Einstein’s theory of relativity replacing Newton’s law of gravity, but even in that example, Newton’s work wasn’t really wrong, it was just incomplete. Einstein didn’t completely throw Newton out the window; rather, he simply showed that Newton’s law doesn’t always work and doesn’t give us a complete picture. He built on what Newton had found.


Finally, attacking science by asserting that it has been wrong before is utterly absurd because science is inherently a process of modifying our understanding of the world. In other words, science is self correcting. This is one of it’s greatest strengths. To publish in science, you have to pass a rigorous peer-review process, which weeds out a ton of junk science. So, most of what gets published is of high quality. Further, when a bad paper gets published, it quickly comes under scrutiny by the rest of the scientific community, who will then point out errors in it (if they exist) and often try to replicate the results. As a consequence, it can be hard to get away with fraudulent science because if someone else tries to replicate your work, they are going to figure out that something was very wrong with your research (this is exactly what happened with Wakefield’s fraudulent paper suggesting that vaccines cause autism). Thus, science is self correcting and constantly replaces erroneous ideas as new evidence comes to light (the same can’t be said for anti-science views which rigidly cling to their positions no matter how much evidence opposes them). Therefore, the fact that science has been wrong is actually a good thing, because if there were no instances where we had discovered that a previous idea was wrong, that would mean that science hadn’t advanced.




Bad reason #3: it’s all about the money


fundingThis is probably the most common response to papers on climate change, vaccines, GMOs, etc., and it’s often simply untrue. The scientific community is massive, and there are thousands of independent scientists doing research. Further, all scientific publications require authors to declare any conflicts of interest, so you can actually check and see if a paper was paid for by a major company, and if you did that, you would find that many of the papers supporting GMOs, vaccines, etc. have no conflicts of interest. Anti-scientists, of course, have no interest in actually looking at the paper. They would rather just assume that it was paid off because that fits with their world-view. Further, even if 90% of the papers on a topic like vaccines had been paid off, that would still leaves us with hundreds of papers showing that they are safe and effective and essentially no papers saying that they are dangerous (you can find more details on the finances of vaccines, GMOs, and climate change here).


money
Based on this person’s follow up comments, everything except for the first sentence was sarcastic (i.e., they think that science does actually blindly reject answers we don’t like, etc.)


Finally, even if a paper does have a conflict of interest, that doesn’t give you carte blanche to ignore it. The fact that someone works for a pharmaceutical company, for example, does not automatically mean that they biased or falsified their data. If a paper has a conflict of interest, then you should certainly give it extra scrutiny, and you should be suspicious if it disagrees with other papers or has questionable statistics, but you cannot automatically assume that it is flawed.




Bad reason #4: there are other results that I disagree with


This is one of my favorites. Someone will say, “I reject the science of X because science also says Y and I disagree with Y.” We can rephrase this as, “I reject science because I reject science.” I would not, for example, accept water fluoridation as evidence that it’s ok to reject the science of vaccines unless I had already rejected the science of fluoridation. In other words, you have to justify your rejection of the science of Y before you can use it as evidence that we shouldn’t trust the science of X. Further, even if you could demonstrate that the science of Y (in this example fluoridation) was wrong, that still would not in any way shape or form prove that the science of X (in this example vaccines) is wrong. In fact, this entire line of reasoning is just a special case of the logical fallacy known as guilt by association. If are going to say that a scientific result is incorrect, you have to provide actual evidence that the specific result that you are talking about is incorrect.


other things I disagree with
I’m not sure which part of this post grieves me the most: their shoddy logic, their rejection of science, their bizarre capitalization, or the fact that someone “liked” it.




Bad reason #5: gut feelings/parental instincts


I encounter this one frequently, and it irritates me to no end. I will, for example, show someone the scientific evidence for vaccines, and they respond with, “well as a parent only I know what is best for my child.” Similarly, when I show people the evidence for GMOs, they often respond with something like, “well I just have a gut feeling that manipulating genes is bad.” I do not give a flying crap about your instincts or gut feelings. The entire reason that we do science is because instincts and feelings are unreliable. When someone presents you with a carefully conducted, properly controlled study, you absolutely cannot reject it just because you have a gut feeling that it’s wrong. Doing that makes no sense whatsoever. It is the most blatant form of willful ignorance imaginable. Don’t get me wrong, intuition is a good thing, and gut feelings can certainly help you in many situations, but they are not an accurate way to determine scientific facts.


Just to demonstrate the true absurdity of this response, let’s imagine for a minute that you went into the ER, and the doctor there said, “according to scientific studies, I should only give you X amount of morphine, but my gut tells me that I should actually give you five times that amount, so that’s what I’m going to do.” I’m pretty sure that you would immediately demand a different doctor. Similarly, imagine someone saying, “science says that smoking causes cancer, but my gut tells me that it’s fine.” Do you see the problem? Gut feelings simply aren’t reliable. That’s why we do science.




Bad reason #6: I’m entitled to my opinion/belief


Daniel Moyniham quote everyone is entitled to his own opinion but not his own factsThis is another very common response, and it is very similar to #5. Science deals with facts, not opinions or beliefs. When multiple scientific studies all agree that X is correct, it is no longer a matter of opinion. If you think that X is incorrect, that’s not your opinion, you’re just wrong. Think about the relationship between smoking and lung cancer again. What if someone said, “well everyone is entitled to their opinion, and my opinion is that it’s safe.” Do you see the problem? Scientists don’t have an opinion or belief that smoking is dangerous; rather, it is a scientific fact that it is dangerous, and if you think that it is safe, you are simply in denial. Similarly, you don’t get to have an “opinion” that the earth is young, or vaccines don’t work, or climate change isn’t true, or GMOs are dangerous, etc. All of those topics have been rigorously tested and the tests have yielded consistent results. It is a fact that we are changing the climate, a fact that vaccines work, a fact that the earth is old, etc. If you reject those, you are expressing willful ignorance, not an opinion or belief.


belief

Bad reason #7: I’ve done my research/an expert agrees with me




meme research you keep using that wordI’ll make this one simple: if your “research” disagrees with properly conducted, carefully controlled studies, then your research is wrong (or at the very least, must be rejected pending future data). There, it’s that simple. The only exception would be if your research is actually a large set of properly controlled studies which have directly refuted the study in question (e.g., if you have a meta-analysis vs. a single study, then, all else being equal, go with the meta-analysis). It’s also worth pointing out that having a few people with advanced degrees on your side does not justify your position (that’s a logically fallacy known as an appeal to authority). No matter what crackpot position you believe, you can find someone somewhere with an advanced degree who thinks you’re right.




Bad reason #8: scientific dogma


This response basically states that all scientists are forced to follow the “dogma” of their fields, and anyone who dares to question that dogma is quickly ridiculed and silenced. I’ve written about this before, so I’ll be brief here. In short, that’s simply not how science works. Nothing makes a scientist happier than discovering that something that we thought was true is actually false. In fact, that is how you make a name for yourself in science. No one was ever considered a great scientist for simply agreeing with everything that we already knew. Rather, the great scientists are the ones who have shown that our current understanding is wrong and a different paradigm provides a better understanding of the universe. To be clear, if you are going to defeat a well established idea, you are going to have to have some very strong evidence. After all, “extraordinary claims require extraordinary evidence,” but if you have that extraordinary evidence, then you absolutely can publish it. If, for example, I actually had powerful evidence that discredited the theory of evolution, not only could I publish in the journal of my choosing, but I would have just guaranteed myself the Nobel Prize. As a biologist, nothing could possibly be better for my career than discrediting Darwin. So why then aren’t biologists rushing to publish that evidence? Quite simply, because it doesn’t exist. Similarly, you don’t see many publications against anthropogenic climate change, vaccines, etc. because the data for those positions just don’t exist (fun fact: “data” is plural so “the data don’t” is actually grammatically correct).




Bad reason #9: distrust of governments/media


I often find that people reject science because of a distrust of governments or the media. For example, anti-vaccers often blindly reject all CDC statistics showing the benefits of vaccines (amusingly they readily accept the reported side-effects, inconsistent reasoning anyone?). Many people, however, take it even a step further. On numerous occasions, I have shown someone a study which was not in anyway affiliated with a government agency, yet they still responded with a lengthy rant about corrupt governments or the media. The basic idea of their argument seems to boil down to, “the government/media agree with these results, therefore they must be false.” This line of reasoning is, however, clearly fallacious (in fact it’s a logically fallacy known as guilt by association). Governments and the media will lie to push their own agendas, I’m certainly not denying that, but that fact does not automatically mean that everything that they say is a lie. For example, the CDC and other government agencies say that smoking is dangerous, does that mean that it’s safe? Obviously not. Similarly, if a news reporter said that you shouldn’t drink lava, would that mean that you should? It’s fine to be skeptical of what you are told by the government/media. In fact it is a good thing, but when you are presented with scientific evidence, then it’s not a matter of trusting the government/media. Rather, it is a matter of whether or not you accept science. In other words, I don’t need to trust the government or media in order to accept the results of a carefully controlled study.


decide for myself
This comment was a bizarre combination of distrust of the government, doing one’s own research, following a personal belief, and confusingly trying to equate science with government decision in order to assert that mistakes have been made in the past (or at least that’s my understanding of this ramble)




Bad reason #10: it’s a conspiracy




This one is very closely related to #8 and 9, but it takes things a step further. It proposes that there is a massive conspiracy and scientists are being paid by governments/big companies to falsify results. Just take a quick look at the anti-vaccine movement or the anti-GMO movement, and you will quickly find that pro-vaccine/pro-GMO scientists are vilified and receive constant accusations of being “shills.” Similarly, there are many people who think that all climate scientists have been bought off by governments. I’ve explained the problems with this line of reasoning in more detail here and here, so I’ll just talk about the biggest problem. Namely, the scope of this conspiracy would be impossibly huge. The scientific community consists of millions of people from all over the world working out of thousands of universities, institutes, non-profits, corporations, agencies, etc. It includes people from countless religions, cultures, political ideologies, etc. There is no way that you could possibly get that many people to agree on a massive deception like this. Just think about what is being proposed here. Do you honestly think that nearly all of the world’s climate scientists have been bought off? We are talking about thousands of people from all over the world. Similarly, there are numerous corporations, universities, non-profits, etc. involved in the research and production of vaccines and GMOs. Do you honestly think that all of those different organizations (many of whom compete with each other and have different goals and purposes) have all managed to come together to make one unified conspiracy? That’s just nuts. The same problems exist for governments. Topics like vaccines, GMOs, and the dangerous of climate change are agreed upon by numerous governments and scientific organizations from all over the world. Honestly ask yourself the following question: which is more plausible, that countless governments, companies, non-profits, etc. have all come together to create the world’s largest conspiracy and buy off virtually every scientist on the planet, or that the thousands of independent scientists who have devoted their lives to science are actually doing real research?




Bad reason #11: anecdotes


This list certainly wouldn’t be complete without talking about personal anecdotes. I can’t even begin to tell you how many times I have presented someone with scientific studies showing that vaccines are safe only to have them responded with, “but I know someone who developed autism after getting a vaccine” or “what about this case where someone became sick after a vaccination.” Anecdotes do not matter in science, because anecdotes don’t allow us to establish causation. Let me give an example. Suppose that someone takes treatment X and has a heart attack 5 minutes later. Can we conclude from that anecdote that treatment X causes heart attacks? NO! It is entirely possible that the heart attack was totally unrelated to the treatment and they just happened to coincide with one another. Indeed, I once heard a doctor describe a time where he was preparing to vaccinate a child, and while preparing the vaccine, the child began having a seizure (to be clear, he hadn’t vaccinate the child yet). He realized that if he had given the vaccine just 60 seconds earlier, it would have looked for all the world like the vaccine had caused the seizure when in fact the kid just happened to have a seizure at the same time that a vaccine was being administered.


From those two examples, it should be clear that anecdotes are worthless because they cannot establish causal relationships (in technical terms, using them to establish causation is a logical fallacy known as post hoc ergo propter hoc fallacies [i.e., A happened before B, therefore A caused B]). Properly controlled studies, however, do allow us to establish causation. If, we took a large group of individuals of the same age, ethnicity, medical history, etc., divided them randomly into two groups, and gave half of them treatment X and half of them a placebo, then and only then would we be able to look for causal relationships. In other words, if the treatment group has heart attacks significantly more frequently than the control group, then we could conclude that treatment X most likely causes heart attacks (science never proves anything with 100% certainty). Nothing else will let us make that claim. Even if you collected a whole series of anecdotes in which people had heart attacks following treatment X, it wouldn’t matter because there wouldn’t be any controls. In other words, I could respond to your anecdotes with anecdotes of people who received treatment X and didn’t have heart attacks as well as anecdotes of people who didn’t have treatment X, but still had heart attacks. Properly controlled studies are the only way to establish that one thing causes another. That goes for side effects of vaccines, alternative “medicines,” fad diets, etc.




Bad reason #12: a scientific study found that most scientific studies are wrong


This argument is fascinatingly ironic because it uses a scientific paper to say that we shouldn’t trust scientific papers, but let’s look closer because this argument actually has some merit. The paper being references is, “Why most published research findings are false” by John Ioannidis, and it is actually a very useful and informative work, but it often gets misused. The paper describes several reasons why published papers are often wrong, and I will go over just a few highlights. First, we have the problem of small sample sizes. As I have previously explained, small sample sizes are unreliable and you really need a large sample size to be confident in your results, yet many studies get published with small sample sizes, and you should be hesitant to place a lot of confidence in a result that didn’t come from a good sample size.


Second, we have publication bias. This can be a bias because of funding sources or preconceived ideas, but often it is a bias that is inherent in the publication system. In science, it is (unfortunately) often hard to publish a “negative” result. For example, if you do a drug trial and you find that it doesn’t work, you may have trouble publishing that result; whereas, if you got a “positive” result (i.e., it does work) you could easily publish it. The problem is that statistical significance relies on probabilities, and some papers will, get a false positive just by chance (this is called a type I error and I explained it in more detail here). So, when journals only publish positive results, you end up with a lot of false positives which aren’t balanced out by the negatives, because the negatives don’t get published. In other words, the type I error rate among published papers is much higher than the rate among all studies, because negative studies often don’t get published.


Now, all of that may sound very bleak, but it should not make you lose all confidence in the scientific process because of a very important component of scientific inquiry: replication. Ioannidis’s work applies mostly to single paper studies. In other words, when only one study has ever looked at drug X, there is a high chance that the results are actually wrong, but when multiple studies have tested drug X and all found that it works, then you can be fairly certain that it is actually effective. So, the arguments set forth by Ioannidis don’t apply to topics like vaccines, GMOs, and other areas of “settled science,” because they have been examined by thousands of studies. When numerous studies all agree, then you can have very high confidence in the results (this is why meta-analyses and systematic reviews are so useful). So, this paper shouldn’t make you question the safety of vaccines, the effects we are having on the climate, etc. It should, however, make you skeptical of the one or two anti-vaccine papers that you occasionally see, or the one paper supporting some “miracle cure,” or the occasional paper on homeopathy, acupuncture, etc. Those studies almost always have tiny sample sizes and countless other studies have failed to replicate their results. This is why it is so important to look at the entire body of literature not just a single study.




Conclusion


In summary, properly conducted, carefully controlled studies are the only way to reliably understand our universe, and you cannot reject them without good justification. Look around you. All of the modern marvels that you see today were brought to you courteously of science. Further, if I asked you, “How many of your siblings died of a terrible childhood disease?” I’m guessing that the answer would be “none.” If I had asked that question a few decades ago, however, most of you would have lost at least one sibling to diseases which are now almost unheard of. Even if you want to erroneously attribute the decline of those diseases to increased sanitation rather than vaccines and modern medicine, it is still science which is responsible for our increased hygiene and access to clean water. So no matter how you cut it, many of you wouldn’t be alive today if it wasn’t for science. Science clearly works and you need an extremely strong justification for rejecting scientific results.


To be fair, some scientists are corrupt and bad science does occasionally get published, but bad research tends to be identified and discredited by other researchers. In other words, there may be a high probability of a single paper being wrong, but when lots of different studies have all arrived at the same conclusion, you can be very confident in that conclusion. Perhaps most importantly, you cannot simply assume that a paper is bad just because you disagree with its results. You need to present actual evidence that it is flawed or biased before you can reject it.


 


 


 


 


 


 

							

	
			

													
						Posted in Nature of Science					
					|
				
								
					Tagged , , , , , , 				
				|
				
				20 Comments