I recently wrote a post in which I reviewed the scientific literature on vaccines and autism, and the responses from the anti-vaccine crowd were predictable. The most common of these responses followed the basic format of, “but it could be X, and scientists haven’t thoroughly studied X yet.” This line of reasoning is very common in anti-science circles. It shows up often in debates on GMOs, climate change, natural remedies, etc., and it is honestly somewhat of a confusing topic because, unlike many arguments used against science, this one is not automatically fallacious. There are times in which it is a completely legitimate argument, but there are other times when it is only somewhat legitimate, and other times when it is totally invalid. Thus, these arguments fall along a spectrum and there is no clear and universal set of rules for determining whether or not a particular argument is valid. Nevertheless, I want to talk about some clues and tools that you can use to help judge whether or not this argument is being used appropriately.

Science can never test everything
First, I want to explain the background of why this line of reasoning is potentially problematic. Technically speaking, science never proves anything. It just shows what is most likely to be true given the current evidence. Part of the reason for that is simply that it is impossible to test every possibility, which is why the argument is often fallacious. No matter how thoroughly a topic has been tested, there will always be some aspect of it that hasn’t been tested, which means that you can always make the argument that, “X hasn’t been tested, so it could be X.”

In the case of vaccines and autism, I frequently encounter arguments like, “well we haven’t tested this specific combination of vaccines” or “what if the vaccines interact with this particular genetic condition.” The problem is that we can make an infinite number of these. For example, “what if the vaccine interacts with this specific food that is often eaten with 24 hours of being vaccinated” or “what if playing on a playground shortly after receiving a vaccine is what causes it” or “what if watching TV after receiving a vaccine causes it.” Hopefully you get the point. We can always make these hypotheses, so you need to justify the hypothesis with something other than simply, “well scientists haven’t tested it.”

Logical fallacies
If you are either knowledgeable about logical fallacies or you read this blog often, you can probably spot the core fallacy that is behind this argument. It is, of course, the argument from ignorance fallacy. In its simplest terms, this fallacy occurs whenever you say, “we don’t know, therefore it’s X.” In other words, it uses a gap in our knowledge as an excuse to insert your own view. Creationists are probably the most famous for this and frequently make “God of the gaps arguments” where they say, “we don’t how X works/is formed, therefore God did it.” The problem should be obvious: when we don’t know something, all that you can say is, “we don’t know.” You can’t try to fill that knowledge gap with your pet hypothesis. This means that even when it is valid to bring up the fact that scientists haven’t tested X, you can never jump to the conclusion that X is true. All that you can say in that situation is, “we don’t know, and we need to do more testing.” You can’t assume that X is actually correct, yet that is often what people do. They jump from, “scientists haven’t tested possibility X” all the way to, “possibility X is true.”

Additionally, these arguments often lead to a logical blunder known as shifting the goal posts. It happens when you claim that X is true, then after X is debunked, you claim that it’s actually Y, then after Y is debunked, you claim that it is Z, etc. In other words, you keep changing your hypothesis without addressing your underlying view. A great example of this comes from thimerosal and the MMR vaccine. For years, anti-vaccers swore that it was thimerosal and the MMR vaccine that was causing autism, but that hypothesis has now been thoroughly debunked and thimerosal is no longer in childhood vaccines. As a result, many anti-vaxxers have shifted to claiming that it is an effect of multiple vaccines, or that it is aluminum, or an interaction with a genetic mechanism, etc. If you look through their arguments, you find that this happens over and over again. They claim that, “vaccines cause X via mechanism Y,” then after mechanism Y is thoroughly tested, they shift to, “well vaccines still cause X, but it is actually mechanism Z.” Hopefully you can spot the problem here. It is always possible to keep shifting your hypothesis, but doing so usually involves an ad hoc fallacy and is, therefore, not valid.

Proposing impossible studies
Now that the background is out of the way, let’s look at some of these arguments and why they often don’t work. One of the biggest clues that someone is not using this line of reasoning properly occurs when they demand impossible studies. For example, I have encountered GMO opponents and anti-vaccers who have said that they won’t be convinced until the technologies have been used for over 300 years. That is, however, clearly absurd. It is an arbitrary demand, and it is one that simply cannot be met in their lifetime. It’s the same thing as a creationists saying that he won’t accept evolution unless he can use a time machine to go back in time and see it for himself. These arguments are cop-outs that are no different from openly stating that nothing will ever convince you that you are wrong.

Another common argument by anti-vaccers is to demand additional randomized controlled trials. However, no ethics committee anywhere in the world is going to approve that study (except for new vaccines*) because the benefits of vaccines are so clear and well established. So once again, this is a demand for a study that simply isn’t going to happen.

*Note: Vaccines do actually undergo randomized trials when they are first being tested for approval. Anti-vaccers ignore this.

Similarly, anti-vaccers often demand a study that compares completely unvaccinated children to fully vaccinated children. At a quick glance, that may seem reasonable, but it is actually very problematic for a number of reasons. For one thing, the number of fully unvaccinated children in developed countries is quite low, and only a small fraction of them will have reliable and detailed medical records that you are able to access in a consistent manner. Further, you are going to have to limit your sample to a cohort of only a few years, otherwise you introduce numerous confounders (e.g., the vaccine schedule has changed over time, the genetic makeup of the population changes over time, routine medications change, etc.). Additionally, you are going to have to find a way to control for socioeconomic factors, geographic area, etc. By the time that you account for all of those, you are going to have a fairly small sample size.

Further, even if you could obtain a large sample size, you have a huge confounding factor that is nearly impossible to deal with. Namely, a family that refuses all vaccines very likely has other medical and lifestyle choices that differ greatly from the average person. Many people who fully refuse vaccines also use all manner of “alternative medicines,” shun science-based medicines, have bizarre diets, etc. This means that you are almost certainly going to have very large and important differences between your two groups other than just vaccination status, and that completely robs you of the ability to assign causation. So even though it sounds good in concept, it is just not a realistic study design.

Plausibility
Another important consideration is the plausibility of the hypothesis in question. You can generally use our current knowledge to figure out whether or not a hypothesis is likely to be true, and demanding research into that hypothesis is usually only reasonable when there is a high probability that the hypothesis will be true (more details here and here).

For example, even though there have been lots of studies of homeopathy, there are still many people who insist that we need more, large studies before we can conclude that it doesn’t work. In reality, however, homeopathy makes absolutely no sense. It breaks multiple fundamental concepts in science, and makes absurd claims like, “chemicals become more potent when they are diluted.” As such, it is exceptionally unlikely that it could work, and studying it is, quite frankly, a complete waste of time and money. You might as well study whether or not pigs can fly or if the moon is made of cheese.

Many (perhaps most) “cures” for cancer also fall into this category. For example, there are many people who insist that drinking Miracle Mineral Solution (which is just an industrial bleach) will cure all types of cancer (as well as “95% of all diseases”). You will not, however, find many large properly controlled studies on using it as a treatment for any of these conditions. Why? Quite simple, because it doesn’t make even the slightest bit of sense. We understand an awful lot about how cancer, bacteria, viruses, etc. work, and because of that knowledge, we know that it is extraordinarily unlikely that Miracle Mineral Solution could live up to any of its claims; therefore, we should be investing time and money into plausible solutions, rather than wasting it on utter nonsense (I explained the absurdity of Miracle Mineral Solution in detail here).

I could give countless additional examples of implausible hypotheses, but rather than continuing to provide examples, I want to make one final point. Namely, the fact that someone proposed a scientific sounding hypothetical pathway does not automatically mean that the pathway is plausible. The human body is amazingly complex, and hypothetical biochemical pathways rarely work in reality. Further, even if you can demonstrate that, in isolation, A causes B, B causes C, and C causes D, that does not automatically mean (or even suggest) that A will lead to D in humans.

For example, there will nearly always be a dose response. So you often end up with a situation where a large dose of A causes B, a large amount of B causes C, etc., but in humans A may only produce a small amount of B, in which case the chain will not proceed to C. There are also lots of other things to consider such as interactions with other chemicals and organs.

To give one actual example, a common pathway claimed by anti-vaccers is that vaccines sometimes cause fevers, and in some cases fevers can cause oxidative stress, and some types of oxidative stress are correlated with autism, therefore vaccines can cause autism. There are numerous problems here. First, yes, vaccines sometimes cause fevers, but those fevers are far less severe than the fevers caused by the diseases that vaccines prevent, so if this pathway was true, vaccines would likely help to prevent autism. Second, correlation clearly does not suggest causation, so just showing that oxidative stress is correlated with autism absolutely does not mean (or even suggest) that oxidative stress actually causes autism. Further, even if oxidative stress actually did cause autism, there would clearly be a certain level of oxidative stress that is required to cause autism, so you would need to establish that vaccines actually cause that level of sustained oxidative stress. In short, there are multiple steps that are questionable or likely don’t work. Now, obviously that doesn’t mean that the pathway definitely doesn’t work, but there is also no good reason to think that it does work, especially given the extent of the studies on vaccines and autism, which leads me into my next major topic.

Thoroughness of current testing
You should carefully consider the strength of the current testing. For example, many people continue to insist that we need more testing before eating GMOs, but in reality we have over 1,700 studies on their safety and environmental effects. Is it possible that there is a dangerous effect that all of those studies missed? Sure, but is it likely? Absolutely not! As such, continuing to insist that we need more studies before we should use them is clearly nonsensical. The ship of reasonable doubt has sailed, and we are well past the point of adequate testing.

Vaccines are the same story. Is it technically possible that vaccines cause oxidative stress which leads autism, and a large study comparing fully vaccinated and fully unvaccinated children would reveal that effect? Yes, but it is incredibly unlikely given the massive studies that we currently have (many of which have over 100,000 children, and the largest of which has over 1.2 million children, details here). To put this another way, if vaccines caused autism, there should be a dose response, and the current studies had the power to detect even very tiny effect sizes. So even if, as anti-vaccers often claim, the current studies compared children with every vaccine but one (usually MMR) with children who received every vaccine, there would still be a dose response, and you would still expect that to show up in the large studies. So the hypothetical pathways proposed by anti-vaccers are highly unlikely because if they were true, then the large studies should have found that children who received more vaccines had a higher rate of autism.

Likelihood that the risk will outweigh the benefits
Finally, when considering whether or not the demand for more research is reasonable, we also have to consider the known benefits/risks as well as the likelihood that the proposed hypothesis will be true. In other words, every decision has risks associated with it, so you have to weigh those risks against the benefits. In the case of vaccines, the odds that they would cause autism are quite low, whereas the known benefits are extremely high. To put this another way, if you demand more studies before we use vaccines, then what you are saying is, “we should not use a medicine that we know has enormous benefits and saves millions of lives annually because there is a slight chance that it causes a disability.” Hopefully you see why that is a problem. The known benefits far outweigh the hypothetical risks. Similarly, with GMOs, those of us in the developed world are sitting around demanding yet another study just in case it will find a problem that the past 1,700 missed, while people in developing countries are suffering from an extremely serious lack of food and proper nutrition that we know GMOs could help to ameliorate.

Conversely, you can have situations where it is reasonable to take action because the risk is enormous and far outweighs the cost of taking action. For example, many people insist that we need more studies before we take action on climate change, and I often hear people say things like, “if the planet is still warming in 100 years, then I’ll believe it.” In reality, of course, we have lots of very solid scientific evidence and logical reasons for concluding that we are causing the climate to change. Therefore, the hypothesis that climate change is just natural is extremely unlikely. More to the point, the consequences of climate change are so dire that waiting for additional evidence before taking action is foolhardy. It’s like standing on a sinking ship after the order to abandon ship has been given and saying, “I want more evidence before I get off this boat. If it is still sinking in a few hours, then I’ll believe it.”

Conclusion
In short, there are many situations in which it is unreasonable to demand additional studies before we take action, and you need to consider things like whether or not the proposed study is possible, whether or not the hypothesis is reasonable, the strength of the current evidence, and the known benefits of the thing in question. More often than not, when people demand additional studies, they are asking for impossible studies on implausible hypotheses, despite a mountain of contrary evidence and extraordinary known benefits. In other words, although the demand for more studies may seem reasonable at first, it is often used simply as a facade to make a position seem reasonable when, in fact, no amount of evidence will ever change the mind of the person making the demand.