Randomized placebo-controlled trials (RCTs) are often referred to as the “gold standard” for clinical research, and they certainly are useful and powerful tools. They aren’t, however, the only reliable tools, nor are they always the best tools. Indeed, there are many situations where other methods are actually superior. Nevertheless, among the general public, there is a common misconception that RCTs are the only reliable method and nothing else counts as valid science. For example, anti-vaccers often demand RCTs showing that vaccines are safe and effective (see note), and they routinely dismiss other methods, such as cohort studies, even when those methods are more suitable to answering the question at hand. At an even further extreme, I have encountered climate change deniers who say that climate science isn’t real science because it doesn’t use RCTs (i.e., we don’t take multiple earths and burn fossil fuels on half of them while using clean energy on the other half). This is, of course, nonsense and represents a fundamental misunderstanding of how science works. Indeed, most fields of science don’t rely on RCTs. Given how common these misconceptions are, I want to spend a few minutes talking about some of the pitfalls and problems with RCTs and explain why, in some cases, other methods are superior (there is actually a term for an over-reliance on RCTs: methodolatry).

Note: There are actually lots of RCTs for vaccines, anti-vaccers just ignore them. These studies usually come from the development of new vaccines, and they are quite good at showing that vaccines are effective; however, while they do support the safety of vaccines, for many safety concerns, other methods are actually superior (see the rest of the post)

Sample size

The biggest problem with RCTs is that they are only powerful when they have a large enough sample size, but getting the sample size necessary to test for rare effects is often impossible. To conduct an RCT on people, you need a large group of people who are eligible for the trial, are willing to come to your clinic/lab, are willing to risk taking an experimental treatment, are willing to risk being in the placebo group, will reliably follow the rules of your study, and will reliably return at the appropriate times for follow-ups. All of that is difficult and expensive to achieve, and, as a result, RCTs often only have a few dozen participants, and an RCT with a few hundred participants would be considered a large study.

Having a few dozen participants is fine if you are testing something with a large effect, but it is problematic if you are testing a rare effect. Imagine, for example, that you were testing a drug that caused the desired effect in 80% of test subjects and the background rate of that effect (i.e., its rate in the placebo group) was only 10%. In that case, if you had 60 participants (30 in each group) you’d expect to have 3 “successes” in the placebo group and 24 or more in the experimental group. That is a large difference that the RCT could easily detect (it would be statistically significant with a P value <0.0001).

Now imagine that you are testing something much more rare. Let’s say that you are testing an adverse event that has a background rate of 1% (i.e., 1% of people will have it even if they didn’t take the drug being tested). Further, let’s assume that the drug being tested does actually cause the side effect in an additional 1% of people (i.e., it causes it in 1 in 100 people, plus a background rate of 1 in 100). Now, we expect 1% of the control group to have the adverse event, and 2% of the experimental group to have it. That’s a problem if we only have 30 people in each group in our RCT. Mathematically, we’d only expect 0–1 people in each group to have the event, and we can’t distinguish that statistically. Further, even if we had a full 100 people in each group, we’d only get 1 event in the control group and 2 in the experimental group, which is far from being statistically significant even though the drug did actually have an effect (P = 1.0). Even at an enormous sample size of 1,000 people per group, we would not be able to detect the effect of the drug (P = 0.0978).  Indeed, would need a sample size of nearly 1,400 people per group before we would be able to detect a significant difference, and even then, we could just barely detect it.

This is the standard hierarchy of science, and in many cases it is correct, but it is not absolute, and other factors, such as sample size, have to be considered when comparing certain methods.

This sample size issue is a huge problem that severely limits RCTs. Fortunately, other methods like case-controlled trials and cohort studies are available. I talked about both of these methods in detail here, but briefly, case-control studies basically work backwards from RCTs. Rather than taking a group of people, giving half of them a treatment, then measuring an outcome of interest, case-controlled studies start by identifying a group of people who have an outcome of interest (the cases) then matching them to people who are as similar as possible in age, sex, race, etc. but lack the outcome of interest (the controls). Then, they look for differences in potential causes of interest. If you were interested in vaccines and autism, for example, you could take a group of people with autism, match them with a group of people without autism, then see if the vaccination rates differed among groups (this has been done, and the vaccination rates don’t differ; Destefano et al. 2004; Smeeth et al. 2004; Price et al. 2010; DeStefano et al. 2013; Uno et al. 2015).

Cohort studies are similar in that they are observational, rather than actually administering a treatment; however, they follow more of a traditional approach (as opposed to the backwards approach of case-controlled studies). They select a group of people who do not currently have an outcome of interest, some of which are expected to receive a treatment of interest. Then, they follow the patients to see whether the treatment causes the outcome. For example, you could take a large group of children who have yet to be vaccinated, follow them, and, presumably, some of them will have parents who vaccinate, and others won’t. So, you can then compare things like autism rates among the vaccinated and unvaccinated without having to actually be the one who administers the vaccine. These types of studies can also be done retrospectively by using medical records (here again, this has been done for vaccines and autism with no evidence that vaccines cause autism; Anders et al. 2004; Madsen et al. 2002; Jain et al. 2015; Verstraeten et al. 2003).

Cohort studies and case-controlled studies do make it more difficult to control confounding factors, but when they are done well, it is possible to account for many of them and, at times, even randomize. Further, they make it much easier to have very large sample sizes because of their observational nature. Studies with hundreds, thousands, and even hundreds of thousands of samples are common for these types of studies. Indeed, for the topic of vaccines and autism, a meta-analysis of these studies had a sample size of over 1.2 million children (Taylor et al. 2014), which is far more power than we could hope for from an RCT. Think of it this way, if an RCT, case-controlled study, and cohort study all had the same sample sizes, the RCT would be more powerful, but if the RCT has a small sample size, the outcome being tested is rare, and the case-controlled and cohort studies have substantially larger sample sizes, then the case-controlled and cohort studies will often yield more meaningful results (assuming that all the studies were set up correctly with adequate case-matching, controls, etc.).

All of this is very important for concerns over vaccine safety, because the known side-effects of vaccines are quite rate, often only occurring at rates of 1 in 1,000 or fewer. As a result, examining them with RCTs isn’t feasible, and would actually make vaccines look better than they are. So when anti-vaccers demand RCTs, they are actually demanding an unsuitable method that almost guarantees the opposite result from what anti-vaccers are hoping for.

This is also true for the adverse-events that anti-vaccers erroneously attribute to vaccines. Take autism and vaccines as an example. The CDC currently puts autism rates at 1 in 68. Further, we know that autism has a genetic component and that children without vaccines can still develop it. So, even if we assume that 50% of autism cases are caused by vaccines (they aren’t), we would need an enormous sample size with over 1,000 children in each group before an RCT could detect a significant effect, and getting that sample size for an RCT just isn’t feasible in most cases. Case-controlled and, especially, cohort studies, however, can achieve those sample sizes, and they have. Madsen et al. 2002, for example, was a cohort study on vaccines and autism that had a total sample size of over 400,000 children, which is far, far larger than could be achieved for an RCT (like all of the others, it failed to find a significant association between vaccines and autism).

You don’t always need a placebo

Some people might try to object to everything that I said above by insisting that, because of the nature of placebo effects, the control has to be a placebo, not simply a lack of treatment. That line of reasoning is, however, nearly always based on a misunderstanding of how placebo effects work. They aren’t magical and they aren’t going to do something like protect you from an infectious disease. Conversely nocebo effects (which occur when people respond negatively to a placebo) aren’t going to give you autism or anything like that. Thus, whether or not the control needs to be a placebo depends entirely on the type of outcome being researched. When the outcome is something subjective, like pain, placebos are critical. They are also important for things that can be subject to regression to the mean, such as the treatment of an existing ailment; however, when the outcome is something like getting an infectious disease or a developmental problem like autism, placebos are far less important.

For more on placebo effects, I recommend this post by Science-Based Medicine.

Ethical issues

Another problem with RCTs is that, at times, they are unethical. This is the case when either there is compelling evidence from other sources to suggest that the treatment is likely dangerous or there is compelling evidence from other sources to suggest that withholding the treatment is likely dangerous. Consider, for example, an experimental drug which, during animal testing, kills half the test subjects. Even though RCTs in humans rank far higher than animal studies on the hierarchy of evidence, in that case, we would have to be content with the animal trials, because it would clearly be unethical to administer that treatment to humans. The inverse is also true. When a treatment is already known to be highly beneficial, it would be unethical to withhold it.

Indeed, this is the reason why RCTs for vaccines are often limited to the development of new vaccines. We know from population-based studies, cohort studies, case-controlled studies, etc. that vaccines are exceedingly beneficial and save millions of lives. Therefore, it would be unethical to give a child a placebo and expose them to potentially deadly diseases.

Most fields of science don’t use RCTs

Finally, I want to address the notion that fields like climate science aren’t real science because they don’t use RCTs. First, as should be obvious by now, even in fields like clinical research, RCTs aren’t the only option. Second, and more importantly, science is simply a systematic process of making and testing falsifiable predictions, and for many fields, those predictions do not require RCTs to test. Consider evolutionary biology, for example. If you have a hypothesis that two species share a common ancestor, you can make predictions about what genetic and fossil patterns you should see if those species share a common ancestor, then you can examine the fossils and genetics and see if those predictions are correct. Testing that hypothesis does not require an RCT and an RCT wouldn’t even be useful for testing it. The same is true for fields like climatology. RCTs are neither necessary nor possible for studying past climates or our current climate, and that’s fine. The definition of science is not, “randomized placebo controlled trials.” Rather, it is a systematic process of making and testing predictions, and the exact methods used to test those predictions vary greatly from one field to the next.

Conclusion

In summary, placebo controlled trials are extremely useful and powerful tools, but they are not the only ones available, nor are they always the best ones. They are often limited by sample size, and in many cases, other methods are actually superior. Further, it is unethical to use them when substantial benefits or serious harms have been established by other methods. Therefore, the necessity and utility of RCTs has to be evaluated on a case-by-case basis. For some questions, using RCTs is extremely important, even vital, but in other cases, methods other than RCTs may suffice or even be superior.

Related posts

• 10 steps for evaluating scientific papers
• 12 bad reasons for rejecting scientific studies
• The hierarchy of evidence: Is the study’s design robust?
• Vaccines and autism: A thorough review of the evidence

Literature cited

• Anders et al. 2004. Thimerosal exposure in infants and developmental disorders: a retrospective cohort study in the United Kingdom does not support a causal association. Pediatrics 114:584–591
• DeStefano et al. 2004. Age at first measles-mumps-rubella vaccination in children with autism and school-matched control subjects: a population-based study in metropolitan Atlanta. Pediatrics 113:259–266
• DeStefano et al. 2013. Increasing exposure to antibody-stimulating proteins and polysaccharides in vaccines is not associated with risk of autism. J Ped 163:561–567
• Jain et al. 2015. Autism occurrence by MMR vaccine status among US children with older siblings with and without autism. JAMA 313:1534–1540
• Madsen et al. 2002. A population-based study of measles, mumps, and rubella vaccination and autism. New England Journal of Medicine 347:1477–1482
• Price et al. 2010. Prenatal and infant exposure to thimerosal from vaccines and immunoglobulins and risk of autism. Pediatrics 16:656–64
• Smeeth et al. 2004. MMR vaccination and pervasive developmental disorders: a case-control study. Lancet 364:963–969
• Taylor et al. 2014. Vaccines are not associated with autism: and evidence-based meta-analysis of case-control and cohort studies. Elsevier 32:3623-3629
• Uno et al. 2015. Early exposure to the combined measles-mumps-rubella vaccine and thimerosal-containing vaccines and risk of autism spectrum disorder. Vaccine 33:2511–2516
• Verstraeten et al. 2003. Safety of Thimerosal-Containing Vaccines: A two-phased study of computerized health maintenance organization databases. Pediatrics 112:1039–1048