This is a sad week for me, because this week I must bid farewell to one of my all time favorite TV shows: MythBusters. In a world where educational television has degraded to the point that it consists largely of extreme fishing, people buying and selling old junk, idiots looking for gold, challenging driving, and fake documentaries about mermaids and extinct sharks, MythBusters has stood almost alone in maintaining a high educational standard while still being immensely enjoyable. If you look beyond the explosions and comical personalities displayed on MythBusters, there actually are some extremely good core lessons. Therefore, as the show draws to a close, I want to celebrate it by talking about all of the things that it got right…as well as some things that it didn’t get right.

To be clear, I’m not going to nit pick specific episodes, nor am I going to argue that the show’s value lies in the specific myths that they tested. Rather, I am going to talk about the overarching lessons and themes from this extraordinary show.

Via MythBusters Episode 187 “Bubble Pack Plunge”

Lesson 1: Question everything
As a skeptic, I think that the single most valuable lesson from MythBusters was simply that we should question everything. We should always demand evidence before believing that something is true, and this show illustrated that brilliantly. Throughout the show’s history, they debunked numerous viral videos, newspaper stories, internet rumors, wives’ tales, etc. Time and time again, things that people believed and thought were true utterly failed once they were tested.

The importance of this lesson cannot be overstated. Everyday on this blog, I deal with people who deny scientific results, and for the most part, they aren’t unintelligent people. Rather, they simply haven’t learned to demand good evidence. They believe things based on hearsay and what some random person wrote on the internet, rather than actually fact checking. MythBusters  did an extraordinary job of demonstrating why that is so foolhardy. This, more than anything else, is why I think that MythBusters did a tremendous public service. It showed the importance of fact checking and looking for good evidence, and it did so in a way that was enjoyable and accessible to everyone.

Lesson 2: Intuition is unreliable
Closely related to the first lesson is the fact that intuition and gut instincts are highly unreliable. In nearly every episode, the MythBusters made predictions about what would happen based on their gut feelings and past experiences, and they were very often wrong. Further, if you’re like me, you probably made predictions at the beginning of each episode as well, and I willing to bet that you were also wrong…a lot (I certainly was).

This is extremely important because people base decisions and views on gut instincts all the time. I constantly encounter people who, “just know in their gut” that pharmaceutical X is dangerous or miracle cure Y works. MythBusters demonstrated why that way of thinking is flawed, and they did so in a very visually engaging way that makes sense to most people.

Lesson 3: Being wrong is exciting/”Failure is always an option”
One of my favorite things about the MythBusters was watching their excitement when they were wrong (especially Adam’s). You could tell that many of their favorite moments were times when they were convinced that X would happen, but Y happened instead. That type of excitement about being wrong exists in real science as well. Sure, there are plenty of times when being wrong is disappointing, but it is very often the case the being wrong is far more exciting that being right. The solutions that nature arrived at are generally far more interesting than the solutions that we’ve come up with (at least in my opinion), and as a scientist, finding something that you didn’t expect is exhilarating. It means that there is more to learn and understand about the system that you are working with. Science is often a process of finding things that don’t work, and you frequently learn far more from being wrong than you do from being right.

Perhaps of more practical importance for most people, you should never be afraid to be wrong. Imagine how boring and annoying the show would have been if the MythBusters always insisted that their original predictions were right, even when their tests said otherwise. Nevertheless, many people go through life that way. They “know” that they are right, and nothing will ever convince them otherwise. That’s a really sad and boring way to view the world. You should always embrace the possibility that you might be wrong rather than running from it.

The best embodiment of the MythBusters’ willingness to be wrong is probably all of the episode revisits. Fans would write in and critique their tests, and they would listen. Rather than stubbornly saying, “no, you’re wrong, we know we’re right” they actually took the comments seriously and tried the fans’ suggestions (and sometimes the fans were right). That’s how everyone should take criticism. It’s not always an easy thing to do, but we should always be willing to accept the possibility that we are wrong, and we should consider contrary evidence when presented with it.

Lesson 4: Other people know more than you do
There is a pervasive and unfortunate tendency for people to downplay the importance of experts. People would rather trust an unqualified blogger than a licensed doctor, experienced scientist, etc. Further, many people go as far as accusing real experts of being arrogant or elitist for having the audacity to think that their years of training and experience have gifted them with more knowledge than could be acquired through a few hours with Google. The MythBusters, however, were more than happy to consult with experts. They constantly got input from qualified individuals and they incorporated that knowledge and experience into their tests. You never saw them saying, “well the experts said to do X, but we read Y on the internet, so the experts must be wrong.”

This is how real science works as well. Look at most scientific publications and you will see a whole string of authors. Topics like science and medicine are extremely complex and most researchers can only claim expertise on a very narrow sub-discipline. As a result, we work with other scientists constantly. We forge collaborations with people who know more than us about a particular area, rather than plowing forward with our ignorance.

To be clear, I’m not suggesting that you blindly accept something just because an expert said it (that would be an appeal to authority fallacy), but you should recognize and acknowledge that experts do, on average, know a lot more about their area of study than someone who has never worked in that field. As a result, you should approach topics on which you have no training or experience with a great deal of humility, and you should be extremely cautious about concluding that you have found something which hundreds of experts have missed.

This quote was obviously made in jest, but it is nevertheless true that science involves a tremendous amount of note taking.

Lesson 5: Basics of the scientific method
Some episodes were more scientific than others, and there were plenty of times where I don’t think that they used proper controls, but overall, I think that the show did a very good job of introducing people to the basic concepts of the scientific method. To be clear, there is no one almighty scientific method that everyone religiously follows, but there are some overarching concepts which are nearly always applied.

First, science always goes from data to a conclusion, rather than starting with a conclusion, then trying to make the data fit, and MythBusters illustrated this nicely. They started with the rationale for the myth, then they tested the myth, then they drew a conclusion. Again, imagine how annoying the show would have been if they started with a conclusion, then tried to manipulate the test to make sure that the outcome fit their conclusion. Nevertheless, many pseudoscientific disciplines do exactly that, and you should be wary of them.

Second, MythBusters always at least tried to have a good control group. Controls are vital if you want to assign causation. If you want to know whether or not X causes Y, you need to know how often Y happens without X happening. Thus, scientific tests that are designed to infer causation involve an experimental group that receives the treatment of interest and a control group which is handled the same, but doesn’t get the actual treatment.  MythBusters generally illustrated this well. Not only did they have a control group, but they usually took time to explain it and make relevant comparisons to it. It would often have been very easy to gloss over the control instead of properly explaining it, so I really appreciate the fact that they highlighted it.

To be clear, some of their controls were better than others. For example, on several occasions they would do something like having Jamie drive an experimental car, while Adam drove a control car. That is a big problem because driver becomes a confounding factor. In other words, the control group and treatment group need to be identical in every way except the treatment, but when you have two different drivers, then differences between the groups may have been from the drivers, not the treatment being tested. Overall though, I think that they illustrated the concept of a control nicely, and, let’s be honest, professional scientists don’t always get controls right either.

Finally, the MythBusters constantly made testable predictions. Prior to most experiments, one of them would make a statement to the effect of, “If the myth is true, then we should see X” or “If we see Y, then this one is busted.” Then, after the test, they would draw conclusions based on whether or not the prediction came true. This is, once again, very much the way that real science works. In fact, testable, falsifiable predictions are central to modern science. So, I think that they did a very good job of illustrating that concept for the public.

Addendum (2-3-16): Although I hinted at this in the original post, I should have directly stated it. In science, you always should be asking open ended questions rather than setting out to demonstrate something, and that’s what the MythBusters did as well. They did not set out with the goal of proving a myth right or wrong. Rather, they tested each myth. They gave each myth the best possible chance of succeeding in order to find out whether or not it actually was true. Similarly, scientists don’t set out to prove that drug X works or chemical Y is dangerous. Rather, we test them to find out whether or not they work, are dangerous, etc.

Lesson 6: Critical thinking
Critical thinking skills are generally refined and honed through practice, and MythBusters provided a venue for fostering the development of those skills. Most of the people I know who watch MythBusters don’t do so passively. Rather, they actively scrutinize every minutia of how the MythBusters designed their tests, and they’ll often debate with their friends (or random people on the internet) about whether or not the MythBusters got it right (the extremely active fan site is great evidence of this). This type of critical thinking and analysis is fantastic and is, in fact, a huge part of science.

Scientists generally don’t read papers passively. Rather, we pick them apart and critically analyze the experimental designs and results. Indeed, during my graduate training, I have heard several professors give lectures on analyzing published papers and critiquing other researchers’ results. So it makes me extremely happy that MythBusters succeeded at getting fans to engage with the material and think about things like whether or not the control group was appropriate. I think it is an enormous testament to the educational value of the show.

Lesson 7: Start with small experiments
Most episodes began with “small scale” experiments where the MythBusters dissected and tested each individual component of a myth. Then, at the end, they put all of the pieces together for a full scale test, and science often works in very much the same way. In medical research, for example, we often start with “small scale” experiments like animal trials. Similarly, we use in vitro studies to look at individual components of biochemical pathways. Then, if those preliminary trials yield promising results, we put all of the pieces together in a “full scale” test such as a randomized controlled trial.

Importantly, in both science and myth-busting, the small scale tests and the large scale tests often don’t agree, and when that happens, you default to the full scale tests. It was often the case on MythBusters that individual components would work on the small tests, but once they scaled it up, there were other complexities or interactions that weren’t evident in the small scale, which ultimately resulted in the full scale experiment falsifying the myth. The same is true for scientific tests. For example, in vitro studies are great for looking at how particular cells respond to specific chemicals, but the human body is far more complex than a few cells in a petri dish. Thus, you often have drugs that are very promising in the small scale in vitro tests but fail during the full scale randomized controlled trials. Similarly, just as the MythBusters’ small models were imperfect representations of the full systems, animals are imperfect models of humans, and drugs often act differently in animals than in humans. When that happens, however, you should generally default to the full scale tests, not the animal models or in vitro studies.

Lesson 8: Lessons in physics
This one isn’t really a single lesson, but rather a whole set of lessons about physics (and to a lesser extent chemistry and biology). Multiple episodes were devoted entirely to physics, and these were often my favorites. As an undergraduate, I had to study physics, so I know the math for things like, “if you fire a bullet and simultaneously drop one from the same height, they will hit the ground at the same time,” but actually seeing those classic physics examples demonstrated and visualized was truly delightful. Further, they provided wonderful, memorable illustrations for people who haven’t studied physics and aren’t familiar with the math.

Additionally, even when the MythBusters weren’t directly testing classic physics examples, the episodes tended to be packed with real science. Yes, there were a lot of explosions and silliness, but there were also outstanding explanations and demonstrations of stoichiometry, pressure waves, fulcrums, masses, forces, etc. They explained the science behind everything that they did, and that was truly wonderful to see. Plus, I think that people tend to remember scientific concepts much better when they are used to blow something up, rather than simply being described in text books.

What they got wrong: Sample sizes and statistics
Finally, I want to talk about the one major thing that the MythBusters got wrong: namely, the small sample sizes and lack of statistics. Most tests were only replicated about three times, which simply isn’t enough to give you a reliable answer. Further, in almost every case, they simply looked at the averages and went with whichever one was larger, even if the difference was very small. This is extremely problematic because that difference may simply be a chance result, and even if you have a large sample size, you need to do statistics in order to determine how likely you are to get that difference just by chance.

Many of the fuel mileage myths illustrate this well. They would drive the car under one condition three times, then drive it under a different condition three times, then compare the average fuel consumption and draw a conclusion. Often, there would be a very slight difference between those results, yet they would still call the myth based on those tiny sample sizes and lack of statistics. That’s a big problem because there is probably a lot of variation from slight differences in how fast they drove, how straight they kept the wheel, how steady they were on the gas pedal, wind gusts, etc. All of those factors create statistical noise which make it extremely difficult to distinguish between an actual difference and a false difference that arose because of chance variation. This is why scientists use statistics and large sample sizes. As your sample size increases, you have more power to cut through the statistical noise and detect true differences.

Although this is a real problem, it is ultimately something that I can forgive the MythBusters for, because I understand that it was necessary for the show. I get that there has to be a balance between education and entertainment, and doing each test 30 times would be really boring. Further, if they tried to go into details of the statistics, I find it extremely likely that many people would start looking for something else to watch. So, as much as I would have loved to have seen them use proper statistics, I realize that doing so would probably have shortened the longevity of the show (though they did do a brilliant job of illustrating the Monty Hall paradox, so maybe they could have succeeded at making math fun for most people).

Note: This criticism only applies to myths where they were comparing two things. Many of the myths were simply, “can you use X to do Y” (e.g., can you use one gram of sodium to blow a man-sized hole in a brick wall?). That type of question differs greatly from the type of questions that most professional scientists today tackle, and for that type of question, simply using X to do Y once is enough to say that it can be done. Conversely, in many cases it is possible to pretty conclusively show that something cannot happen, even without a large sample size. 

Conclusion
When it’s all said and done, I think that MythBusters did an extraordinary job of making science exciting and teaching scientific concepts to the general public. It had its faults and some tests were better than others, but I still contend that it taught many valuable lessons and truly lived up to the title, “educational television.” So, to Jamie, Adam, Kari, Tory, Grant, and everyone else involved with this tremendous show, thank you for all of the science, laughs, and explosions. You will be missed.