Basic Statistics Part 5: Means vs Medians, Is the “Average” Reliable?

Posted on October 10, 2016 by Fallacy Man

To many people, this may seem like the most boring topic in the world, but it is actually vitally important not only for understanding scientific results, but also for understanding much of the data that we are presented with on a daily basis. We are constantly confronted with claims about the “average” from friends, scientists, doctors, politicians, etc., but in many cases, those data are being presented in a misleading or even deceptive way, and you should be able to spot that deception. For example, during the first calendar year of my blog, my posts were viewed an average of 9,272 times per post. When faced with a number like that, you would normally expect that roughly half of my posts had more than 9,272 views, and roughly half had fewer than 9,272 views. In reality, out of 81 posts, only 12 had more than 9,272 views! You see, this data set is not one that can be accurately analyzed using the calculation that most people refer to as the average (what scientists call the mean), so I presented you with an unrealistic view of my blog by using an inappropriate measure of the central tendency of the data. This example is, of course, trivial, but people do this all the time. You can find plenty of scientific publications that made this blunder, politicians frequently cite inappropriate statistics like this, etc. So I think that it is very important for people to understand the different types of data distributions and how they should be analyzed.

Note: I want to clarify upfront that in this post I am making generalizations about understanding the central tendency in the type of data sets you are likely to encounter in your daily lives. There are certainly exceptions, particularly when you get into statistical modeling, where there are many different distributions, and you often use a mean in association with multiple other variables to understand and analyze them. Those sorts of situations are not what I am talking about here. Rather, I am trying to give a general introduction to means and medians and why it can sometimes be misleading to report means. Again, the goal here is that if you hear a politician, science reporter, etc. say, “the average was…” you should be able to tell if that is biased measure of central tendency in that case. That is the context in which I am making statements like, “the mean is useful for…”

Means, medians, and modes
For most data sets, we are interested in knowing the central tendency of the data. This is generally accomplished by presenting a single number that summarizes the data and presents a value that you expect most of the data points to be near. Most people do this by presenting an “average.” As all of you hopefully know, you calculate the average value by adding all of the data points together, then dividing by the total number of data points (scientists refer to this statistic as the “arithmetic mean”). This is by far the most common measure of central tendency among the general public, but it is not the only one, and in many cases, it is a horrible statistic to use (more on that later).

The primary alternative to the mean is what is known as a median. For this statistic, you line all of your data points up from smallest to largest, and the median is simply the middle value. For example, if your data points were: 2, 4, 5, 7, and 30, the median would be 5 (in contrast the mean would be 9.6). In situations where you have an even number of data points, you simply take the mean of the middle two. In other words, if your data points were: 2, 4, 7, and 30, then the median would be 5.5 ([4+7]/2).

Finally, we also have the mode. This is simply the value that appears most often. This statistic is generally not appropriate for numerical data because it doesn’t really show you the central tendency in most cases. It is however useful for nominal data. In other words, when you are simply counting things by category. For example, if you wanted to know the most popular brand of car in your neighborhood you might count them all. Now, suppose that you did that and you found 20 Toyotas, 5 Fords, and 6 Chevrolets. You obviously can’t take a mean or median, but you can report that there were more Toyotas than anything else. That’s a mode (again, you can do that with actual numbers as well, but it often doesn’t tell you much). I thought that it was important to explain what a mode was, but for the remainder of this post I really want to focus on means and medians, because they are the ones that often get used inappropriately (especially the mean).

 

Figure 1: This shows three different types of symmetrical data distributions. (A) shows a normal distribution, which is the only type of data distribution for which means are appropriate.

Figure 1: This shows three different types of symmetrical data distributions. (A) shows a normal distribution, which is the only type of data distribution for which means are appropriate.

Data distributions: when can’t you use means?
Now that we have established what means and medians are, we can talk more about when they can and cannot be used, but to do that, we need to talk about data distributions. As I mentioned in the opening paragraph, when we talk about averages, we generally think that roughly half the data points should be above the mean and half the data points should be below it. Indeed, that intuition is correct. In order for the mean to be really useful, that situation should be roughly true, but if you think back to our definitions of means and medians, you will realize that what we have just defined is a median, not a mean. This brings me to the most important point of the entire post: as a general rule, if you are interested in knowing the central tendency of your data, means are more informative when the data are normally distributed, but they can be very problematic when the data are skewed to one side (as noted earlier, there are exceptions). The easiest way to explain what I mean by that is simply to show you, so look at Figure 1A on the right. This is what we call a “bell-shaped” or “normally distributed” data set, and the mean is at 11, which is exactly where we expect it to be (on a side note, for data sets with a perfectly normal distribution, the mean and median will always be the same).

Technically speaking, you can also use a mean anytime that the data have a symmetrical distribution (i.e., if you folded the graph in half, both sides would be the same), but as you can see in Figure 1B and 1C, although you could report a mean, the mean is still not very useful. In Figure 1B, all of the values are equally frequent, so there isn’t really a central tendency, and in Figure 1C, the distribution is bimodal, so there are really two central tendencies. Data sets like either of those are, however, fairly rare, and the far more common alternative to a normal distribution is a skewed distribution (Figures 2 and 3).

Figure 2: This shows a data set that would be normally distributed if it wasn’t for one data point that is way out at 10,000. That data point gives it an extremely long tail, which results in a very inaccurate mean. As a general rule, the longer the tail, the less accurate the mean is. In contrast, medians are affected by the number of data points on the tail, but the length of the tail is irrelevant. Note: I gave this figure and extremely long tail to illustrate how much one huge outlier can affect means, but the distributions in Figure 3 are more typical of what we mean when we say that a distribution is "skewed."

Figure 2: This shows a data set that would be normally distributed if it wasn’t for one data point that is way out at 10,000. That data point gives it an extremely long tail, which results in a very inaccurate mean. As a general rule, the longer the tail, the less accurate the mean is. In contrast, medians are affected by the number of data points on the tail, but the length of the tail is irrelevant. Note: I gave this figure an extremely long tail to illustrate how much one huge outlier can affect means, but the distributions in Figure 3 are more typical of what we mean when we say that a distribution is “skewed.”

When we say that a distribution is “skewed” we mean that it is not symmetrical like a normal distribution. Rather, the data clump on one side with a “tail” stretching off to the other side. We would further describe the graphs that I have illustrated here as either “right-skewed” or “right-tailed” (because the tail is on the right). Skewed data sets like this are extremely common, but you often cannot use a mean to describe them, because the mean gives you a misleading view of the data. For example, in Figure 2, the median is 11, which makes sense based on just looking at the data. In other words, 11 is a good description of the central tendency of that data set, and saying that the median = 11 tells you something useful. In contrast, the mean for that data set is 20.8, which is obviously a terrible representation of the central tendency of that data set. Almost all of the data points are less than 20.8, and that statistic is extremely misleading.

So what’s going on here? Well this data set has an extremely long tail because there was one data point all the way out at 10,000, and if you think about the math behind the mean, it should be obvious that having a single data point that is so much higher than all of the rest will seriously bias a mean (this is the same thing that happened with my blog data). In contrast to the mean, the median will still be robust because it just ranks that data, then selects the middle data point. So it wouldn’t matter if that last data point was ten thousand or ten trillion, the median would be same (in contrast, the mean will keep going up).

Figure 3: This shows two different left-skewed data sets. The more skewed that they become, the less accurate the mean is. Although Figure 2 is technically skewed, when we use that term to describe data, we are usually referring to distributions that look more like this.

Figure 3: This shows two different right-skewed data sets. The more skewed that they become, the less accurate the mean is. Although Figure 2 is technically skewed, when we use that term to describe data, we are usually referring to distributions that look more like this.

From what I have just said, it should make sense that the length of tail and the number of data points on the tail have a huge effect on whether or not you can use a mean. Consider, for example, the two panels in Figure 3. Both are still right-skewed, but not nearly as severely as Figure 2. Indeed, 3A doesn’t look that far from a normal distribution and, in fact, the mean and median are pretty similar (6.5 and 6, respectively). So although the median is a better statistic, the mean is still pretty good. The more skewed that we make it, however, the further that the gap between those two becomes. In 3B, for example, the media is 3, whereas the mean is 4.3.

All of that may have seemed complicated, so let me boil it down to two take home messages. First, means are reliable measures of the central tendency when the data are normally distributed (or at least close to normal), but when the data are skewed and you have many outliers, the median generally gives you a better representation of the data. Second, the more skewed that a data set is, and especially the longer that its tail is, the less reliable the mean becomes.

Note: There are cases in which the typical relationships between means and medians that I have presented do not hold true, but these generally occur for discrete variables rather than continuous variables. A discrete variable is simply one for which there are a finite number of values, such as count data (e.g., the number of individuals per household would be discrete because you can’t have a fraction of a person). Continuous variables are ones for which there are (at least in concept) an infinite number of values (e.g., measurement data). For more information, please read Hippel 2005. Mean, Median, Skew: Correcting a Textbook Rule. Journal of Statistics Education 13.

 

How to tell if means are being used correctly
At this point, you may be wondering how on earth you are supposed to tell when someone is using a mean when they should be using a median, and there are a couple of things to watch out for. First, familiarity with the type of data being worked with is usually very helpful, because if you know something about how those distributions generally appear, you can often intuit what the distribution will probably look like. Let me give you an example. If someone is reporting the mean income for all of the US, do you think that is appropriate? Well, if you know anything about the wealth distribution in the US, then you know that there is a very large lower and middle class, accompanied by a tiny upper class that makes way, way more than the other two classes. Now, picture in your mind what that distribution will look like. You should be picturing a very skewed graph with most people in the low to moderate income categories on the left, and a few rich people way out on a tail on the far right, and, indeed, that is what the distribution looks like. So in that case, people should be reporting medians, not means (note: I am not making any political statements here, I am just using a simple example that most of my readers should be familiar with).

To be clear, I’m not suggesting that you go with your gut instead of actually looking at the data, but background knowledge about the type of data being presented is useful as a first pass filtering mechanism to see if any red flags go up. It is also one of the reasons why it is important to be knowledgeable in a particular scientific field when trying to assess that literature. Knowing what the data sets for a given topic typically look like helps you to spot shoddy statistics.

In cases where you can’t intuit or easily look up the distribution, ranges become really important. If, for example, someone tells you that the mean for something is 100, that isn’t very useful without out also knowing something about the distribution of data around that mean. Standard deviations and variances are generally really useful for that purpose, but for checking normality, the range is far more valuable, because it gives you the highest and lowest values. Suppose, for example, that you were told not only that the mean was 100, but also that lowest value was 10 and that the highest value was 110 (i.e., the range was 10–110). That tells you something very useful about the data, because it tells you that the data has a long left tail (i.e., it is left-skewed), which also tells you that the mean might be misleading and is not capturing the central tendency particularly well.

 

Conclusion
There are several key things to take away from this post. First, people often report averages (means), but it is often misleading to do so, and you should be cautious of them. Means are generally most informative as a measure of central tendency when the data are fairly close to a normal distribution, and when the data sets are skewed, you often should use medians, not means. As a result, when someone presents you with a mean, you should think about the distribution of the data and look at other pieces of information such as the range.

As a fun concluding exercise, I want you to evaluate two technically true claims:
Claim 1: The average number of followers per Twitter account is 208.
Claim 2: The average height of women in the USA is 163 cm.

The goal here is to assess whether or not the mean is a useful measure for the central tenancy of the data for those two data sets. So, for both claims, I want you to think about how you expect the data to look. Think about the types of accounts that often exist on Twitter and the range of human heights, and see if you can intuit what the data sets will look like. Then, actually look into the data a bit and see if you were right. This is a very easy exercise that won’t take you more than a few minutes, but it is the type of skepticism that you should apply to all data sets, so I think that it will be useful for you to actually work through this mentally. Also, yes, I did just assign you homework from a blog post.

The answer for twitter is here, and for height you can just check good old Wikipedia for a comparison of the mean and median.

Related Posts

Note: This is the second time that I have posted this basic article. I removed the first version almost as soon as I posted it because someone pointed out that I made several errors and over-generalizations. I did not immediately have time to edit the post, so I simply removed it rather than furthering the spread of misinformation. I have now corrected the post, and I apologize profusely for my mistakes and appreciate having them pointed out to me. I try very hard to write accurate and informative posts, but I am only human and do make mistakes, which is why it is important to fact check everything that you read and hear (including from me). Thank you for not letting me get away with shoddy work.

Posted in Nature of Science | Tagged , | 3 Comments

Don’t tell people to “Google it.” That’s your job, not theirs

Posted on September 27, 2016 by Fallacy Man

I spend a lot of time debating people on the internet, and, unsurprisingly, I frequently encounter people who make outlandish claims without providing any evidence to support those claims. In such situations, I typically ask them to present their sources, at which point they usually respond indignantly with something along the lines of, “just Google it,” “look it up yourself,” “do your own research,” or “so you expect me to do your work for you? I don’t think so.” However, as I will explain in this post, these responses are fundamentally flawed because they fallaciously shift the burden of proof and ignore the rules of logic.

The first major problem is simply that these responses misuse the burden of proof. The person making the claim always bears the burden and is responsible for providing supporting evidence, especially when their claim goes against the generally accepted wisdom. In other words, if I say, “aliens caused 9/11” then the burden is entirely on me to provide reliable sources to back up my assertion. You, in contrast, would be under absolutely no obligation to refute my claim or even take it seriously until I provided that evidence. So I could not tell you to “just Google it” because that would be placing the responsibility on you to debunk my claim, when in reality, the responsibility is on me to support my claim. To put it simply, you do not have to take a claim seriously until your opponent provides reputable sources to back it up. It is 100% their responsibility. So when people say things like, “so you expect me to do your work for you?” they have things totally backwards.

A useful way to think about this is to apply it to courtroom scenarios. The prosecution is the one making the claim or assertion (i.e., “person X is guilty”), so it is their responsibility to provide the evidence. The defense only has a responsibility after the evidence has been presented. In other words, the defendant does not need to prove their position (i.e., they don’t need to prove that they are innocent), rather they simply need to show that the prosecution’s evidence does not prove guilt. So the defense only bears a burden to defeat the prosecution’s evidence, rather than bearing a burden to provide evidence supporting their innocence. To be clear, there is certainly nothing to stop them from providing that evidence, but they aren’t required to do so. In other words, the prosecution must provide evidence of guilt, whereas the defense does not have to provide evidence of innocence.

The exact same rules apply in logical debates. Any time that you make a claim like, “X cause Y,” “Z is dangerous,” “studies have found X,” etc., you have just placed yourself into the role of the prosecution. At that point, it is your responsibility to provide evidence for your claim, whereas your opponent does not have to provide evidence refuting your claim. They are certainly welcome to do so, but they are under no obligation to prove you wrong until you provide evidence.

When you think about this in the context of a debate about science, the reason that the burden of proof works this way should become obvious. Imagine, for example, that I said, “studies have found X,” but I refused to actually provide you with such studies. How would you prove that I was wrong? Quite frankly, you couldn’t. You could show me that you failed to find such papers in any major database, but that wouldn’t prove that the studies don’t exist because they could simply have been published in a minor journal that isn’t well archived. See the problem? Proving a negative claim (e.g., “papers that found X do not exist”) is nearly impossible. It’s like trying to prove that Bigfoot doesn’t exist. It can’t be done because no matter how many trail cameras we put out, it is always technically possible that Bigfoot does exist and has simply managed to elude all of the cameras. That is why the burden is always on the person making the claim. I would be responsible for providing you with the papers that found X, just as Bigfoot hunters, UFO spotters, etc. are all responsible for providing evidence for their position. In other words, if one person says, “prove to me that Nessie, yetis, etc. exist” and the other says, “prove to me that they don’t,” the second person is being irrational because they are trying to shift the burden of proof. They are required to provide evidence for their position, not the other way around (note: you may notice that these arguments also contain an argument from ignorance fallacy, that is because shifting the burden of proof is simply a special case of that fallacy).

Adam Savage internet minefield information

Via MythBusters Episode 187 “Bubble Pack Plunge”

Getting back to the original topic of simply telling people to Google something rather than actually providing them with the source, there is another serious problem there. Namely, Google is a mess. You can find websites supporting pretty much any quack position out there. So for any topic, you will be able to find “sources” supporting a given view, but those sources are often utter crap. As a result, simply telling your opponent to Google something is a terrible idea because they have no way of knowing specifically which sources you read. Imagine, for example, that you tell me that X causes Y and I should just Google it, so I do, but the first 20 or so results that I find are all from unreliable, quack websites. At that point, most people would probably stop looking, but the fact that the most popular hits are junk obviously doesn’t mean that there isn’t a good source out there somewhere. This is once again the problem of trying to prove a negative (i.e., trying to prove that a given source doesn’t exist). It is unreasonable to expect your opponent to spend hours looking for information to support your position. That is your job, not theirs.

Finally, knowing exactly which sources your opponent is basing their arguments on is extremely important, because if they are bad sources, then you can discredit their argument right there and then. To be clear, using bad sources doesn’t mean that the conclusion of the argument is wrong (that would be a fallacy fallacy), but it does mean that the argument itself must be abandoned until such time as reliable sources can be presented. Once again, the person making the claim must provide the evidence to support it, and until they do that, you are under no obligation to entertain their fanciful notions.

At this point, you might be thinking that is entirely unfair to expect people to have sources to back up all of their claims, and if you are thinking that, then that’s really too bad because this is simply how logical debates work. If you don’t have the sources to back up your claims, then you shouldn’t be debating people, because no one has to take you seriously until you provide your sources. It’s really that simple. Personally, I make lists and databases of sources on various topics, and every time that I read a useful new article, I archive it into my lists, that way I always have the sources ready to back up my positions. Microsoft Word or Excel are just fine if you want to make lists of URLs, but given that peer-reviewed publications are the relevant sources for most scientific debates, I recommend using a PDF organizer (I personally like Mendeley, which is free unless you need to store over 2GBs of PDFs).

The point in all of this is really quite simple: if you are claiming that something is true, then it is your responsibility to provide high quality sources to back up your position, and your opponent is under no obligation to refute your claims unless you provide those sources. Indeed, this entire post was summed up nicely by Christopher Hitchens in what has come to be known as Hitchens’ Razor, “What can be asserted without evidence can be dismissed without evidence” (or if you prefer the original Latin version, “Quod gratis asseritur, gratis negatur”).

Suggested further reading:
The Logic of Science — The Rules of Logic Part 5: Occam’s Razor and the Burden of Proof
RationalWiki — Burden of Proof
Science or Not — The reversed responsibility response – switching the burden of proof
Philosophy of Religion — The Burden of Proof
The Free Dictionary — Burden of Proof

Posted in Rules of Logic | Tagged , , , | Comments Off on Don’t tell people to “Google it.” That’s your job, not theirs

Debunking creationism: a visual comparison of “micro” and “macroevolution”

Posted on September 6, 2016 by Fallacy Man

Creationists often like to claim that there are two fundamentally different types of evolution: microevolution and macroevolution. They argue that microevolution does actually occur, but only produces small changes within a species or “kind” of animal. For example, most creationists are generally okay with the concept that all finches evolved from a common ancestor, all crows evolved from a common ancestor, all ducks evolved from a common ancestor, etc. However, they draw the line roughly at the taxonomic level of family (e.g., ducks are in the Anatidae family), and they argue that evolution beyond that level (what they call macroevolution) is impossible and has never and can never happen. Thus, they dismiss the notion that finches, crows, and ducks all share a common ancestor. I have written about this before and explained that this distinction is completely arbitrary and meaningless because the exact same evolutionary mechanisms that caused the evolution of finch species could (and indeed did) cause the evolution of all birds. In other words, macroevolution is simply the accumulation of microevolutionary steps, and one inherently leads to the other.

Since I have written a detailed explanation of the problems with creationists’ reasoning, I decided to take a different approach for this post and provide a visual explanation. The image below shows a hypothetical pathway through which turtles could have evolved from their lizard-like ancestors (several of these images are renderings of actual fossils: B6 = Milleretta, A15 = Eunotosaurus, C22 = Odontochelys, B30 = Proganochelys, D37 = Chelydra [modern turtles]; these are just screen shots from Dr. Tyler Lyson’s excellent video). This full progression is, of course, what creationists would consider to be macroevolution, and creationists are adamant that today’s turtle families were uniquely created and did not evolve from a lizard-like ancestor. However, because they accept microevolution, most creationists would have no problem with any particular pair of images, and they would accept that A1 could evolve into B1, B1 could evolve into C1, etc. In other words, each pair of images shows “microevolution” (which creationists almost universally accept), but when we string all of those steps together, we get “macroevolution” (which creationists say is impossible).

You can probably see where I am going with this, but just to be sure, I will state it explicitly. If you are going to say that macroevolution is impossible and turtles could not have evolved from lizard-like ancestors, then which step do you think is impossible? Please show me which step could not have occurred, and justify that claim. Additionally, please explain the obvious transitional fossils. Remember, B6, A15, C22, B30, and D37 are actual fossils, and they perfectly match the expectations for what a transitional fossil should look like (details here). So, if turtles and their lizard like ancestors were uniquely created kinds, then at what point in this progression do lizard-like reptiles end and turtles begin?

These images are simply screen shots from Dr. Tyler Lyson's video

These images are simply screen shots from Dr. Tyler Lyson’s video

 

Evolution is like the visible light spectrum (A) with each color (species) gradually changing into the next without a clear point at which one ends and the other begins. Creationists predict something like B, where each “kind” is unique and distinct form the other kinds. What we actually find in the fossil record is C. There are clearly transitionals that creationists say shouldn’t exist, but because the fossil record is incompletely, we will never have every single step in the spectrum.

At this point, some people will likely be inclined to ignore my questions and harp instead on the fact that this pathway is hypothetical, but that argument completely misses the point in several ways. First, this pathway is only partially hypothetical because B6, A15, C22, B30, and D37 are actual fossils that we have found. Additionally, of course the pathway is partially hypothetical. We will never find every single one of these steps, and we don’t need to in order to defeat creationism. Evolution is very much like the visible light spectrum. Each color gradually fades into the next color without a clear breaking point. In other words, there is a point along the spectrum that is clearly red, and there is a point that is clearly blue, and there is a point that is clearly violet, but there is a spectrum of change in between those points and it is not possible to pick an exact point where the blue ends and violet begins, just as you cannot pinpoint the exact step at which the reptile becomes a turtle as we know it.

The problem is that the fossil record is extremely incomplete. The conditions that are required for a fossil to from very rarely occur, and the vast majority of dead animals are eaten rather than fossilized. So only a remarkably small subset of animals ever become fossilized. Additionally, even if a fossil is formed, it has to survive for millions of years without being eroded away, and we ultimately have to find. As a result, the fossil record will always be incomplete and it is totally unreasonable to expect that we would find every single one of the steps illustrated above. However, what we have been able to find is plenty to refute creationism, because creationism claims that animals were created as distinct kinds. In other words, it does not predict a spectrum. Rather, it predicts that there should be blues and there should be violets, but there should not be intermediate steps. What we actually find in the fossil record is, of course, an incomplete spectrum with lots of intermediates, which is exactly what we expect from evolution. Look at A15 (Eunotosaurus), for example. It’s certainly not a modern turtle, but it’s not a lizard either. It has half the traits of both groups. It is precisely the type of intermediate that creationism says shouldn’t exist.

Finally, the argument that this pathway is meaningless because it is partially hypothetical misses the point because it is absolutely fine to use hypotheticals to defeat absolute claims. Creationists claim that macroevolution cannot happen, and this pathway shows that it can happen. In other words, to defeat the claim that macroevolution is impossible, I don’t need to prove that this pathway actually occurred; rather, I simply have to show that a pathway is possible, which it clearly is. We can of course do this for tons of examples of macroevolution. For example, scientists have known the steps involved in the evolution of an eye for a very long time, and a close examination of the structure of bacterial flagella has shown that it is entirely possible for flagella to have evolved by evolution.

In short, if you are going to insist that macroevolution is impossible, then I want you to look at the evolution of the turtle and tell me which step is impossible (and justify that claim). I also want you to explain the existence of the known intermediate fossils (without committing an ad hoc fallacy), and I want tell me the exact point at which modern turtles first appear. If you are going to comment in defense of creationism, then I expect an actual answer to those questions.

Note: Please read this post before bringing up the fundamentally flawed “irreducible complexity argument” (spoiler alert: it ignores the fact that evolution is blind and each step simply needs to be useful for something, rather than being useful for a particular end product).

 Note: Before anyone responds by saying that “Darwin himself said that the notion that an eye could evolve was ‘absurd,’” please realize that this argument misquotes Darwin. What he actually said was that saying that an eye could form naturally seems absurd…until you understand evolution. He went on to explain how the eye could have evolved via natural selection.    

 Related posts

Posted in Science of Evolution | Tagged , , , , , | 14 Comments

Who reviews scientific papers and how do reviews work?

Posted on August 29, 2016 by Fallacy Man

I spent my afternoon reviewing a paper for a scientific journal and making a recommendation about whether or not the paper should be published. As a scientist, this is not an uncommon task for me, but it is a process that is largely foreign to the general public. Indeed, the peer-review system often seems to be a mystery to those who don’t participate in it, and, as a result, it is a frequent topic for this blog. For example, I have previously written about what it takes to publish a paper. However, I have not yet written a post specifically about what it is like to be a reviewer or even who reviewers are. So, I thought I would take this opportunity to explain the process from a reviewer’s point of view and offer you a window into the system that determines which papers get published.

Who are reviewers and how are they chosen?
In short, reviewers are scientists. The peer-review system is rather interesting because essentially everyone involved with it acts as both an author and a reviewer. In other words, reviewers are themselves scientists who also submit papers for review. This is a good system, because it means that scientific papers are being reviewed by other scientists, not by politicians, corporations, etc.

Nevertheless, reviewers obviously aren’t chosen at random from within the scientific community. Rather, journal editors choose them based on relevant experience and expertise. This can happen in several ways. Often, journals require you to recommend reviewers when you submit your paper. In other words, as part of the submission process, you have to nominate several people to serve as reviewers, as well as providing contact information and (often) a justification for why they would make suitable reviewers. As a general rule, you want to suggest people who have published similar papers (often papers that you cited in the paper you are submitting) and who aren’t in anyway affiliated with you (this insightful post provides more details about how to select reviewers). Ultimately though, the editor has the final say, and he/she will consider your suggestions and make the final call.

Another common method that journals use to select reviewers is simply to recruit reviewers during the paper submission process (or via society memberships). In other words, the submission form often includes an optional check-box that says something to the effect of “I am willing to be a reviewer for this journal.” Generally, this is also accompanied by a section where you list your areas of expertise. Thus, if you are submitting to a herpetology journal (i.e., reptiles and amphibians) and you check the box and list yourself as a turtle expert, then you will go on the list of potential reviewers for turtle papers.

A third mechanism that editors use is simply to contact authors who recently published similar research in the journal that they edit. Indeed, after publishing a paper in a particular journal, it is quite common for an editor of that journal to contact you about reviewing a different paper for that journal (this has happened to me several times).

Finally, editors may choose simply to send the paper to someone who they know does research in a similar area or who was recommended to them by someone else. For example, the editor may have recently read a similar paper and contact one of the authors of that paper. Indeed, the paper that I reviewed today was in a journal that I have never published in or signed up to be a reviewer for, and the paper did not cite my previous work, so I doubt that the authors recommended me. However, I have previously published extremely similar and relevant research, so I suspect that I was selected by the editor because of my previous papers on this topic.

Regardless of how they are chosen, the point is that reviewers are generally experts on the topic of the paper that they are being asked to review. In other words, editors select scientists who have the necessary skills, knowledge, and experience to assess the paper and determine whether or not it is worthy of publication. Usually, at least two reviewers are selected per paper, but it is not uncommon to have three reviewers, and some journals use four or more.

Who can’t be a reviewer?
Most publishing scientists also serve as reviewers, but not all scientists are eligible to be a reviewer for a given paper. First, to be a potential reviewer, you have to have expertise on the particular field that the paper is about. I, for example, am predominately a herpetologist, so I would never get asked to review a paper on physics. For that matter, I wouldn’t even get asked to review a paper on botany. I might, however, be asked to review papers on other areas of zoology, but only if the topic of the paper was closely aligned with topics that I study (e.g., one of my areas of research is population ecology, and although I study the population ecology of reptiles and amphibians, the same concepts, statistics, etc. apply to other taxa, so I have the necessary skills and knowledge to assess a population paper on birds, for example).

A second criteria is a lack of conflicts of interest. Exactly how that is defined is variable, but as a general rule, reviewers should not be in anyway associated with the paper in question, and they should not be institutionally linked to any of the authors. I could not, for example, serve as a reviewer for a paper that was written by another graduate student in my lab, because that would be a conflict of interest (i.e., even though I would try to be objective, I would be less likely to criticize the paper because I work with and like the authors).

Finally, in many cases, authors can recommend people who should not serve as reviewers. Editors are under no obligation to follow these recommendations, but if there is someone who you really don’t want reviewing your paper, you can make that case. For example, if you have a long standing rivalry with someone and, as a result, that person likely would not be objective, then you can argue that he/she should not be selected as a reviewer. To be clear though, there needs to be a legitimate reason why that person would not make a good reviewer, and you can’t include someone in that list simply because you are concerned that they will find problems with your paper.

What happens when you receive a request to review a paper?
Each journal is different, but this is generally how things play out. First, you receive an email from an editor asking if you would be willing to review a given paper. This email usually includes the abstract for the paper, or, at the very least, the topic that the paper is on. It may also include other information such as the number of pages, words, figures, and/or tables in the paper. Sometimes this email will have information about the reviewing standards of that journal such as whether or not the review is anonymous and the time frame in which you are expected to provide the review.

As a reviewer, you then look at the information that you have been given and decide whether or not the paper is on a topic that you are sufficiently knowledgeable about, whether or not you have any conflicts of interest, and whether or not you currently have the time to review it. You then respond to the editor to tell them whether or not you are willing to be a reviewer. If you reject the request, it is generally considered good form to suggest an alternative reviewer (which is another way that editors identify potential reviewers).

Once you accept your duty as a reviewer, you generally receive a copy of the paper as well as paperwork on confidentiality. Some journals also provide you with a review template that you are supposed to fill out, whereas others let you provide feedback in whatever way you see fit. Generally, with either system, the final review will consist of several summary statements about your views on the paper, a list/explanations of your major criticisms, and a list of comments on specific lines (these are either annotations to the original document, or a list with corresponding line numbers).

What do reviewers look for?
Imagine that you are a reviewer, and you have just been given a paper to read. What are you going to look for? There are actually lots of things that you should look for, but the key thing to keep in mind is that the job of a reviewer is to assess the quality of the research rather than acting as an editor. In other words, reviewers do look at things like grammar, readability, and the presentation of the data (e.g. good/appropriate figures and tables), but their main duty is to act as a filtering mechanism that blocks bad science from getting published and helps authors to improve potentially useful research.

As such, the most important thing that reviewers look at is the methodology. They check things like sample size, experimental design, the statistics that were used, how the statistical models were set up, etc. All of this is intended to identify poor methodology and ensure that the study was done correctly. Following that, reviewers will check to see if the results were reported and discussed properly. It is not at all uncommon for authors to jump to conclusions that are not supported by the data, and it is the reviewers’ job to reign them back in and make sure that all of the conclusions are merited.

Additionally, reviewers are tasked with making sure that the paper is well grounded in the scientific literature. All research inevitably builds on previous research. As such, papers are supposed to cite and discuss relevant papers, especially if similar studies reached different conclusions. Thus, reviewers check to make sure that this was done appropriately. This is another reason why it is important for reviewers to be experts on the given field. As experts, they know the literature, so if important papers are missing, they will be able to point them out. Indeed, it is exceptionally common for reviewers to suggest specific papers that the authors should have cited (I made several such suggestions on the paper that I reviewed today).

Finally, it is the reviewers’ job to actually be helpful to the authors. Many otherwise nice people become utter jerks when reviewing, but the idea is actually for reviewers to provide constructive criticism that will help the authors improve their paper. Thus, reviewers are asked not only to point out problems with the paper, but also to suggest ways to fix those problems (e.g., “your statistical method is inappropriate, and you should use method X instead”).

What do reviewers recommend?
You have now read the paper and made a list of comments, which means that it is time for you to make a decision. Does this paper deserve to be published? You generally have 4–5 options to choose from (some journals don’t use option 2, and some journals use slight variations of these).

  1. Reject without the option to resubmit — This means that the paper is seriously flawed and will rejected without further consideration.
  2. Reject with the option to resubmit — This means that the paper has serious flaws, but it also has merit if those flaws can be correct. The authors can then revise the manuscript based on your comments and resubmit it back to that journal. At that point it goes back out for review, and you will often (but not always) be asked to review the revised paper.
  3. Accept pending major revisions — This means that the study has merit, but there are still some substantial issues that need to be addressed. If the authors can correct those errors to the editor’s satisfaction, then it will be accepted for publication without further review.
  4. Accepted pending minor revisions — This means that the paper is solid, but there are some minor issues that need to be dealt with before it will be published.
  5. Accepted in its current form — This is a theoretical state in which a paper is accepted without any changes being required. I’m not convinced that it ever actually happens to real papers (it definitely hasn’t in my experience as either an author or reviewer).

After you make and justify your decision, the editor will look at your comments as well as the comments made by the other reviewer(s), then make the final decision about the fate of the paper. This system of having several reviewers is another strength of the peer-review process, because even if one reviewer does a crappy job and misses major flaws, the other reviewer(s) are there to pick up the slack.

Why do scientists serve as reviewers?
It is worth mentioning that you don’t get paid to be a reviewer. It is entirely a volunteer service, and it is quite time consuming. I, for example, had a large data set that I was hoping to analyze this afternoon, but instead I spent six hours reviewing a paper, and I never got around to the data (and this was a fairly short paper). So you may be wondering why on earth do scientists do it. Why don’t we always just reject requests to review papers? I obviously can’t speak for every scientist, but I can tell you my views and why I do it and take it seriously, and I know many other scientists who feel the same way. So I can’t state anything statistical, but I suspect that this would be a common response.

First, it is simply reciprocity. Every time that I submit a paper, I am imposing on other scientists to take time out of their busy schedules and review my research. As such, it is only fair that I then take time out of my schedule to review other scientists’ research. If everyone tried to pass the buck, then all of the reviews would be being conducted by an increasingly small and disgruntled group of scientists, and that would be bad for everyone. So acting as a reviewer is really just paying my dues as a member of the scientific community.

Second, as a reviewer, I get to play an active role in ensuring the quality of the research in my field, and that is a duty that I take very seriously. I obviously care greatly about my field and the advancement thereof, and I want the papers in my area of research to be of the highest quality possible. So, by serving as a reviewer, I get to block flawed research, promote high quality research, and make recommendations about how to improve research. This lets me extend my influence and impact on my field far beyond my own publications, and I see that as both a duty and a privilege.

To put this another way, I think that being a reviewer is an enormous responsibility. I know what it takes to do original research. I know the amount of work and effort that is involved, and I know what it is like to have all of your work torn to shreds by a reviewer. So, when I receive a paper, I always want it to be good and publishable, because I want those scientists to be rewarded for their extraordinary effort. At the same time though, when I agree to review a paper, I accept responsibility for preventing bad research from being published, and that responsibility motivates me to make absolutely sure that the research is solid before I give it my stamp of approval.

Summary
In short, reviewers are simply other scientists who have been matched with a paper based on their experience and expertise. It is their duty to carefully examine the paper, determine whether or not it was conducted correctly, and recommend if it should be published or rejected. Scientists do not get paid for this service, but it is an important task that researchers tend to take seriously.

Note: Journals vary widely with regards to anonymity. In some cases, reviewers don’t know who the authors are and authors don’t know who the reviewers are. In other cases, reviewers have the authors names, but authors don’t know who the reviewers are, and in yet other cases, both parties know the identities of the other party, and some reviews are even made public. There is an interesting debate about which system is best, and I had planned on going into it, but this post became longer than intended, so I will save that topic for a later post.

flowchart diagram how to publish scientific peer-reviewed paper blog

This flowchart summarizes the steps required to publish a peer-reviewed paper. See this post for details.

Posted in Nature of Science | Tagged | 10 Comments

Measles is not better than autism: Debunking anti-vaccine arguments

Posted on August 23, 2016 by Fallacy Man

Over the weekend, I was unfortunate enough to come across an article by Jaclyn Harwell at “Modern Alternative Health” called “5 Reasons That Measles is Better Than Autism.” Unsurprisingly, it was full of misinformation and shoddy arguments. Indeed, it was so full of counterfactual claims and dishonest distortions of reality that I felt compelled to write a rebuttal, especially since the faulty arguments contained in the post are prevalent among antivaccers. Therefore, I am going to dissect that post and explain why it is nonsense. Before I get to Jaclyn’s “5 reasons,” however, I need to deal with several serious problems in the opening statements of the article.

First, this post is fundamentally flawed because the entire thing is based on the false dichotomy that you have to choose between because vaccines and autism. In reality, of course, vaccines do not cause autism. As I explained at length in this post, numerous enormous studies have tested the vaccine/autism hypothesis and failed to find any evidence of vaccines causing autism. The only studies that suggested that vaccines cause autism were tiny and riddled with problems. Anecdotes don’t matter, deceptive documentaries are irrelevant, and the “CDC whistle-blower” did not present any actual evidence of vaccines causing autism (or of CDC corruption, for that matter). Only scientific evidence matters, and science is overwhelmingly on the side of vaccines.

measles isn't harmless meme anual deaths

Second, the core argument throughout the article is that measles is a mild illness, but that argument is a blatant lie. I’ll deal with this at length under reason #5, but briefly, in developed countries with access to modern medicine, measles still has a death rate of 1 in 1,000 infected individuals. Nevertheless, it is true that in terms of sheer numbers, there are few measles deaths in industrialized countries today, but that is only because of vaccines! Indeed, in the US, in the 10 years prior to the introduction of the measles vaccine, measles killed an average of 440.3 children each year in the US alone (details and sources here; also note that the death rate per infected individual was roughly 1 in 1,000 then as well). Further, when we look at the entire world, measles still kills well over 100,000 people annually, and the World Health Organization describes it as, “one of the leading causes of death among young children” (WHO 2016). That’s not fear-mongering, that’s a fact. So while anti-vaccers want you to think that measles is trivial, actual epidemiologists have a very different view of this disease.

Further, even when children don’t die from measles, there is still a 1 in 10 chance that they will develop an ear infection (which can cause permanent hearing loss), a 1 in 20 chance that they will develop pneumonia, and a 1 in 1,000 chance that they will develop encephalitis (a swelling of the brain; CDC 2015a). There are also other complications such as febrile seizures in 0.1–2.3% of patients (Orenstein et al. 2004). Indeed, during a large outbreak in France, nearly 25% of victims had to be hospitalized (Antona et al. 2013), and in the US from 2001–2008, 40% of measles patients were hospitalized (CDC 2011). You simply cannot describe a disease that hospitalizes 25–40% of its victims and kills over 100,000 people annually as a minor illness (WHO 2016). That is extremely dishonest (see reason #5 for a more thorough explanation of just how deadly this disease truly is).

measles outbreaks, low vaccine rates

Please carefully note how the measles outbreaks are centered around the communities with low vaccination rates. Image from Knol et al. 2013

Third, the author claims that, “in the majority of outbreaks, most affected people have, indeed, been vaccinated.” This is an extremely common anti-vaccine argument, and it is horribly flawed. First, in the case of measles the claim itself isn’t even true. For example, in the 2001–2008 study that I mentioned earlier, 89% of patients were unvaccinated (CDC 2011). Similarly, during an enormous outbreak in France that involved over 20,000 people, 80% of patients were unvaccinated (Antona et al. 2013), and during a large outbreak in the Netherlands, 91.7% of patients were unvaccinated (Knol et al. 2013). Those are not cherry-picked examples. Rather they are the norm for measles outbreaks.

Additionally, and far more importantly, this argument ignores rudimentary math. You have to look at the proportions not the raw numbers. Most people are vaccinated, so of course many outbreaks will affect vaccinated people. Simply looking at the raw number of cases doesn’t tell us whether or not vaccinated people actually get the disease more often, and when we look at the actual rates, we find that infection rates are substantially higher among the unvaccinated (King et al. 1991; Schmitz et al. 2011). To give a completely analogous example, most car accidents involve sober drivers, but that doesn’t mean that driving drunk isn’t dangerous. Most people drive sober. Therefore, in terms of raw numbers, most accidents involve sober drivers. However, when you look at the rates, you find that the rate of accidents is far higher among drunk drivers than among sober drivers.

Finally, the author insinuates that “the efficacy of vaccinations is questionable at best.” Again, this is simply not true, especially for the measles vaccine. Even beyond the examples that I cited early of outbreaks corresponding to unvaccinated communities and higher measles rates among the unvaccinated, many other studies have examined the efficiency of the measles vaccine and it is quite high (93% with one dose and 95–97% with two doses; King et al. 1991; CDC 2015b). For example, Clemens et al. (1988), found that introducing the measles vaccine into a population reduced the measles death rates by 57%, but please go ahead and say that the vaccine isn’t effective or important (note the immense sarcasm).

Now that we are clear on the actual facts, let’s look at the five arguments in the article. My intention is to address the original article point by point, but the original jumped around quite a bit and didn’t follow a great logical structure. As a result, I thought that it might be useful to provide a bullet list of key points and the sections in which they are discussed.

  • Vaccines don’t cause autism (introduction, #1)
  • Getting measles to avoid getting measles is idiotic (#1)
  • Measles actually weakens the immune system (#2)
  • Neither measles nor autism can be cured, but measles can be prevented (#4)
  • Total measles mortalities are relatively low because of vaccines (#1, #5)
  • Measles still kills over 100,000 people annually (introduction, #1, #5)
  • Without vaccines measles would kill over 1 million people annually (#5)
  • Without vaccines, at least 0.35 in 100 children under the age of five would die from measles each year (#5)

Note: To be clear, if you want to say that surviving a measles infection without any serious or lasting consequences is better than a life-long affliction with severe autism, fine, I’m not going to disagree with you. What I take issue with is the dishonest and misleading claims about measles mortality rates, the dangerous and unmerited vilification of vaccines, and the damaging way that autism is presented. We need to stop talking about autism as if it is the world’s worst disease and as if people with autism are inferior, damaged, and in need of repair. That is an indignity that autistic people should not have to suffer. Autism is not the worst thing that could happen to your child.

 

Bad reason #1: “Measles imparts lifelong immunity.”
I have repeatedly had to address this argument from anti-vaccers (for example here and here), which is frustrating because it is such an obviously ridiculous argument. We can rephrase this argument simply as, “getting measles is good because it prevents you from getting measles.” Think about that for a second. This argument i

s actually proposing that you should avoid getting sick by getting sick. In contrast, a vaccine will prevent you from ever getting sick. In other words, a vaccine prevents you from ever getting measles, whereas measles only prevents you from getting measles a second time. Using a measles infection to protect you from a measles infection is like using pregnancy as a contraceptive.

To be fair, some studies have found that vaccine-induced immunity does not last as long as natural immunity (Christenson and Bottiger 1994), but other research has found that both forms last equally well for many years (Jokinen et al. 2007), and since measles is predominantly a childhood disease, those first years are the really important ones. Additionally, the longevity of immunity can easily, safely, and effectively be extended with boosters, which, once again, prevent you from getting measles in the first place. Further, I have already provided multiple citations showing that the measles vaccine is very effective and infection rates are much higher among the unvaccinated. In other words, this anti-vaccine argument makes it sound like getting measles is the best way to avoid getting measles, but that is clearly ridiculous. The vaccine is by far the best way to avoid ever having to suffer through measles.

Jaclyn goes on to reiterate that measles “isn’t a big deal” by citing the fact that between 1950 and 1960 the death rate for the US was less than 1 per 100,000 individuals (for the entire population, not per infected individual). As I showed earlier, however, that comes out to well over 400 deaths annually, which is in fact a big deal (also see #5 for an explanation of why the 1 in 100,000 figure is misleading). Even if vaccines caused autism (which again, they don’t), a life of autism is not worse than death by measles.

She also claims that there were no deaths among the 1,153 US cases of measles from 2001–2013. First, that claim is not true. There were two measles deaths from 2001–2003 (CDC 2004), another two in 2009 (Kockanek 2011), and another two in 2010 (Murphy 2010). Nevertheless, the death rates are low, but this argument totally ignores the fact that the reason that we have so few deaths is because we have so few cases of measles, and the reason that we have so few cases is because we have vaccines! Vaccines are the only reason that we don’t have thousands of measles deaths each year. Indeed, it is estimated that between 1994 and 2013, the measles vaccine prevented >70 million cases of measles in the US, which comes out to a total of 57,300 deaths according to the calculations used by the authors (Whitney et al. 2014). That number may, however, be higher. If, for example, we simply apply the normally accepted 1 in 1000 death rate, then it would be 70,000 deaths (3,500 measles induced deaths annually). Either way, the point is that the vaccine prevents thousands of deaths each year, and you simply cannot pass that off as a minor thing.

Finally, she quotes a “study” which said that, “The mass of scientific evidence compiled by researchers clearly indicates that the incidence of autism occurs following vaccination and is most closely associated with the schedule of vaccines culminating in the MMR vaccine” (Ewing 2009). However, that “study” was not actually a study. It wasn’t even a proper review. It was an entirely speculative opinion piece that made one false claim and unmerited assumption after another. It was based on a correlation fallacy, and it never once provided actual evidence to support the quoted statement. Further, it ignored the vast body of large studies that failed to find any evidence of vaccines causing autism. In other words, all that this paper did was propose a hypothesis, but that hypothesis has already been thoroughly tested and falsified.

Indeed, even the first sentence of the paper is demonstrably false. It says, “that the occurrence of autism has risen steadily in the last decades is not in dispute.” In reality, the rise in autism rates is very much in dispute, with many studies concluding that it is at least largely due to a change in diagnostic criteria rather than an actual increase (Rutter 2005; Taylor 2006; Bishop et al. 2008; Baxter et al. 2015; Hansen et al. 2015). When I first looked at this paper, I was baffled by how a paper that opened with such a clearly false statement could possibly have passed peer-review. Then, I checked the journal it was published in, and it was a journal that that is so minor that it doesn’t even have an impact factor (in other words, the scientific community doesn’t take it very seriously). You should always be wary of journals like this.
Bad reason #2: “Measles strengthens the immune system.”
No it doesn’t. I explained this in detail here, but in short, after a measles infection, your body produces antibodies that are specific for measles. So it only “strengthens” your immune system in that you can’t get measles again. It doesn’t help you fight any other infections. Further, as I explained under #1, thanks to vaccines, you can get those exact same antibodies without actually getting measles.

Further, recent research has shown that measles infections are so hard on your immune system that it actually takes up to 2–3 years for a child’s immune system to return to normal functional levels (Mina et al. 2015). In other words, measles infections weaken the immune system for several years, and these weakened immune systems lead to infections and deaths that aren’t generally attributed to measles. As a result, the actual measles death tolls are higher than we realize (not to mention that these additional infections often come with lasting side-effects).

Finally, it is true that associations with some microorganisms help to prevent autoimmune disorders, but that is actually because those organisms “train” the immune system not to over-react. So they aren’t “strengthening” the immune system, they are training it give a reduced response (a “strengthened” or “boosted” immune system is actually what causes autoimmune disorders). Further, the microbes that do this are generally beneficial or benign, not pathogenic (Gaurner et al. 2006). So there is no evidence that a measles infection will help you out later down the road.

 

Bad reason #3: “Autism causes long-term damage.”
I certainly don’t deny that autism can cause life long-problems (though I would like to stress that autism is a spectrum of disorders, and many people who are clinically diagnosed as autistic are very high functioning and live normal lives). However, in this section, Jaclyn reiterates her claims that measles isn’t dangerous, a measles infection gives life-long immunity to measles, etc. This section is really just a rehash of previous sections, so I won’t belabor the point.

 

Bad reason #4: “Measles is easier to cure than autism.”
I’m not arguing against the core statement here, but the pseudoscience that she uses to try to back up that statement is deplorable. For example, she opens with, “There’s actually no cure for measles. Know why? Because it’s a benign childhood disease” (her emphasis). First, there is no cure because it is a viral infection, and they are notoriously hard to cure (that is why we are also lacking cures for HIV, H1N1, the common cold, etc.). Second, another reason that there is no cure is that most people aren’t looking for a cure, because we have vaccines. In other words, we know that vaccines work very well, so it is more effective to invest money in increasing vaccine coverage, rather than in looking for a cure. Third, once again, measles has a death toll of over 100,000 people annually, and, as I’ll explain in #5, without vaccines that number would be well over 1,000,000. You simply cannot describe that as “benign.”

Next, she further tries to downplay the severity of measles by claiming that treatment with vitamin A reduces death rates by 62%. You’d think that the fact that she is talking about reducing death rates would make her rethink her claim that measles is “benign,” but apparently not. Nevertheless, there is some evidence that vitamin A is useful in overcoming a measles infection. However, citing that 62% figure without context is very deceptive, because it comes from a meta-analysis of studies in Africa where vitamin A deficiencies are major problems (Sudfeld et al. 2010). In industrialized counties, however, most of us have plenty of vitamin A, and vitamin A is a standard part of measles treatments (CDC 2016a). In other words, that 1 in 1,000 death rate that we talked about early already includes vitamin A treatments, but if you read Jaclyn’s post, it sounds like vitamin A is a way for you to improve those odds.

Finally, she claims that autism can be cured, which is a dangerous way to give parents false hope. There is currently no cure for autism (after all, it’s largely genetic). There are ways to help manage it, but not cure it (please don’t flood the comments with anecdotes, because anecdotes are not valid scientific evidence. If you don’t have large, properly controlled clinical trials that were published in reputable journals, then you don’t have any evidence).

Note: If you want to be pedantic, neither measles nor autism can be “cured” because your body, not a medication, is actually what fights the measles infection. So if we are going to succumb to pedantry, the core claim of this argument isn’t technically true.


Bad reason #5: “Measles complications are uncommon.”
Much of this section is a regurgitation of previous arguments, so I will focus on the novel parts. The first of which is the claim that 1 in 45 children in the US have autism, coupled with the claim that 1% of the entire world’s population is autistic. I don’t have too much trouble with the 1 in 45 claim because it came from a legitimate study (though it is worth noting that the estimate that is currently accepted by health organizations is actually 1 in 68 children in the US; CDC 2016b), but the 1% figure is misleading. No source is given, but the source that I found lists it as 0.6%, not 1%, and when we are dealing with the world’s entire population, that rounding error is substantial (roughly 28 million people; Elsabbagh et al. 2012). Also, I have a serious problem with the way that those numbers are being used here. Jaclyn is clearly trying to use them to argue that autism is a very common thing, and, therefore, should be feared. However, autism is a spectrum of disorders, and early in the post, Jaclyn said that her post was “referring to children with severe, regressive cases of autism, not those that are high-functioning and more self-sufficient.” Do you see the problem? It is extremely dishonest to cite the overall autism rate if you are only talking about the most extreme versions of it. In other words, 0.6% of the world’s population has some form of autism, but only a small subset of that 0.6% has the type of severe autism that this post was supposedly about. So this is yet another instance of her using numbers dishonestly in a failed attempt to support her flawed position.

Next, she makes the argument that most measles deaths are actually from developing countries that have poor sanitation standards. The claim itself is true, but the insinuation that sanitation standards are the cause of the infections is false. The deaths are from those countries because they are the ones that don’t have vaccines. As I explained here and here, the sanitation standards in the US were essentially the same in the 50s as they are now, yet they still had hundreds of measles deaths each year. Why? Because they didn’t have vaccines. Similarly, when we introduce vaccines into countries with very poor sanitation standards, the death rates drop (Clemens et al. 1988). Why? Because vaccines work (King et al. 1991; CDC 2015b). Further, as I explained earlier, industrialized countries continue to have large measles outbreaks when vaccination rates drop, and some of those outbreaks result in unnecessary deaths (Antona et al. 2013; Knol et al. 2013).

She then goes on to act as if the 100,000+ annual deaths from measles aren’t really a big deal because they only represent a small portion of the total population. First, I find that attitude unconscionable, since those deaths are preventable. Second, vaccines are the only reason that the death rate is so low! Even if we back the clock up to the year 2000, the death toll was 546,800 (WHO. 2016). Why did it drop so rapidly between then and now? Because we have had a massive vaccination campaign and have taken vaccines to many developing countries, including ones that lack sanitation. Indeed, it is estimated that between 2000 and 2014, the measles vaccine prevented over 17.1 million deaths (WHO 2016). That’s 1.14 million per year. So you simply cannot present the current number of mortalities as evidence that we shouldn’t vaccinate, because the current numbers are low because of vaccines.

Additionally, there is a huge problem here because the autism rates and measles death rates aren’t exactly comparable. For example, she claims that 1% of the world’s population has autism (that should be 0.6%), whereas only 0.002% of the world’s population dies from measles each year. Her argument seems to be that the odds of having autism are far greater than the odds of dying of measles, but that is not a fair comparison because measles deaths are not distributed evenly by age, and roughly 55% of deaths occur in children under 5 (Orenstein et al. 2004). So when you want to look at risk, you can’t simply divide the death rate by the world’s entire population (as she did) because children that die at a young age aren’t recruited into the older age groups. Thus, her calculations result in a gross underestimate of the mortality risk. Let’s get around that problem by looking at the risk that a child will die from measles before reaching five years of age, which is the time period during which 55% of measles mortalities occur (Orenstein et al. 2004).

There are currently roughly 177 million children under the age of five (census.gov/popclock), which gives us a mortality rate of 0.036% per year*. You might think that this means that there is a 0.6 in 100 chance of developing autism (using world-wide data), and a 0.036 in 100 chance of dying of measles before age five (using world-wide data), but that’s not actually fair either, because that estimate uses the death rates given our current vaccination status. If we remove vaccines, we jump to 1.14 million deaths per year, and our risk of death for children under five jumps to 0.35 in 100. At this point the mortality and autism rates are quite similar, but we aren’t even done yet, because Jaclyn specified at the beginning that she was talking about severe autism. I don’t know exactly what she counts as “severe,” but let’s assume that it is the top 50% of cases. Well then guess what, at that point, the autism risk for a child is 0.3 in 100 whereas the risk of measles death before the age of five is 0.35 in 100** (i.e., death from measles is more likely). Now, obviously the exact numbers will vary by country and what you count as “severe autism,” but the point is that when you actually do the math, in the absence of vaccines, the measles death rate would be very close (probably even greater) than the current rate of severe autism. Also note that this is just the probability of death prior to the age five, but 45% of measles deaths happen at later ages, so the overall mortality risk is actually much higher.***

In other words, if you want to agree with Jaclyn’s argument, you have to argue that death by measles is better than a life with severe autism, because if everyone stops vaccinating, then the risk of a child dying from measles will be roughly equal to the current risk of a child developing severe autism. So that (according to Jaclyn) is what you are choosing between: death or life with a disability. In reality, of course, you don’t have to choose because vaccines don’t cause autism. So your children can enjoy an exceptionally low risk of death by measles without it affecting their risk of developing autism.

*Note: I calculated the risk by taking the number of annual measles deaths (114,900; WHO 2016), multiplying by the proportion of mortalities that occur in children under five (0.55), dividing by the number of children on the planet (177 million), then multiplying by 100 to convert it into a percentage. The calculation for the death rate without vaccines was identical but it used 1.17 million deaths instead of 114,900.

**Note: the risk of death is actually much higher than I described here because I left out another really important factor. Namely, the autism risk is estimated per child (you only “get” autism once), but the measles deaths are per child per year, so there are four opportunities for death. I left this out of the calculations because the math is convoluted since most children only get measles once. As a result, the probabilities change each year, and the math was more intense than I felt like explaining here, but I thought it was worth at least mentioning that my estimate is actually a gross underestimate. (I’ve found that few people bother to read math-heavy posts, which is quite unfortunate).

***Note: I was too lazy to calculate the overall mortality risk, but if anyone feels like doing it, it is a cumulative probability. So you calculate the probability for each age class, then sum those. You have to do it this way because anyone who dies of measles in one age class is automatically eliminated from all subsequent age classes. In other words, if you try to do the math using a large age range (say 1–20) you won’t get reliable results, because the mortality risk for a 20 year-old is vastly different from the risk for a 1 year-old and, as a result, many 1 year-olds die and never make it to 20. The best way to think about this is that for each age bracket, you are calculating the mortality risk given that you survived to enter that age bracket. Thus, the mortality risk for children under five is quite high, but if you survive to age five, then the mortality risk before reaching age eight is much lower, and the overall mortality risk for a 0–7 year old is the sum of the risk from ages 0–4 and the risk from ages 5–7.

 

Conclusion/summary
In short, measles is in fact a very serious disease. It currently kills over well over 100,000 people annually, and it is estimated that without vaccines that number would be over one million. Further, you have to remember that the majority of measles deaths occur in children under five years old, so without vaccines, we would expect a minimum of 0.35 deaths per every 100 children age 1–4. That is not something that should be taken lightly or described as “benign.” Additionally, this entire post is based on a false dichotomy, because vaccines don’t cause autism. So you don’t have to choose between the risk of a measles death and the risk of autism, because getting vaccinated does not increase your chance of developing autism. In short, this post made one unscientific, misleading, and downright dishonest claim after another. The measles vaccine is extremely safe and effective, and you should not give in to the baseless fear-mongering.

Related Posts

Citations

  • Antona et al. 2013. Measles elimination efforts and the 2008-2011 outbreak in France. Emerging Infections Diseases 19.
  • Baxter et al. 2015. The epidemiology and global burden of autism spectrum disorders. Psychological Medicine 45:601–613.
  • Bishop et al. 2008. Autism and diagnostic substitution: evidence from a study of adults with a history of developmental language disorder. Dev Med Child Neurol 50: 341–345.
  • CDC 2004. Epidemiology of Measles—United States, 2001–2003. Morbidity and Mortality Weekly Report 53:713–716.
  • CDC. 2011. Measles — United States, January–May 20, 2011. Morbidity and Mortality Weekly Report 60:666–668.
  • CDC. 2015a. Measles (Rubeola): Complications of Measles. Accessed 20-Aug-16.
  • CDC. 2015b. Vaccines & Immunizations: Measles–Q&A about Disease & Vaccine. Accessed 20-Aug-16.
  • CDC. 2016a. Measles (Rubeola): For Healthcare Professionals. Accessed 21-Aug-16.
  • CDC. 2016b. Autism Spectrum Disorder (ASD): Data & Statistics. Accessed 21-Aug-16.
  • Christenson and Bottiger. 1994. Long-term follow-up study of rubella antibodies in naturally immune and vaccinated young adults. Vaccine 12:41–45.
  • Clemens et al. 1988. Measles vaccination and childhood mortality in rural Bangladesh. American Journal of Epidemiology 128:1330–1339.
  • Elsabbagh et al. 2012. Global prevalence of autism and other pervasive developmental disorders. Autism Research 5:160–179.
  • Ewing. 2009. What is regressive autism and why does it occur? Is it the consequence of multi-systemic dysfunction affecting the elimination of heavy metals and the ability to regulate neural temperature? North American Journal of Medical Sciences 1:28–47.
  • Gaurner et al. 2006. Mechanisms of disease: The hygiene hypothesis revisited.  Nature Clinical Practice Gastroenterology and Hepatology 3:275–284.
  • Hansen et al. 2015. Explaining the increase in the prevalence of autism spectrum disorders: the proportion attributable to changes in reporting practices. JAMA Pediatrics 169:56–62
  • Jokinen et al. 2007. Cellular immunity to mumps virus in young adults 21 years after measles-mumps-rubella vaccination. Journal of Infectious Diseases 196:861–867.
  • King et al. 1991. Clinical efficacy of measles vaccine during the 1990 measles epidemic. Paediatric Infectious Disease Journal 10:883–888.
  • Knol et al. 2013. Large ongoing measles outbreak in a religious community in the Netherlands since May 2013. Eurosurveillance 18.
  • Kockanek. 2011. Deaths: Preliminary Data for 2009. National Vital Statistics Reports 56.
  • Mina et al. 2015. Long-term measles-induced immunomodulation increases overall childhood infectious disease mortality. 6235:694–699.
  • Murphy et al. 2010. Deaths: Final data for 2010. National Vital Statistics Reports 61.
  • Rutter. 2005. Incidence of autism spectrum disorders: changes over time and their meaning. Acta Paediatr 94:2–15.
  • Orenstein et al. 2004. The clinical significance of measles: a review. Journal of Infectious Diseases 189:S4–S16.
  • Schmitz et al. 2011. Vaccination status and health in children and adolescents findings of the German health interview and examination survey for children and adolescents (KiGGS). Dtsch Arztebl Int 108:99–104.
  • Sudfeld et al. 2010. Effectiveness of measles vaccination and vitamin A treatment. International Journal of Epidemiology 39:i48-i55.
  • Taylor et al. 2014. Vaccines are not associated with autism: and evidence-based meta-analysis of case-control and cohort studies. Elsevier 32:3623-3629.
  • Whitney et al. 2014. Benefits from immunization during the vaccines for children program era — United States, 1994–2013. Morbidity and Mortality Weekly Report 63:352–355.
  • WHO. 2016. Measles. Accessed 20-Aug-16.
Posted in Vaccines/Alternative Medicine | Tagged , , , , | 106 Comments