The overwhelming consensus on climate change

Posted on January 20, 2020 by Fallacy Man

The climate is changing, and we are the primary cause. These are simple facts that are supported by a vast body of evidence and agreed upon by virtually allcr experts. Nevertheless, many people continue to think that the science isn’t “settled” and there is widespread disagreement among experts. Unfortunately, these myths have been propagated and supported by very active misinformation campaigns, so I want to take a few minutes to explain why they are incorrect. First I will explain what we mean when we say that a topic is “settled” or that there is a “consensus,” then I will demonstrate that such a consensus exists for the topic of anthropogenic climate change.

“Settled” science

First, I need to explain what I mean by “settled science,” because there are many people who argue adamantly that science is never “settled” because it is always possible that some future discovery will overturn the current thinking. That is technically true, but it can be misleading and requires clarification.

It is true that science, by its very nature, does not provide “proof.” Rather, science shows us what is most likely true given the current evidence. So to that extent, it is true that science is never 100% “settled,” because it is always technically possible there is something we have missed. However, there is a huge difference between a technical possibility and practical doubt. For example, it is technically possible that we are wrong about smoking causing cancer. It is technically possible that all of the countless studies on smoking and cancer are wrong and smoking is actually safe or even beneficial. Further, you can even find a handful of doctors that argue that we are wrong about smoking causing cancer. Does that mean that the science isn’t “settled” or that there is serious debate on the topic? Of course not! The topic has been so well studied so many times by so many people that the odds that we are wrong are insanely low. They are so low that for all intents and purposes, we can treat them as if they are zero. The notion that smoking causes cancer is “settled” in the sense that it is supported by such a massive and consistent body of evidence that it is extraordinarily unlikely that it is wrong, and we must act as if it is correct until such time as compelling evidence arises to the contrary.

This is true for a very large number of scientific topics. There are many things that have been so thoroughly studied that they are as close to “settled” as science can possibly come, and it does not make sense to talk about them as if there is any practical doubt. It would be absurd, for example, for a politician to say, “science is never settled, therefore we can’t really be sure that smoking causes cancer.” The link between smoking and cancer is “settled” in the sense that it is supported by such a vast body of evidence that it is extraordinarily unlikely that we are wrong about it. As I will demonstrate, the same is true for climate change.

See these posts for more information about “settled science.”

Note: please read to the end before arguing that there used to be consensus that smoking was safe. That is a myth and I deal with it below.

What is a scientific consensus?

There are really two different levels at which we can talk about a consensus, and this can become confusing because most people are bad at specifying the level at which they are talking (I have been guilty of this myself). At one level, there is a consensus of experts. In other words, this exists when the vast majority of experts agree on something. This is what most people think of when they think of a scientific consensus, but it is not actually the best level to look at.

You see, when we say something like, “this is a fact” or “the science is settled,” we aren’t basing that on a consensus of experts, but rather a consensus of evidence (i.e., a large body of studies that all agree with and support each other). The consensus of experts is a secondary by-product of the consistent body of evidence. This is really the level we need to look at when asking questions like, “is there any serious debate on topic X.” Science is not a democracy. It is about evidence, not authority. So simply finding some people with advanced degrees who disagree with X does not mean that there is serious scientific debate about the topic. Rather, if there is serious debate, it will be reflected in the peer-reviewed literature, because people will be publishing papers presenting evidence that X is not correct. So that is really the level we should focus on when we talk about a scientific consensus: the evidence, not the experts.

Having said that, there is value in having a consensus of experts when it comes to the general public. No one can be an expert on everything, so even though we would ideally always look for a consensus of evidence, there are many topics on which a given individual simply is not equipped to do that (this is true for everyone, myself included). So, when encountering those topics, it makes sense to look for a consensus of experts, because, on average, an expert will know more about the topic of their expertise than a layperson will, and a consensus of experts usually reflects a consensus of evidence.

I do want to pause here for a second to emphasize the “on average” bit of my last statement. On pretty much any topic, you can cherry-pick an expert who holds an extreme position. You can find doctors who think HIV doesn’t cause AIDs, immunologist who think vaccines are dangerous, etc. That does not mean that there is not a consensus of experts on those topics. You won’t find 100% agreement among experts on just about any topic. So that is not the standard by which we assess a consensus of experts. Further, the fact that you found an expert who agrees with you absolutely does not mean that your position is legitimate or scientifically valid. It is always possible to cherry-pick experts who agree with you, and it is imperative that you avoid falling into that trap. If you go to 100 doctors and all but one of them says you have cancer, you shouldn’t trust the one who disagrees and proclaim that doctors just aren’t sure about your diagnosis. It is intuitively obvious that you should listen to the 99 who said you have cancer.

See this post for the difference between deferring to experts and appealing to authority.

 See this post for more information on why you should avoid cherry-picking experts, including discussions of some of the handful of climatologists who deny anthropogenic climate change.

The consensus on climate change

With the semantics now out of the way, let’s look at climate change. The most famous (or infamous depending on your point of view) study looking at the consensus on climate change is the Cook et al. 2013 study that produced the 97% statistic that we have no doubt all heard. This paper has been widely criticized by the good people of the internet, mostly for invalid reasons. Nevertheless, there are some issues that are worth talking about (I have talked about it at length here, so I’ll be brief).

In short, this study did not actually look at a consensus of experts, but rather looked at a consensus of evidence (studies), from which a consensus of experts (authors) was inferred. It took 11,944 papers (by 29,083 authors) and scored them based on whether or not their abstracts stated agreement, uncertainty, or disagreement with the notion that humans are causing climate change. They found that of the papers that made an explicit statement about humans causing climate change (agree, uncertain, or disagree), 97% agreed that we are causing climate change.

That seems like pretty straight forward and compelling evidence of a strong consensus of evidence, so why the hoopla over this study? It mostly comes from the large number of papers that did not explicitly express a view one way or the other. 62.5% did not express an opinion, so the 97% agreement figure comes only from the subset of papers that did express an opinion, and that fact has drawn criticism from both sides.

On one hand, climate change deniers often erroneously claim that Cook et al. “threw out” nearly two thirds of the studies, and actually only 36.9% of the papers agreed that we are causing climate change. This is a faulty argument because it tries to treat papers that did not express a view as if they expressed uncertainty or disagreement, which is clearly false. If they did not express a view, then we cannot draw any conclusions about whether or not the authors agree with anthropogenic climate change, therefore they must be removed before calculating percentages. If we translate this into a more standard survey, if you sent a survey to 10,000 people asking if they thought smoking causes cancer, and 1% said it does not, 49% said it does, and 50% didn’t respond, you would not conclude that only 49% of people think smoking causes cancer, because that would assume that the 50% who didn’t respond are all either uncertain or think that smoking does not causes cancer, which is clearly absurd. It’s the same thing with the Cook et al. study. You can’t conflate “did not express an opinion” with “does not have an opinion” or “disagrees with the consensus,” yet that is exactly what this criticism does.

Further, a lack of explicit statement is precisely what we’d expect on a topic that has reached a consensus. Most studies on vaccines, for example don’t include a statement of “vaccines are safe” in their abstract, because that is so well-established that there is no need to explicitly state it (abstracts have tight word limits). Nevertheless, by the criteria used in this study, we would relegate those studies to the “did not express a view” category, even though the authors almost certainly think that vaccines are safe. Indeed, Cook et al. actually found that the number of explicit statements decreased over time, which is exactly what we expect from a growing consensus.

This does, however, lead to the other criticism. Namely, that Cook et al. actually severely underestimated the consensus because most of the papers in the “did not express a view” category probably were by people who do actually accept anthropogenic climate change. This is actually a fair criticism (though it is often stated too forcefully and unnecessarily denigrates the work of Cook et al.). An awful lot of those studies were on the impacts of climate change, and it is quite a stretch to think that most of those authors disagree with fundamental scientific facts about climate change. Indeed, many of those papers were by authors who are known to agree with anthropogenic climate change, and some of their other papers were categorized into the “accepts the consensus” category. So, it does seem extremely likely that Cook et al. underestimated the consensus.

To solve this problem, James L Powell took a different approach. He argued (I think correctly) that if there is actually disagreement on a topic, that disagreement should be prominent in the literature. It should be easy to find papers that explicitly reject anthropogenic climate change, whereas if there is a consensus, most papers simply won’t make an explicit statement one way or the other (just as they don’t for most “settled” topics). Therefore, if we want to look for a consensus, we should simply count the number of papers that explicitly reject anthropogenic climate change. As an example, he cited 500 recent studies on plate tectonics, none of which either explicitly endorsed or rejected the theory. Based on the Cook et al. criteria, this would erroneously lead to the conclusion of no consensus, whereas based on the criteria of explicit rejection, we would correctly conclude that there is a strong consensus.

scientific consensus on global climate change, global warming

Image via Powell

When we apply this rejection criterion to the climate change literature, we find almost no studies that argue against the position that humans are causing climate change. For example, Oreskes (2004) reviewed 928 papers published between 1993 and 2003 and failed to find a single one that rejected anthropogenic climate change. Similarly, Powell has looked at this at several time points, always with the conclusion of a very strong consensus. For example, he examined 13,950 articles published from 1991 to November 2012, and only found 24 that rejected anthropogenic global warming. That’s a 99.83% agreement among studies. He later followed that up by looking at the 2,258 climate change papers published from November 2012 to December 2013. This only revealed 1 paper that rejected anthropogenic climate change (a 99.96% consensus). Admittedly, neither of those were published in peer-reviewed journals (but you are welcome to replicate his results), but a subsequent analysis of papers in 2013 to 2014 was peer-reviewed. In it, he examined 24,210 papers by 69,406 authors, and found a grand total of 5 articles published by 4 scientists that rejected the notion of anthropogenic climate change (Powell 2015). That gives us a consensus of evidence (studies) of 99.98%, and a consensus of experts of 99.99%. To put that another way, for every 1 publishing climatologist who disagrees with anthropogenic climate change, there are 9,999 who agree with it. That’s a pretty extraordinary consensus of experts.

Powell also examined the same papers used in Cook et al. 2013, only 24 of which rejected anthropogenic climate change (99.78% agreement; Powell 2016; many of these were the same papers in his non-peer-reviewed analysis). Finally, part way through last year (2019), he examined all of the studies on climate change that had been published so far that year (11,602), and not a single one rejected anthropogenic global warming (Powell 2019).

Additionally, Lynas et al. (2021) took 88,125 papers published between 2012-2020, randomly subset the dataset to 2,718 papers (a representative sample; see note), and examined those abstracts for stated agreement or disagreement with anthropogenic climate change. Of these, only four rejected anthropogenic climate change. That’s a 99.85% consensus of evidence. Similarly, if we apply the Cook et al. criteria, we get a 99.53% consensus of evidence.

These surveys of the literature are extremely compelling evidence that a consensus has been reached and the topic is “settled.” If there was actually serious debate, if actual evidence existed discrediting anthropogenic global warming, we would see that in the literature. We would see numerous studies publishing evidence against anthropogenic climate change, but we don’t see those studies because that evidence doesn’t exist. All of the available data very clearly shows that we are causing climate change. The scientific consensus on this topic is truly overwhelming. Nevertheless, I am sure many people are preparing to fire off responses, so I want to spend the rest of this post preemptively dealing with them.

Note on Lynas et al. They started with a random selection of 3,000 papers, but some were not actually about climate change (database searches for papers often have false positives), so they were removed. Also, it’s worth mentioning that for their sample, they calculated the 99.999% confidence interval to be 99.212%–99.996%. In other words, were are 99.999% sure that the true consensus of evidence (for all papers) is >99% of climate change studies.

“But what about those petitions/letters where thousands of scientists said we aren’t causing climate change?”

There have been many attempts to discredit the consensus of experts by accruing lists of signatures, but if you examine those lists, their fraudulent behavior becomes apparent. Probably the most famous is the “Oregon Petition” which (depending on the source commenting on it) received signatures from 16,000, 30,000, 31,000 or 32,000 scientists. That sounds impressive, until you do even a modicum of fact checking, at which point you’ll realize that this petition is a fraud.

First, there was virtually no verification process. As a result, there were lots of fake signatures, including celebrities and fictional characters. Further, even for the signatures that were real, the only requirement was a B.Sc. in science, which hardly makes someone a scientist, and certainly doesn’t make them an expert on climate change. A huge portion of people who get undergraduate degrees in science never actually use their degrees. Further, even for those who went on to obtain additional degrees and pursue careers in science-related fields, many were experts in totally unrelated fields. For example, how does an orthopedic surgeon, veterinarian, or mechanical engineer qualify as an expert on climate change? When you cut through all the crap, you are left with only 39 people who actually have relevant degrees and expertise in climatology. That is hardly an impressive number and certainly doesn’t discredit the notion of a consensus of experts (more details here, here, and here).

A more recent attempt was a letter to the UN, supposedly signed by 500 scientists, arguing that there is no climate emergency. Once again, however, when you start looking at the signatures, most weren’t even scientists, let alone climatologists. Further, many of them had conflicts of interest, and the claims made in the letter aren’t supported by actual scientific evidence (more details here and here).

There is also another more fundamental issue with these attempts to discredit the science. Namely, they are only about the consensus of experts, not the consensus of evidence. Science is not a democracy, and even if there were hundreds of climatologists who rejected climate change, that would be irrelevant unless they actually had data to back up their position, which they don’t.

“But what about the list of 500 studies showing that climate change isn’t happening/is natural?”

Science deniers love lists of studies. I have, for example, written extensively about the lists anti-vaccers have assembled. The problem is that these lists are inevitably assembled without an actual understanding of the science, and when you look at the papers, they don’t say what the science-deniers think they say. For example, when I went through anti-vaccers’ lists of 160 studies that supposedly showed that vaccines cause autism, I found that 33 of their studies weren’t about autism, 82 weren’t about vaccines, multiple studies explicitly stated that vaccines don’t cause autism, and only 13 were actual human trials arguing that vaccines caused autism (all of which were riddled with problems). The exact same thing has been true of every list of papers I have ever been shown that supposedly discredits anthropogenic climate change. The lists are inevitably filled with papers that talk about regional trends (not global), talk about past climates (without addressing the current warming), talk about how natural climate forcing work (without discussing the current warming), etc. As with the anti-vaccine lists, nearly all of them are misrepresented, most are irrelevant, and many actually argue the opposite of what science-deniers are claiming.

“But what about the few studies that do actually argue against anthropogenic climate change?”

At this point, you might be thinking, “fine, most climatologists agree, and very few studies disagree, but there are a few studies that disagree, and in science, any position can be overthrown by new evidence, so what about those studies?” This is a fair question given two caveats: first, we always have to examine all studies in the context of the broader literature. Given that the context in this case is literally thousands of studies with numerous lines of evidence showing that we are the cause, the evidence in the dissenting studies had better be pretty good.

This brings me to the second point. We always have to critically examine studies rather than assuming that they are valid, and when we do that, we find that these studies used weak designs, shoddy statistics, and are full of problems (Benestad et al. 2016). So they do not in any way discredit the overwhelming mountain of evidence.

“But scientists are just following the ‘dogma’ of their field”

This well-worn trope argues that lots of scientists actually have evidence against anthropogenic climate change, they just don’t publish it because in science it is supposedly forbidden to go against the “dogma” of your field. This is one of those fundamental misunderstandings of science that just will not die. Science is extremely adversarial. We love to prove each other wrong. Further, every scientist who was ever considered great was great precisely because they discredited the views of their day. No one gets anywhere in science by blindly going with the “dogma” of their fields. If anyone actually had compelling evidence that we weren’t causing climate change, they would publish in a high-ranking journal and collect their Nobel Prize. No one has done that precisely because those data don’t exist.

“But scientists have been wrong before”

This is another trope that I have dealt with many times before, so I’ll be brief. First, there are few (if any) examples where modern science has been wrong about something with the same level of evidence that we have for climate change. The “evidence” that was used for things like the sun orbiting the earth is not even remotely comparable to the evidence for climate change.

Second, past mistakes do not automatically negate the evidence for climate change. If it did, then you could use it any time that you wanted to discredit any scientific study. “You think that smoking causes cancer? Well science has been wrong before, so I don’t have to accept that.” See how stupid that is? You need actual evidence to discredit climate science.

Third, this argument is inherently self-contradictory, because it is only through science that we know that previous scientists have been wrong, but based on this argument, we can’t trust science. Therefore, we have no more reason to trust the evidence that the earth moves around the sun than we do for the discredited evidence that the sun moves around the earth. In other words, if the fact that scientists have been wrong before means that we can’t trust scientific discoveries, then we can’t trust the scientific discoveries that were used to show that scientists had been wrong before. It’s a paradox.

See these posts for more details

 Also read this post before arguing that “most scientific studies are actually wrong”

“But there used to be a consensus that smoking was safe”

This is just a special case of the “science has been wrong before” argument. Further, it’s not even true. Tobacco companies certainly ran a good misinformation campaign (much as fossil fuel companies do today), but actual scientific studies have consistently shown that smoking is dangerous. Indeed, scientists suggested that smoking was dangerous way back in the early 1900’s, and essentially all of the research since then (minus a few industry-driven papers) confirmed their suspicions (you can find an overview of this history in Proctor 2012).

“But in the 70s there was a scientific consensus on global cooling”

No, there wasn’t. There was certainly media hype about this, but it was never a prominent scientific position. Indeed, there were a grand total of 7 papers on it, compared to 42 during the same time span that argued that we were causing global warming (Peterson et al. 2008).

“But scientists don’t agree about the extent to which we are causing climate change”

This is a very common tactic among science deniers: taking a minor disagreement and conflating it with a major one. There is some disagreement among analyses about exactly how much we are contributing to climate change, but they all agree that the majority of the change is being caused by us. There is no serious disagreement that we are the primary cause. If there was, this would, once again, be easy to find in the literature, but good luck finding many studies that argue that we are only playing a minor role. They are virtually non-existent.

“But….”

There are tons of other invalid counterarguments that I’m sure I’ll get assaulted with, but I have already addressed most of them in previous posts so please read them before making inane comments. Also, if you want to more information about why simply looking for papers that reject climate change is a good approach for testing a consensus, read Powell’s papers (cited at the end) as they explain things in much greater detail.

  • This post covers most common arguments and counter points.
  • This one explains the evidence that makes us so certain that the current warming is not natural and is being caused by us
  • This one goes over the evidence that climate change is already having serious consequences
  • This one debunks the absurd notion that scientists are just in it for money
  • This one talks about Cook et al. 2013 in more detail and discusses other attempts to estimate a consensus

Conclusion

In short, there is an overwhelming consensus that we are causing climate change. This consensus exists both among studies and among scientists. Indeed, recent estimates put it at over 99.9% agreement that we are causing the climate to change. Thousands of studies have confirmed that we are the cause, and virtually none argue that we aren’t. Further, the handful of contrarian studies are riddled with problems and are easily debunked. Every shred of evidence confirms that we are causing climate change, and there is no serious debate among experts.

This level of consensus is important, because it means that there is no valid reason for doubting the reality that we are causing climate change. The level of consistent evidence for it is on par with the evidence for things like smoking causing cancer. Both topics have been extremely thoroughly studied, both topics have a huge and remarkably consistent body of evidence (i.e., a consensus of evidence), and for both topics, that body of evidence has resulted in a nearly unanimous consensus among relevant experts. Nevertheless, on both topics, it is possible to cherry-pick studies and experts that disagree with the consensus, but doing so is folly! As I explained, it is always best to look at the evidence itself, but for most people, that’s not possible, in which case it is rational to simply listen to experts, but why would you choose to listen to the 0.01% of experts who disagree with the consensus? You wouldn’t do that on a topic like smoking causing cancer, so why would you do that with climate change? If 9,999 doctors diagnosed you with cancer and told you to immediately start treatment, but 1 doctor told you that you had nothing to worry about, would you blindly follow that one doctor? I highly doubt it, so why would you do that with climate change? Why would you listen to the 1 scientist saying we aren’t causing it rather than the 9,999 who are saying that we are causing it and need to change our actions?

Related posts 

 Literature Cited

  • Benestad et al. 2016. Learning from mistakes in climate research. Theoretical and Applied Climatology 126:699–703.
  • Cook et al. 2013. Quantifying the consensus on anthropogenic global warming in the scientific literature. Environmental Research Letters 8:024024
  • Lynas et al. 2021. Greater than 99% consensus on human caused climate change in the peer-reviewed scientific literature. Environmental Research Letters 16:114005
  • Oreskes. 2004. The scientific consensus on climate change. Science 306: 1686.
  • Peterson et al. 2008. The myth of the 1970s global cooling scientific consensus. Bulletin of the American Meteorological Society 89:1325–1337.
  • Powell 2015. Climate scientists virtually unanimous: Anthropogenic global warming is true. Bulletin of Science, Technology & Society 35:121-124.
  • Powell 2016. The consensus on anthropogenic global warming matters. Bulletin of Science, Technology & Society 36:157-163.
  • Powell 2019. Scientists reach 100% consensus on anthropogenic global warming. Bulletin of Science, Technology & Society
  • Proctor 2012. The history of the discovery of the cigarette-lung cancer link: evidentiary traditions, corporate denial, global toll. Tobacco Control 21: 87-91
Posted in Global Warming, Nature of Science | Tagged , , , , | 75 Comments

More anti-vaccine cherry-picking: A rebuttal to, “Should you be afraid that measles can give you immune amnesia?”

Posted on November 7, 2019 by Fallacy Man

Earlier this week, I wrote a post about measles-induced “immune amnesia” and the growing body of evidence supporting it. Afterwards, I was directed to an anti-vaccine “rebuttal” to this evidence (not to my post specifically) which has been making its rounds in anti-vaccine circles and is being presented as a checkmate against the science. The article is titled “Should you be afraid that measles can give you immune amnesia?” by Tetyana Obukhanych, and it has the trappings of being based on evidence, but if you do even a little fact checking, you will quickly realize that its sources are cherry-picked and its arguments blindly disregard large chunks of immunology. Given how popular this article seems to be, I am going to go through it piece by piece and demonstrate that the evidence was cherry-picked and distorted to fit the authors’ preconceptions.

Note: After writing this, I discovered that Orac beat me to the punch by 5 hours. Our overall assessments of the article are quite similar, but he brings up several good points that I did not, so I suggest reading his rebuttal as well.

Before I begin, I want to give an extremely brief overview of the topic being discussed. Scientists have known for several years that an infection with the measles virus causes immune amnesia, in which patients have increased rates of infections and even death from other diseases for several years following the initial measles infection. Recently, studies have shown that this occurs because measles attacks memory lymphocytes (B and T cells) and depletes your body’s antibody repertoire. These memory cells and antibodies are the things that provide lasting immunity. They are specific for particular diseases and persist after an initial infection, thus protecting you into the future. The measles virus destroys these cells, thus leaving you vulnerable to diseases that you were protected against (more details on the immune system here and immune amnesia here). Tetyana Obukhanych disagrees with all of this, but as you’ll see, her evidence is shoddy to say the least.

Note: I will refer to Tetyana Obukhanych by her first name throughout, rather than her last (as one would typically do in academia) simply to avoid creating the illusion that she is on par with the actual scientists whose work I will cite. She’s not. She abandoned rational thought years ago to pursue a career in pseudoscience, so I will not refer to her as if she is a legitimate scientist (see further note at the end of the article).

Moving on to her actual post, she does at least cite studies to support her argument, but, as I frequently write about on this blog, you can cherry-pick a study to support just about any position. Therefore, you have to read the studies critically and look at the entire body of evidence. She did not do this. She simply grabbed a few studies that she liked and ignored multiple much larger studies that discredited her views. Let’s start with the four epidemiological studies she cited as evidence that immune amnesia is not real.

Her first study is Aaby et al. (2002). There are two important things to note about this study. First, it looked at the effects of mild measles infections on long-term survival, but the studies of immune amnesia found that more severe cases resulted in more severe immune amnesia. So, a negative result from a study on mild measles cases is hardly good evidence that immune amnesia is wrong. Second, this study was small for a mortality study (215 children) and the results were just barely significant. In other words, there is a high probability that this is a false positive, and it is far from a compelling result.

Another of her studies is Aaby et al. (1996a). This study looked at both mortality and T cell counts for patients following measles infections compared to controls. For the mortality aspect, the sample size (140 children) was, once again, far too small to have any confidence whatsoever in the results. It is hardly surprising that such a small study failed to find a difference in mortalities. That sample size is, however, reasonable for a study of cell counts; however, the type of counts that were done where quite crude (total lymphocyte count, CD4 percentage, CD8 percentage, etc.). The studies on immune amnesia that Tetyana is trying to refute where were looking at the diversity of memory lymphocytes, but that wasn’t addressed by Aaby et al. (1996a). In other words, this study does not provide any evidence that the studies on immune amnesia are wrong.

Her other two studies (Aaby et al. 1996b; 2003) had more reasonable sample sizes, but they suffered from an inherent problem that was present in all four of these studies. Namely, they were conducted in impoverished areas with little access to healthcare where mortality rates are very high. This is a problem first because it means there are lots of confounding factors at play from other diseases, and second, because it biases the comparison between measles survivors and uninfected (often vaccinated) children. In an area with access to modern medicine, most children survive measles because of the medicine. Thus, the inherent strength of the child’s immune system is fairly irrelevant (unless they are immunocompromised). In areas without modern medicine, however, we inherently expect that children who naturally have stronger immune systems will be more likely to survive the infection. Thus, our population of survivors is biased towards children who naturally have strong immune systems. An absolutely fundamental component of scientific comparisons is that the groups should be alike in every way except for the variable of interest, but that is inherently untrue in these studies. They weren’t simply comparing uninfected children with children who survived measles, rather they were comparing uninfected children with children who were strong enough to survive measles. See the difference? That inherently makes these studies problematic.

The other problem with her use of these papers is simply that they are cherry-picked. She failed to mention, for example, that Aaby also did a study in 1990 (Aaby et al. 1990) that showed substantially higher mortality in the years following measles infection. To be fair, it wasn’t a large study (276 children) and it also had the rural area issues I mentioned, but she should have mentioned it if she was actually giving a fair representation of the literature.

She also utterly failed to mention the massive 2015 study (Mina et al. 2015) that used decades of data from England, Wales, the United States, and Denmark and clearly showed that measles infections increase mortality rates for 2–3 years following infection! This study did a very good job of clearly laying out falsifiable predictions (as science should do), found consistent patterns (as expected for a real result), avoided the survivorship bias inherent in all the Aaby et al. studies, and had large data sets. It is by far the most compelling of any of the studies discussed so far. This is a huge omission on Tetyana’s part and demonstrates obvious cherry-picking. You can’t just pretend that a study like that doesn’t exist.

Further, she declined to mention a massive cohort study in the UK that compared disease rates for 2,228 children following measles infection with 19,930 children who were not infected with measles (Gadroen et al. 2018). They found significant increases in illnesses at every time point for five years after the measles infection. This is another huge study from a developed country, and this form of cohort analysis is generally more robust than the type of population-based analysis used by Mina et al. (2015), but once again, Tetyana failed to mention it. She mentioned the relatively tiny studies from rural areas, but not the massive, robust studies from countries with good health care. It’s almost like she’s deceptively cherry-picking which evidence to show you…

Beyond this epidemiological evidence, we now have several studies showing that measles can infect and kill memory lymphocytes (i.e., the cells responsible for long-term immunity), thus depriving you of your immunity (de Vries et al. 2012; Petrva et al. 2019). In other words, we now have a clear mechanism for immune amnesia that is consistent with and builds on previous research on the ability of certain viruses to target memory cells (Selin 1996; Kim and Welsh. 2004).

She responds to this extremely clear evidence by saying, “So what?  When was it ever proven that immunologic memory has anything to do with protection from re-infection?” Had I been drinking something when I read that, I would have spit it all over my computer. The notion that immunological memory is a key component of protecting people from infections is extremely well-established. It is a fundamental and extremely basic concept of immunology that is literally in every single immunology text book.

Her “evidence” that immunological memory does not protect people is the work of Rolf Zinkernagel which she claims “proves” that immunological memory does not confer protection from disease (she also commits a minor appeal to authority fallacy by unnecessarily pointing out that he is a Nobel prize winner as if that automatically makes him right about everything), and she cites his 2012 review (Zinkernagel 2012). There are several things to unpack here, the first of which is that Zinkernagel is pretty close to alone in his views. Indeed, in his papers, he readily acknowledges that the vast majority of immunologists disagree with him and his interpretation of the data. Also, he has been arguing for his view for a long time (since at least the late 1990s) and has not been able to convince many immunologists that he is right (because he lacks sufficient data), and that review from 2012 has a grand total of 21 citations (not much for a review that claims to overturn a massive component of immunology), and even the papers citing him argue that immune memory is important for being protected from diseases, they just point out that there are caveats and some special situations for certain pathogens (see Hohman and Peters [2019], for example). So Tetyana essentially wants us to accept that all of immunology is wrong because this one man says so. That’s going to be a hard pass from me. That’s simply not how science works.

To be clear, the fact that Zinkernagel’s views have not been widely adopted does not automatically make him wrong (but it is suggestive). Rather, the issue is simply that the evidence does not appear to be on his side. That 2012 review, for example, was not a systematic review (i.e., one that considers all literature on a topic). Rather, it was a critical review, which means that he gets to pick and choose which papers to include to build the argument that he wants to. In contrast, a systematic review of the literature (specifically looking at memory T cells and their effect on disease) looked at 147 studies covering 25 human disease and found that immunological memory is indeed very important in providing protection from disease (Muruganandah et al. 2018). There’s also lots of other reviews (not all systematic) either on immunological memory or that talk about it and explain that it is real and important (e.g., Macallan et al. 2017; Pennock et al. 2013; Pulendran and Ahmed 2006), and, as I said, this is covered in literally every text book on immunology.

Having said all of that, I actually think that Tetyana is somewhat misinterpreting Zinkernagel. To be fair, I did as well the first time that I read the 2012 review, and Zinkernagel certainly hasn’t done himself any favors in how he has written his arguments. Following my confusion with his 2012 review, I read his 2018 critical review, which is essentially the same paper and still (in my opinion) fails to provide compelling evidence for his position, but it made a bit more sense. Then I started reading his older papers and finally think I figured out what he is actually arguing. Consider this quote from a paper (Ochsenbein et al. 2000) that he was an author on,

“Therefore, for vaccines to induce long-term protective antibody titers, they need to repeatedly provide, or continuously maintain, antigen in minimal quantities over a prolonged time period in secondary lymphoid organs or the bone marrow for sufficient numbers of long-lived memory B cells to mature to short-lived plasma cells.”

His argument, as I understand it, is that memory cells alone are not sufficient for protection. Rather, they need to constantly be replicating and maturing into active cells that produce antibodies, and it is the population of active cells (and their antibodies) that actually provides the protection, but for this population to be maintained, a low level of antigens (the surface recognition molecules that identify a given pathogen) must be present to stimulate the memory cells. I’m still not convinced that he’s right, but this makes a lot more sense than simply saying that immunological memory is unimportant for protection. Indeed, other complex interactions between the innate immune system and the adaptive immune system have certainly been documented (e.g., Castellino et al. 2009). If I am interpreting him correctly, then the argument is not that memory lymphocytes are not important, but rather that they alone are not sufficient, and after the initial infection, there is a perpetual cycle of antigens stimulating the memory cells, resulting in the production of antibodies, effector cells, etc. This means that, contrary to what Tetyana is arguing, those memory lymphocytes are critically important. If they get taken out by a disease like measles, then they cycle will be broken because they will no longer be there to mature into the active cells that Zinkernagel argues are responsible for protection, meaning that you will, once again, be back to being unprotected. Thus, even if Zinkernagel is right, Tetyana’s argument is wrong (she later cites another of his papers [Steinhoff et al. 1995], but just refer back to this section for that; sufficient to say she is extrapolating far beyond what we can actually conclude from the paper).

On a quick side note, I am curious about why Tetyana didn’t cite Zinkernagel’s more up-to-date 2018 review. I have to wonder if it is because he spoke disparagingly of “vaccine deniers” in the abstract. You see, Zinkernagel is no anti-vaccer, and here we can see another example of the inconsistencies of the anti-vaccine mindset. According to Tetyana, we should blindly believe him instead of virtually all other immunologists when it comes to immune memory (after all, he won a Nobel prize), but when it comes to vaccines, we should ignore him and his Nobel prize. This is classic cherry-picking of experts and even the views of an expert.

Next, we get to her response to the Mina et al. (2019) which found that measles infections reduce the diversity of circulating antibodies for diseases that you were previously protected against. As before, she responds to this by denying immunology 101. She asks, “When was it ever proven that antibodies offer protection?” Questions like this are baffling to me. Yes, antibodies can offer protection! This is extremely well established. To give probably the most well-known example, newborns get antibodies from their mother, and those antibodies protect them until their own immune system builds up memory for the common pathogens around it (Niewiesk 2014; van der Lubbe et al. 2017). This is really basic stuff. Sure, there are complexities and interactions among different parts of the immune system, and different diseases require different immune responses, but pretending that antibodies aren’t important for protection is either insane or dishonest. Further, even some of the papers she cites says this. For example, she totally ignores the numerous times that Zinkernagel discusses the importance of antibodies in offering protecting. Let me quote the abstract of his 1995 paper (Steinhoff et al. 1995) that she somehow thinks proves her point (my emphasis)

“Adoptive transfer experiments showed that neutralizing antibodies against the glycoprotein of VSV (VSV-G) protected these mice efficiently against systemic infection and against peripheral subcutaneous infection.”

Similarly, remember that 2012 review that she is so found of? Here is a quote from its abstract (again, my emphasis)

“Protection depends on pre-existing neutralizing antibodies or pre-activated T cells at the time of infection.”

Sure seems like her man Zinkernagel thinks that antibodies provide protection. To be clear here, I’m not saying that antibodies provide protection because Zinkernagel says they do, rather, I am trying to demonstrate the absurd levels of cherry-picking she is going through. Her own sources defeat her arguments.

Further, her “evidence” that all of immunology is wrong about antibodies is simply a report of four healthcare workers who became infected with measles despite having measles antibodies (Ammari et al. 1993). Now it’s my turn to ask, “so what?” No one has ever said that having detectable circulating antibodies is a 100% guarantee that you won’t get the disease, especially for someone like a healthcare worker who will be exposed to it regularly. Further, as eluded to earlier, different diseases are most effectively targeted by different parts of the immune system. So even if antibodies provided 0 protection against measles (which is not the case) that would not change the fact that they are very effective for many other diseases, meaning that it is a big problem when an infection with measles reduces their diversity.

Additionally, with regards to her arguments about both immunological memory and antibodies, keep in mind that we do have large epidemiological studies showing that measles-induced immune amnesia is a real thing that increases infections and death for several years following measles infections. This is a fundamental point that she totally ignores. We aren’t talking about hypothetical mechanisms here. Rather, the recent studies were looking for a mechanism to explain a phenomenon that was already established!

She rambles on for a while about a few other points that are fairly irrelevant, so I’ll just quickly comment on two of them. First, she suggests that the chicken pox virus likely can target memory cells as well and asks why people aren’t freaking out about it, as if that somehow proves her point. First, people aren’t freaking out about it because scientists are a cautious bunch and don’t like jumping to the conclusion that the effects of one virus will be the same as the effects of another. Right now, we only have good data for measles. Second, it is likely that chicken pox can do something similar, but that is simply a good argument for getting the chicken pox vaccine! It doesn’t discredit the science or alleviate the concern.

The other thing I want to comment on is one of her last paragraphs where she seems to suggest that measles infection is good because, if it kills memory cells, it should alleviate allergies and asthma. I’m not convinced that infection would alleviate allergies, but let’s assume for a minute that it does. We have epidemiological data clearly showing that mortalities increase for 2–3 years following measles infections, and she thinks this is overridden by a possible reduction in allergies? Further, let’s not forget that measles is itself deadly. Those Aaby et al. studies she cited showed that. Are we just supposed to pretend that it isn’t deadly? As an adult with allergies, I prefer a life of daily antihistamines (allergy meds) to a childhood death from a disease, thank you very much.

Conclusion

In short, Tetyana’s post is a whole lot of nonsense. It cherry-picked its evidence, relied on a fundamental lack of understanding of basic immunology, and ignored clear epidemiological evidence that measles-induced immune amnesia is a real thing with deadly consequences. The actual evidence overwhelmingly supports immune amnesia and shows that the beneficial effects of the measles vaccine go far beyond simply preventing measles.

A note about Tetyana Obukhanych and appeals to authority

 I put this after the main article, because it is tangential, but I do want to make a quick comment about the way anti-vaccers respond to Tetyana. Much to my surprise, she actually does have a PhD in immunology. This does not, however, automatically mean that she knows what she is talking about. Having a PhD (or any advanced degree) does not guarantee that someone is smart or even particularly knowledgeable. Further, as far as I can tell, she only ever published 8 papers, and hasn’t published any research since 2012, when she left academia to pursue a career writing anti-vaccine books and posts, giving talks, and offering online pseudoscience courses. None of this automatically makes her wrong, of course. My point is simply that you shouldn’t be lulled into a false confidence in her views just because her name has the letters “PhD” after it. Indeed, after leaving academia, she has gained a reputation for writing highly counter-factual posts that are devoid of reasoning (here are examples of other skeptics debunking some of her previous writing: Skeptical Raptor, Science-Based Medicine, and Snopes). Again, this doesn’t automatically make her wrong about immune amnesia. Rather, I am simply pointing out that she is not particularly reputable, and she certainly isn’t the world-renowned immunologists that many anti-vaccers seem to think she is (also, see this post by Vaxopedia).

Related posts

Literature cited

(some of these are behind paywalls, see this post for suggestions on how to access them)

  • Aaby et al. 1990. Delayed excess mortality after exposure to measles during the first six months of life. AM j Epidemiol 132:211.
  • Aaby et al. 1996 No persistent T lymphocyte immunosuppression or increased mortality after measles infection: a community study from Guinea-Bissau. Pediatr Infect Dis J 12:39–44.
  • Aaby et al. 1996b. No long-term excess mortality after measles infection: a community study from Senegal. Am J Epidemiol 143:1035–1041.
  • Aaby et al. 2003. The survival benefit of measles immunization may not be explained entirely by the prevention of measles disease: a community study from rural Bangladesh. Int J Epidemiol 32:106–116
  • Aaby et al. 2002. Low mortality after mild measles infection compared to uninfected children in rural West Africa. Vaccine 22:120–126.
  • Castellino et al. 2009. Generating memory with vaccination. European Journal of Immunology 39: 2100–2105
  • Gadroen et al. 2018. Impact and longevity of measles-associated immune suppression: a matched cohort study using data from the THIN general practice database in the UK. BMJ 8
  • Hohman and Peters. 2019. CD4+TCell-Mediated immunity against the phagosomal pathogen Leishmania: Implications for vaccination. Trends Parasitology
  • Kim and Welsh. 2004. Comprehensive early and lasting loss of memory CD8 T cells and functional memory during acute and persistent viral infections. J. Immunol. 172: 3139–3150
  • van der Lubbe et al. 2017. Maternal antibodies protect offspring from severe influenza infection and do not lead to detectable interference with subsequent offspring immunization. Virology Journal 12:1695
  • Macallan et al. 2017. Human T cell memory: A dynamic view. Vaccines 2017: 5.
  • Mina et al. 2015. Long-lasting measles-induced immunomodulation increases overall childhood infectious disease mortality. Science 348: 694–699.
  • Mina et al. 2019. Measles virus infection diminishes preexisting antibodies that offer protection from other pathogens. Science 366:599–606
  • Muruganandah et al. 2018. A systematic review: The role of resident memory T cells in infectious diseases and their relevance for vaccine development. Frontiers in Immunology 9:1574
  • Niewiesk 2014. Maternal antibodies: Clinical significance, mechanism of interference with immune responses, and possible vaccination strategies. Frontiers in Immunology 5:446.
  • Ochsenbein et al. 2000. Protective long-term antibody memory by antigen-driven and T help-dependent differentiation of long-lived memory B cells to short-lived plasma cells independent of secondary lymphoid organs. PNAS 97: 13263-13268.
  • Pennock et al. 2013. T cell responses: naïve to memory and everything in between. Adv Physiol Educ 37: 273­–283.
  • Petrva et al. 2019. Incomplete genetic reconstitution of B cell pools contributes to prolonged immunosuppression after measles. Science Immunology 4: eaay6125
  • Pulendran and Ahmed 2006. Translating innate immunity into immunological memory: Implications for vaccine development. Cell 124: 849–863
  • Selin 1996. Reduction of otherwise remarkably stable virus-specific cytotoxic T lymphocyte memory by heterologous viral infections. J. Exp. Med. 183: 2489–2499
  • Steinhoff et al. 1995. Antiviral protection by vesicular stomatitis virus-specific antibodies in alpha/beta interferon receptor-deficient mice. J Virol 69:2153–2158.
  • de Vries et al. 2012. Measles immune suppression: Lessons from the macaque model. PLOS Pathog. 8: e1002885
  • Zinkernagel 2012. Immunological memory ≠ protective immunity. Cellular and Molecular Life Sciences 69:1635–1640
  • Zinkernagel 2018. What if protective immunity is antigen—driven and not due to so-called memory” B and T cells? Immunological Reviews 2018: 283–246.
Posted in Vaccines/Alternative Medicine | Tagged , , , , | 13 Comments

Measles infections weaken your immune system and increase your risk of other diseases. Vaccines prevent this

Posted on November 5, 2019 by Fallacy Man

Two of the most persistent anti-vaccine tropes are that unvaccinated children are healthier than vaccinated children and that “natural” immunity is better than “artificial” immunity. There has never been any evidence to support these claims, and plenty of evidence that they are wrong (Schmitz et al. 2011; Grabenhenrich et al. 2014), but two recent studies have shed new light on just how wrong they are. These studies built on previous work and showed that infection with the measles virus actually destroys memory cells, resulting in “immune amnesia” for years to come. In other words, becoming infected with measles makes you far more likely to be infected with other diseases for several years after the original measles infection. It actually weakens your immune system, rather than building it.

Before I can talk about these studies and their implications, I need to briefly explain how the immune system and vaccines work. I have done so in more detail here, so I’ll be brief. When a pathogen first enters your body, it is attacked by the innate immune system, which provides a non-specific response. In other words, it does not have specific cells for fighting specific pathogens. While this is happening, however, your adaptive (aka acquired) immune system goes into action. This immune system is more specific and generates T and B cells (specialized immune cells) that are specific for targeting a particular pathogen. This is a very powerful arm of your body’s immune defenses and is vital for fighting things like measles infections. It takes time, however, for your body to learn to recognize a new pathogen and build appropriate cells and subsequent antibodies to respond to it. While this is happening, the pathogen multiplies, and you become sick.

After the infection (assuming you survived). Your body maintains memory T and B cells for that pathogen so that it can respond quickly in the future. Your body also retains antibodies from the initial infection which can respond immediately to future infections by that pathogen. This is how natural immunity works.

Vaccines activate the same system, but do so by presenting your body with the antigens (proteins your body uses to recognize pathogens) for the pathogen in question (or a dead or weakened version of the pathogen) rather than giving you the live, healthy pathogen. This causes your body to go through the same cycle of producing B and T cells and releasing antibodies that it would go through for a real infection, but there is one critical difference: you don’t get the disease. This is the fundamental reason why it is absurd to argue that natural immunity is better than artificial immunity. To get natural immunity, you first have to get the disease! It is literally arguing that it is better to get the disease so that you don’t get it again rather than just never getting it in the first place!

But what about the claim that getting a disease helps build the immune system? As you can hopefully now see, it only “builds” the immune system in that it teaches the body how to respond to one particular pathogen, which is exactly what vaccines do without ever making you sick. There is, however, another catch here, which is where the new studies come in. As it turns out, while measles infections are “building” the immune system by teaching it how to respond to the measles virus, they are also destroying memory cells and greatly weakening the immune system. You see, immunity can be lost. This is true for both natural immunity from infections and artificial immunity from vaccines (though the latter can be easily remedied with boosters). Over time, memory B cells and T cells die, and the number of antibodies circulating in your body for a particular pathogen diminish. This can eventually lead to a loss of immunity. This also means that, in concept, a pathogen could destroy existing immune cells and make you vulnerable to diseases that you were previously protected against. We now know that this is exactly what the measles virus does.

We’ve known for a long time that measles virus infections have a suppressive effect on the immune system, andt5hat suppression is partially why secondary infections are so common for measles patients. This knowledge goes back at least as far as 1908 (Pirquet 1908) and has been corroborated by more recent research (Griffin 2010; de Vries et al. 2012), but what we didn’t realize was just how severe this suppression was or how long it lasted for. Several studies on mice found that viral infections could actually take out previously existing memory cells and, presumably, put the mice at risk for future infections (Selin 1996; Kim and Welsh. 2004), but, as regular readers of this blog know, animal studies are useful starting points, but they only go so far, and we really need studies on humans to get a clear picture of the situation.

Compelling epidemiological evidence of measles having a lasting impact on human immune systems arrived in 2015, when researchers found that measles infections increased mortalities from other infections for 2–3 years after the measles infection (Mina et al. 2015)! This result was corroborated by a large cohort study (one of the most powerful study designs) that found increased infection rates for diseases (other than measles) for five years following infection with measles (Gadroen et al. 2018). These studies provided really good evidence that measles did something harmful to the immune system, but we still weren’t quite sure what it was doing.

This brings us finally to the two recent studies: Petrva et al. (2019) and Mina et al. (2019). Both of these studies took blood samples from children before and after natural infection with the measles virus, and Petrva et al. looked at the effect on B cells, while Mina et al. looked at the effect on circulating antibodies. They both found the same thing: the measles virus reduced the diversity of the immune system (B cells or antibodies) thus putting patients at risk for other diseases. In other words, the virus destroys the components of your immune system that had previously learned to respond to other diseases. Thus, the natural immunity you had to those diseases is gone (or at least greatly diminished) and you are susceptible to them again. Further, this impact was not small. Mina et al found that severe measles infections caused children to lose 11–62% (median = 40%) of their existing antibody repertoire! That’s a huge loss.

Now, you may be wondering what affect the vaccine has. Does it suppress the immune system like an actual infection does? Mina et al. (2019) looked at this as well, and no, it doesn’t. All that it does is make children immune to measles. That’s it. This makes good sense if you understand what is going on here. The measles virus actually infects cells (including memory B and T cells), which is why it can do so much damage to you and your immune system. Vaccines don’t do that. They can’t infect anything because the either don’t contain the pathogen at all, or contain a dead or weakened version of it that can’t infect you. Thus, all that they do is teach your immune system how to respond to a pathogen without any of the damaging effects of actually becoming infected with the pathogen.

Before concluding this post, I also want to point out that another common anti-vaccine myth is that natural immunity is lifelong, whereas artificial immunity is temporary. The papers discussed in the post very clearly show that natural immunity can also be temporary and acquiring natural immunity for one disease can actually cost you previous immunity to others. Further, vaccine-induced immunity often lasts just as long as natural immunity (Jokinen et al. 2007), and even in cases where natural immunity does last longer, it is often not life-long (Wendelboe et al. 2005), and, again, the vaccine prevents you from ever getting the disease in the first place (and their protection can be extended with boosters).

Conclusion

In short, the notion that unvaccinated children are healthier than vaccinated children is not only wrong, it is backwards. Recent research shows that diseases like measles actually do a lot of damage to your immune system and rob you of immunity you had previously acquired to other diseases. This puts you at risk for a wide range of diseases beyond the one that the vaccine protects against. In other words, not only does “natural” immunity to diseases like measles require you to actually get the disease before you can be protected against it, but it also weakens your immune system and makes you susceptible to many other diseases. Vaccines save lives and keep you healthy. It’s that simple.

Rebuttal to a popular anti-vaccine article on this topic

Related posts

Literature cited

  •  Grabenhenrich et al. 2014. Early-life determinants of asthma from birth to age 20 years: a German birth cohort study. Journal of Allergy and Clinical Immunology 133:979–988.
  • Gadroen et al. 2018. Impact and longevity of measles-associated immune suppression: a matched cohort study using data from the THIN general practice database in the UK. BMJ 8
  •  Griffin, DE. 2010. Measles virus-induced suppression of immune responses. Immunol. Rev. 236: 176–189
  • Jokinen et al. 2007. Cellular Immunity to Mumps Virus in Young Adults 21 Years after Measles-Mumps-Rubella Vaccination. Journal of Infectious Diseases 196: 861–867.
  • Kim and Welsh. 2004. Comprehensive early and lasting loss of memory CD8 T cells and functional memory during acute and persistent viral infections. J. Immunol. 172: 3139–3150
  • Mina et al. 2015. Long-lasting measles-induced immunomodulation increases overall childhood infectious disease mortality. Science 348: 694–699.
  • Mina et al. 2019. Measles virus infection diminishes preexisting antibodies that offer protection from other pathogens. Science 366:599–606
  • Petrva et al. 2019. Incomplete genetic reconstitution of B cell pools contributes to prolonged immunosuppression after measles. Science Immunology 4: eaay6125
  • Pirquet, C. 1908. Das Verhalten der kutanen Tuberkulinreaktion während der Masern. Dtsch. Med. Wochenschr. 34:1297–1300.
  • Schmitz et al. 2011. Vaccination status and health in children and adolescents. Medicine 108:99–104.
  • Selin 1996. Reduction of otherwise remarkably stable virus-specific cytotoxic T lymphocyte memory by heterologous viral infections. J. Exp. Med. 183: 2489–2499
  • de Vries et al. 2012. Measles immune suppression: Lessons from the macaque model. PLOS Pathog. 8: e1002885.
  • Wendelboe et al. 2005. Duration of Immunity Against Pertussis After Natural Infection or Vaccination. Pediatric Infectious Disease Journal 24: S58–S61.
Posted in Vaccines/Alternative Medicine | Tagged , , , | 2 Comments

How to find and access peer-reviewed studies (for free)

Posted on September 17, 2019 by Fallacy Man

The peer-reviewed literature is where scientists publish their research, and it is the source for scientific information. As a result, I spend a lot of time on this blog talking about it. I have explained how the peer-review system works (also here). I have provided advice on how to evaluate studies and how not to evaluate studies. I have explained the hierarchy of evidence. I’ve explained P values and false positives. I’ve explained why many studies are unreliable and why it is important not to cherry-pick studies. I have provided worked examples of how to dissect studies (e.g., here, here and here), and I do my best to cite studies to back up all the claims I make on this blog. Nevertheless, it was recently pointed out to me that I have utterly failed to explain something important and fundamental: how and where to find peer-reviewed studies. So I am going to remedy that by providing a brief primer on how to go about finding articles on topics you are interested in, and how to get free copies of them.

Where to look

Let’s start with where to look. You can try simply doing a standard Google search, but odds are that you will get flooded with tons of blogs and websites, and it is a pretty inefficient way to find what you are after. A much better option is to use a database specifically tailored to peer-reviewed literature. There are two major ones that are freely available that I’m going to talk about: Google Scholar and PubMed (there are many others that are behind paywalls, but I am going to assume that most people reading this are not academics and don’t have access to those).

Let’s start with Google Scholar. First, I need to make it absolutely clear that this is not the same thing as a regular Google search. Literally anyone can get a blog, write an article, and it will show up in a Google search. In contrast, Google Scholar is tailored for academic articles, and you cannot manually add articles to it*. Instead, Scholar pulls from several academic databases (e.g., JSTORE) and employs bots to scour the internet for DOIs, abstracts, titles, etc., which it uses to identify peer-reviewed articles and add them to its repository. It’s not a perfect system; some articles get missed by the bots, and occasionally they pick up a non-peer-reviewed article that has the trappings of a peer-reviewed article (e.g., a non-reviewed report). Nevertheless, it is an extremely useful tool. It is a massive database that is very easy to use (more on that later) and even though I have access to more well-curated databases, Scholar is usually what I default to for quick searches.

Scholar also has the advantage of being a generalist database. In other words, it is not topic specific, and articles on medicine, zoology, climate change, GMOs, evolution, physics, chemistry, archeology, etc. can all be found within its digital walls. Sometimes though, it is useful to use a more focused database, and that is where PubMed comes in.

As its name suggests, PubMed is a repository for medical papers. It gets its papers both directly from journals and from author submissions. These submissions are checked to ensure that they are scientific papers. As a result, it tends to be more curated than Scholar, and you don’t get as many results that aren’t actually peer-reviewed papers.

There are lots of other databases out there, and if someone reading this has one that they love, feel free to mention it in the comments, but these are the two I’m going to focus on. I will quickly mention though that Mendeley’s database is often a good place to find more obscure articles. It is another generalist, but it allows author submissions, and on multiple occasions I have found papers there that didn’t show up elsewhere. So, while I wouldn’t use it as a primary database, it can be useful (you have to make an account, but it is free).

*If you are a research and have an account, you can manually add the bibliographic information for an article to Scholar, which may help Scholar to locate it if it hadn’t done so already, but you cannot simply upload an article.

How to search

Now let’s move on to how to actually find the papers you are after. For both PubMed and Scholar you can use them like a standard internet search and type in “vaccines autism,” for example, but that is going to return a ton of studies, so it is usually best to be as specific as possible. For example, if you specifically want to see results from randomized controlled trials, include that in your search terms.

Both databases also have very helpful advanced search settings. For PubMed, there is an “advanced” tab under the main search bar, and this returns a screen with a bunch of pretty self-explanatory options. For example, you can limit results to a specific author, specific journal, specific date range, specific word in the title, etc. Google Scholar is similar, but with fewer options (to get to it, click on the three lines on the left-hand side indicating a drop-down menu, then select “Advanced search”).

It can also be useful to either include or exclude specific words or phrases. PubMed and Scholar both let you include specific words or phrases by simply putting the word or phrase that you care about in quotes, at which point they will limit the searches to articles that contain that quote. This can be very useful if you are getting a lot of irrelevant results that include some parts of your search terms, but not exact phrases you are after. Conversely, there may be times when it is useful to eliminate a word. For example, if you are only interested in studies on humans, you might want to exclude a word like “mice” or “in vitro.” In PubMed, this has to be set in the advanced search option, but in Scholar, you can just ad a minus sign to the beginning of the word (or quoted phrase) that you want to exclude. This should be done cautiously, however, as you may inadvertently exclude relevant studies. For example, if you exclude the word “mice” you may accidentally exclude a study on humans that discussed rodent studies in the introduction or discussion, or even just cited a study with the word “mice” in the title. So, while this feature can be useful, it should be used carefully, and it is often better to put quotes around a word you care about, rather than eliminating a word. For example, you could put “human” in quotes, to force the search to give you more human trials. Having said that, quotes can bias search results and make it easier to cherry pick results (particularly when using long phrases). So, use these tools carefully.

Another really useful approach is to find one relevant study, then look both at the studies it cited and the studies cited by it. To my knowledge, PubMed does not have a “cited by” tab, but Scholar does under each article, thus allowing you to see which articles cited it. Also, both databases have a “related articles” or “similar articles” link under each article, which you can use to find other relevant research.

Personally, I find the citations within a paper to be the most useful. If you really want to understand a topic, then as you go through a paper, you should note the references to related studies that are worth reading. Then, you can use the literature cited section of the paper and Scholar or PubMed to look up those articles and read them. As you read them, you should find yet more articles. As you can well imagine, the number of articles you need to read balloons out pretty quickly, and it is why scientists have to spend so much time reading. This can, however, also provide a useful check for how well you have covered a topic. After reading a large number of papers, you should start to notice that the number of new, relevant papers being cited decreases. You should start to see a lot of familiar citations to papers you’ve already read. In other words, at first, the number of new citations to papers you need to read should be quite large after each paper you read, and that number will continue to grow until you start to get a good grasp on the literature. Then, it will slowly start to decrease as you read more and more of the relevant studies (i.e., it becomes harder and harder to find papers you haven’t read yet). This doesn’t mean that you are an expert and have read all relevant studies, of course, but it is a useful proxy for assessing your thoroughness.

How to get papers for free

Now comes the critical question, how do you actually get the paper without paying for it? In many cases, you can do so directly though Google Scholar or PubMed (Scholar is particularly good at finding and including links to free copies if they are available). Failing that, you have several options.

The first, is to do a standard Google search for the title of the paper. Sometimes, this brings up copies that Scholar missed. You can also check Research Gate and Mendelely, but usually Scholar picks those up. For papers on “physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics,” you can also try arXiv.org, which is run by Cornell and offers free, legal, open access to many papers in those fields.

The second option (which is often the best) is simply to contact the author and ask for a copy. In almost every case, they will be more than happy to send one to you. I want to pause here for a moment to make a brief point. Scientists do not get paid for their publications. Those fees to access papers go directly and entirely to the publishers. Scientists do not get one cent from them. So, don’t feel bad about asking a scientist for their research, because you aren’t costing them anything, and they will be thrilled to know that someone is interested in their work.

To actually get a hold of an author, email is usually the best option. At least one author always includes an email address on the paper. If that address doesn’t work, they may have switched universities, but a Google search will usually bring up their current position with their current email. Failing that, you can try to contact them via Research Gate, but at least for me personally, I find that to be an inefficient way for people to get in touch with me. I don’t get notifications from my Research Gate (because they were obnoxious) nor do I check it often, so when people ask me for my papers via Research Gate, it often takes me a long time to respond. In contrast, emailing me usually results in a response is a few hours. I think this is probably true for most academics, so I’d start with email.

One final note about emailing scientists, sometimes people feel like they are inconveniencing scientists by asking for a paper (particularly people who are not academics or students) so they write them a lengthy story about what they are interested in and why they want the paper. Don’t do that. You don’t need to justify your desire for knowledge and you are just wasting their time. All you need to say is, “Dear Dr X, could you please send me a copy of your paper titled, “Y.” Thank you very much, Your Name” or something to that effect. It doesn’t have to be quite that terse, but academics often get hundreds of emails a day, so keeping your message short is appreciated.

If all of that has failed, you can go old school and drive to a University, go to the periodical room of their library, and read the actual physical journal. It sounds antiquated, but periodical rooms are pretty neat, and some older papers haven’t been digitalized.

Finally, there’s always Sci-Hub. Sci-Hub has often been called the “Pirate’s Bay of academia,” and that is pretty apt. I don’t pretend to know all the details of how it works, but basically, the people who run it got access to a bunch of log-in credentials for journals and have used them to make those journals available to everyone. So, you can go to the site, drop a URL, title, or DOI for a paper, and 99% of the time, a free PDF will open. Is it legal? That is questionable. It has been sued several times, and it has had to switch domain names more than once. In my opinion, however, a more relevant question is, “is it ethical?” and as far as I’m concerned, the answer is, “yes.”

For obvious reasons, I cannot tell you that you should be using Sci-Hub, but I will tell you my personal view on the situation. I think that information should be available to anyone who wants it, and I think that it is wrong for data to be locked behind paywalls (particularly given how much research is publicly funded via tax dollars). I also think that the current publishing system is an unethical scam. Without going too much into the details, scientists have to pay “page charges” to publish in most journals, ostensibly to cover the cost to the journal for their editorial staff (see section later on predatory journals). Then, the journals sell the papers, and, as mentioned earlier, the scientists get no money back. Every single year, millions (probably billions) of dollars of grant money are paid by scientists for the privilege of being allowed to publish our work. Meanwhile, the journals rake in billions of dollars in profit from selling the articles, and in turn, stopping many people from having access to them.

To put all of that another way, the money flow goes like this:

  1. You pay the government via taxes
  2. The government gives a tiny portion of that to scientists to do research
  3. Scientists have to spend a good chunk of that money to publish their research
  4. Journals make billions of dollars in profit by charging you (the public) to access the results of the research that you already paid for via taxes.

It is an insane system that robs scientists of countless amounts of precious research funding that we could be using to actually test new questions, all while preventing many from reading the research that, in many cases, they funded with their taxes. Sadly, scientists are trapped in this system. We have to publish our research, and if we want good jobs, we have to publish in high-ranking journals, which means we have to publish in journals that charge us. Publishers know this and exploit it. Papers often cost $3,000 or more to publish. So, if you want to know my personal opinion about academic publishing companies and whether or not it is ethical to bypass their fees via Sci-Hub, I say screw them. It’s a stupid, unethical system that should be overthrown. Read up me hearties, it’s a pirate’s life for me (here endeth my rant).

Organizing your papers

This is somewhat tangential, but I think it is important. As you read papers, you should be taking notes and organizing your papers in a way that makes it easy for you to find the papers again in the future. There are several reference organizing programs specifically for this purpose, with Mendeley and Endnote being the two front runners. I started using Mendeley years ago (before it was bought by one of the massive publishers I just ranted about) and moving to a new system now would be too difficult to be worth it. Having said that, I’m really happy with Mendeley. It is free unless you need to store an ungodly number of pdfs, and it lets you organize papers in a lot of useful ways. You can create folders in the program to store different categories of papers, highlight the text, and write notes. Most usefully of all (IMO) you can “tag” papers with custom tags, then subset within a folder (or your whole collection) by those tags. For example, you could have a folder called “climate change” and tags such as: models, hurricanes, and heat waves. Then, if you need to look at a paper on hurricanes, for example, you can just subset by that tag. On top of that, you can then sort by title, author, journal, etc., or do a search for text in your notes or the papers themselves. Additionally, Mendeley backs up to the cloud, so you can access your files from any computer with an internet connection. It is very useful, and I highly recommend it (or EndNote or some other program) if you plan on reading lots of papers.

Predatory journals and reading critically

Finally, I need to make an important point about critically assessing the results you get from your searches. First, as mentioned earlier, databases like Scholar may return results other than peer-reviewed articles. So just because it showed up in the results, doesn’t automatically make it valid research.

Second, there are, unfortunately, a large number of “predatory journals.” These are, to a large extent, “pay-to-publish” journals that lack an actual peer-review system. I need to explain what I mean by this carefully, because this is not the same thing as the page charges I mentioned earlier. For real journals, you submit your paper for review with the acknowledgement that you are willing to pay the charges if the paper is accepted. Then, the paper goes out for review by other scientists, and if it is accepted you have to pay the charges. These journals care greatly about their reputation and at least try to keep shoddy research from being published (though see the next two paragraphs). In contrast, predatory journals are not real journals. They don’t actually do proper peer-review. You pay them just to publish any junk paper without critically assessing it. They are frauds and should not be treated as if they are real journals. Sometimes proper scientists get duped by them, but an awful lot of the papers in them are there because no legitimate journals would take them. Spotting predatory journals can be hard, but Beale’s List has a pretty good collection of journals and publishers to watch out for.

Beyond predatory journals, there is a wide range in quality for journals. Some journals aren’t technically predatory, but also aren’t really legitimate. To give a really extreme example, a while ago, a Bigfoot “researcher” was tired of actual journals rejecting their nonsense paper, so they started their own journal (de Novo) and published their “paper” there. I’m sure they reviewed their own paper with the highest of standards (sarcasm). That’s obviously the far end of the spectrum, but there are many journals out there that appear reputable, but actually have a strong bias towards fringe positions and tend to have pretty lax standards for review (looking at journals’ editorial boards, their scope, and their impact factor can be helpful for evaluate them).

Further, even really good journals sometimes publish bad papers. As I have said repeatedly on this blog, the peer-review system is good, but it is far from perfect, so you always have to read critically and look for a consensus of studies. The fact that a study found X doesn’t mean that X is automatically true. Scrutinize the study. Ask questions like, was this published in a reputable journal? Was the sample size large enough? Did they control confounding factors? Did they use appropriate statistics? Then, look at what other studies have found. Look at the entire body of literature rather than cherry-picking a handful of studies that agree with you. If there actually is good evidence that X is true, then you should find multiple large studies that used good methodologies and were published in reputable journals, and you should find few studies that disagree (or the dissenting studies should have small sample sizes, be published in questionable journals, etc.).

In short, databases like Google Scholar and PubMed are wonderful, powerful tools, but with great power comes great responsibility. It is extremely easy to do a quick search, find a paper that confirms your biases, then ignore all other studies and claim that you are right and everyone else is wrong, but it is your responsibility to avoid that temptation. It is your responsibility to be intellectually honest, read papers critically, and carefully examine the entire body of research, not just the studies that confirm your biases.

Key points

  • Google Scholar and PubMed are great databases for scientific research
  • Their advanced search options are very useful for wading through a mountain of literature
  • Citations within papers are also very useful for finding other relevant research
  • Papers that are behind paywalls can be obtained for free by either contacting authors (totally legal) or using Sci-Hub (questionably legal)
  • Some journals are “predatory” and do not conduct a proper peer-review
  • Journals and papers range widely in quality and you should avoid blindly believing the first study that agrees with you. Read critically and look at the entire body of literature.
Posted in Uncategorized | Tagged , , , | 14 Comments

Can we blame climate change for storms like Dorian?

Posted on September 10, 2019 by Fallacy Man

Last week, the world watched in horror as Hurricane Dorian ripped though the Atlantic, leaving devastation in its path, and as usual, debates online have swirled around the same question that arises every time there is a major weather event, “is this climate change?” Many people point to storms like Dorian as evidence of climate change and the very real dangers it poses, but others cry foul and insist that climatologists and science-advocates are being hypocritical and inconsistent. “Whenever we point to a blizzard or cold front as evidence that the climate isn’t changing,” they say, “you all accuse us of cherry picking and shout that ‘weather isn’t climate,’ but when there is a hurricane or a heatwave, suddenly you’re convinced it’s global warming.” This accusation of a double standard deserves some discussion, because there is a small kernel of truth to it, but, as usual (always?) there are key pieces that climate change contrarians are ignoring. So, let’s look at this for a few minutes.

Let’s start by dealing with the part that contrarians get right. If someone’s entire argument is, “it was hot today, therefore climate change is real” or “there was a big heatwave, hurricane, drought, etc., therefore climate change is real” then they are, in fact, conflating weather with climate. I will admit that this line of reasoning is no different than saying, “it was cold today, therefore climate change isn’t real.” However, and I can’t stress the importance of this enough, that is almost never what I see people doing. Rather, the argument usually points to a current weather event as the latest in a trend of changing weather patterns. This is a key distinction for several reasons.

First, we need to consider why it is problematic to conflate weather and climate. Weather is what happens over a short period of time, whereas climate refers to long term trends and patterns. Thus, an individual storm, cold front, etc. is weather, but a pattern of increasing storms, heatwaves, etc. over time is climate. So, when people use a storm like Dorian as an example of the latest event in a trend of changes, they aren’t conflating weather and climate, rather they are talking about climate. When we look at the trends, we see that the average intensity and the proportion of hurricanes/cyclones that are very powerful has increased over time, just as climate models predicted (see note 1: Emanuel 2005; Elsner et al. 2008; Holland and Bruyere 2014; Walsh et al. 2016). There is also a trend of hurricanes hitting further from the equator (Kossin et al. 2014), and a pattern of storms increasingly “stalling” (just as Dorian did; Kossin 2018; Hall and Kossin 2019; see note 2). So, it is completely valid to talk about a storm like Dorian in that context. By the same token, it is completely right and proper to bring up climate change when a heatwave occurs, because there are strong, decades-long trends of heatwaves increasing in frequency, duration, intensity, and the extent of area they affect (Klein Tank and Konnen 2003; Della-Marta et al. 2007; Russo et al. 2014; Tanarhte et al. 2015; Perkins et al. 2012; Habeeb et al. 2015). In contrast, saying “this part of the world experienced a record cold this week” and using that as evidence against climate change is using an isolated weather event, rather than a climate trend. There is no trend of increasingly cold winters. Rather, there is a global trend of increasingly warm temperatures. Indeed, as of the writing of this post (2019), all five of the five hottest years on record occurred in the past five years, and 18 of the 20 hottest years happened in the past 20 years (based in NASA’s data; the remaining two hottest years occurred in 1997 and 1998). There is a strong trend of increasing temperatures, which is why we are justified in bringing up climate change when we experience record-breaking temperatures (no, the warming hasn’t paused, that’s a myth).

This brings me to my second, and closely related, point: the nature of cherry picking. Cherry picking occurs when you isolate and cling to any pieces of evidence that conform to your beliefs while ignoring a (usually larger) body of evidence that disagrees with you. Hopefully from the paragraph above you can see where I am going with this. Citing a particular cold front, blizzard, etc. as evidence against climate change is, by definition, cherry picking, because it is using isolated events rather than trends, and it is ignoring the big picture of increasing global mean temperatures. Conversely, talking about heatwaves, hurricanes, etc. in the context of their trends is not cherry picking. It is literally the opposite of cherry picking because it is about trends and the big picture. On that note, I want to go down a brief side tangent to point out that using regional data to argue against climate change is also cherry picking. No one ever said that every part of the planet will be warmer all the time. Rather, we are talking about global patterns and global temperatures. Citing a few cherry-picked locations ignores that big picture.

By way of example, it would be crazy to isolate one person who smoked their whole life and never got cancer and say, “see, smoking doesn’t cause cancer because they smoked and are fine.” That would obviously be cherry picking, and it would be nuts because it would ignore the overarching picture of the strong trend of smoking increasing cancer risk. That is, however, exactly what climate change deniers are doing when they cherry pick a particular cold front, blizzard, ice sheet, etc.

Now, at this point, someone is inevitably thinking, “yeah, but you can never actually say that a particular cyclone, heatwave, etc. was caused by climate change, so it’s still deceptive to try to blame climate change when one happens.” My response to this is, again, two-fold. First, in many cases, we can actually use statistical techniques to show that certain weather events were so extraordinary that they were unlikely to have occurred naturally. For example, see the 2003 heatwave that caused over 70,000 deaths in Europe (Schar et al. 2004; Stott et al. 2004; Robine 2008). Similarly, studies showed that climate change very likely contributed to the extreme rainfall observed during Hurricane Harvey in 2017 (Risser and Wehner 2017; van Oldenborgh et al. 2017; Wang et al. 2018). There are many storms like this where we can, in fact, say with a high degree of confidence that climate change contributed to them (note: science never deals in 100% confidence).

The second consideration (again, related to the first), is that although we can never say with 100% certainty that a given event was caused by climate change, we can say that climate change is increasing them or their intensity, so we are justified in using them as examples of the dangers of climate change, and whether or not climate change caused any one particular storm is irrelevant, because we know the effect climate change is having in general.

Let’s go back to smoking as an example. Smoking causes cancer. This is well-established. However, that does not mean that everyone who smokes will get cancer, nor does it mean that everyone who gets lung cancer smoked. As a result, we can never say with 100% certainty that any particular case of cancer was caused by smoking, but that is really beside the point. The point is that there is a general trend of smoking increasing cancer risks. Thus there is a high probability that a smoker’s cancer was connected to their smoking, and it makes perfect sense to show people pictures of cancerous lungs and stories of people who suffered from lung cancer after a history of smoking. To put that another way, we can never point to an individual and say with 100% certainty that smoking caused their cancer, but we can point to them as an example in a general trend that we should take seriously. The same is true with climate change. Whether or not a particular storm was definitely caused by climate change is beside the point. The point is that climate change is making intense hurricanes more common, heatwaves more intense, frequent, and long, droughts more common in some areas while floods increase in others, etc. These are serious issues that need to be treated accordingly, and it is completely right and proper to talk about climate change when these extreme weather events occur.

Note 1: It is important to clarify that it is the average intensity and proportion of hurricanes/cyclones that are high intensity (e.g., category 4 and 5) that is increasing, not the total number of hurricanes/cyclones. This is consistent with model predictions.

 Note 2: There is a general pattern of increasing hurricane stalling, but there is some disagreement among scientists about the cause. Several models did predict a slowdown in hurricane transition speed due to climate change, but others predicted no change. So not all scientists agree that climate change is the cause (despite what many think, scientists love to argue and don’t automatically assume that everything is being caused by climate change). I personally think that the evidence more compelling suggests that climate change is the cause, but I will wait for more data before reaching a verdict, and I wanted to include this note since the evidence does not point to climate change as conclusively as it does on other issues (despite what many think, I really do care about accurately portraying evidence and the scientific literature).

Related posts

Literature cited

  • Della-Marta et al. 2007. Doubled length of Western European summer heatwaves since 1880. Atmospheres 112:D15103.
  • Elsner et al. 2008. The increasing intensity of the strongest tropical cyclones. Nature 455:92–95.
  • Emanuel 2005. Increasing destructiveness of tropical cyclones over the past 30 years. Nature 436:686–688.
  • Habeeb et al. 2015. Rising heatwave trends in large US cities. Natural Hazards 46:1651–1655.
  • Hall and Kossin 2019. Hurricane stalling along the North American coast and implications for rainfall. Climate and Atmospheric Science 2
  • Holland and Bruyere 2014. Recent intense hurricane response to global climate change. Climate Dynamics 42:617–627.
  • Kossin et al. 2014. The poleward migration of the location of tropical cyclone maximum intensity. Nature 509:349–352.
  • Kossin 2018. A global slowdown of tropical-cyclone translation speed. Nature Letters 558: 104-108.
  • Klein Tank and Konnen 2003. Trends in indices of daily temperature and precipitation extremes in Europe, 1946–99. Journal of Climate 16:3665­–3680.
  • Perkins et al. 2012. Increasing frequency, intensity and duration of observed global heatwaves and warm spells. Geophysical Research Letters 39:L20714.
  • Risser and Wehner 2017. Attributable Human‐Induced Changes in the Likelihood and Magnitude of the Observed Extreme Precipitation during Hurricane Harvey. Geophysical Research Letters 44:12457–12464.
  • Robine et al. 2008. Death toll exceeded 70,000 in Europe during the summer of 2003. Epidemiology 331:171–181.
  • Schar et al. 2004. The role of increasing temperature variability in European summer heatwaves. Nature 427:332–336.
  • Stott et al. 2004. Human contribution to the European heatwave of 2003. Nature 432:610–614.
  • Tanarhte et al. 2015. Heatwave characteristics in the eastern Mediterranean and middle East using extreme value theory. Climate Research 63:99–113.
  • van Oldenborgh et al. 2017. Attribution of extreme rainfall from Hurricane Harvey, August 2017. Environmental Research Letters 13:019501.
  • Walsh et al. 2016. Tropical cyclones and climate change. Climate Change 7:65–89.
  • Wang et al. 2018. Quantitative attribution of climate effects on Hurricane Harvey’s extreme rainfall in Texas. Environmental Research Letters 13:054014
Posted in Global Warming | Tagged | 1 Comment