Even if you have never paid any real attention to the climate change “debate,” you have probably seen someone say that, “97% of climatologists agree that we are causing climate change.” This is a number that I have personally cited on numerous occasions, and it is a number that is highly contested by the climate change deniers. Indeed, I rarely mention the consensus without people responding by adamantly proclaiming that the 97% number is a myth, and the study that produced it (Cook et al. 2013) has been debunked. Therefore, in this post, I want to deal with the consensus on climate change from several angles. First, I want to focus on the prominent Cook et al. study and explain what the authors actually did, what they found, and why their study was robust. I also want to deal with some of the common criticisms of their study. Finally, I want to look at several other lines of evidence that show that there is a strong consensus on global climate change.

 

How was the Cook et al. study conducted?

The key study in question is Cook et al. 2013 “Quantifying the consensus on anthropogenic global warming in the scientific literature,” so I want to quickly run through what the authors actually did. They accessed the ISI Web of Science database (this is a database for scientific publications), and searched it for any articles on “global warming” or “global climate change” that were published between 1991 and May 2012. This returned 12,465 papers, but 186 were not peer-reviewed, and 288 were not actually on climate change, so those papers were eliminated. This left them with 11,944 papers, written by 29,083 authors, and published in 1980 journals (that’s a pretty large data set).

The abstracts of these papers were anonymously rated by two reviewers who could only see the titles and abstracts of each paper. A total of 24 people reviewed the abstracts, but 12 people were responsible for 97.4% of the ratings. Each paper was assigned one of seven categories which were later lumped into three broad categories: explicit or implicit agreement that humans are causing the climate to change, no statement or uncertainty about humans changing the climate, and explicit or implicit rejection of the notion that humans are changing the climate. Thirty-three percent of papers had disagreeing endorsement ratings (based on the initial  seven level system), so those papers were sent back to the reviewers to be re-assessed. After re-assessment, only 16% disagreed, and those papers were then rated by a third party. Finally, they emailed 8,547 authors (they used the emails provided in the publications), and asked them to rate their own papers (these self-rated papers are going to be very important later).

 

What did Cook et al. find?

Out of the 11,944 papers that they examined, 32.6% endorsed the idea that humans are changing the climate, 0.7% rejected it, and 66.7% were uncertain or made no statement. They randomly selected 1,000 of the uncertain/no statement papers and carefully examined their abstracts to determine if they actually expressed uncertainty (e.g., “While the extent of human-induced global warming is inconclusive. . .”) or simply did not make any statement about whether or not humans were at fault. This revealed that only a tiny percentage (0.3% of the papers) were truly uncertain, with the majority simply not making any statements. From that, they found that of the papers that expressed an opinion on climate change (accept, uncertain, or reject), 97% agreed that we are causing the climate to change. When they looked at authorship of those papers, they found that 98.4% of all authors endorsed anthropogenic climate change. They also found that the percent of abstracts rejecting human induced climate change stayed the same overtime, but the percentage accepting it decreased, and the percent not making a statement increased (that is actually a really important point that I will return to later).

For the self-rated papers, they received replies from 1,200 authors, and the responses were really interesting. Among the papers that received a self-rating, 62.7% were self-rated as endorsing anthropogenic global warming (i.e., the authors rated their own papers as endorsing it), whereas Cook et al. had only rated 36.9% of those abstracts as endorsing it. Similarly, 35.5% of abstracts were self rated as having no position or being undecided, whereas Cook et al. had assigned 62.5% of the papers to that category. Finally, authors self-rated 1.8% of abstracts as rejecting human induced climate change, and Cook et al. rated 0.6% as rejecting it. So, even among the abstracts that were self-rated as having an opinion on climate change,  97% endorsed human-caused climate change (note: this number was originally reported as 99.7 due to an error on my part, I apologize for the mistake).

 

Why the results of Cook et al. are robust/responses to critics

Large sample size
In statistics, the larger your sample size, the more accurate your results will be. Cook et al. examined roughly 12,000 abstracts, which is a very large sample and should yield solid results. Nevertheless, it clearly does not include all of the publications on global climate change, and many critics have been quick to jump on this. Richard Tol is one of the most outspoken examples of this. He argued that Cook et al. should have used the search term, “climate change” rather than “global climate change,” because the former returns more hits. The problem with this argument is that there is no good reason to think that using different search terms would have yielded substantially different results. For example, when Tol compared the search results from both search terms, he found that using “global climate change” under-represented meteorology (by 0.7%), geosciences (2.9%), physical geography (1.9%), and oceanography (0.4%). These differences are, however, fairly minor, and there is no a priori reason to think that those small differences would result in a large difference in the results of Cook et al. Indeed, Tol even admits that those differences, “likely introduce a bias against endorsement.” In other words, the level of agreement could be even higher among the larger sample.

Tol also argues that Cook et al. should have used the Scopus database, rather than the Web of Science. His argument is that the Web of Science is more exclusive than Scopus, and would thus bias against articles published in fairly minor journals, which are often more likely to publish papers opposing climate change. He is technically correct, but here is the important catch: papers in minor journals tend to be of lower quality than papers in large journals. If we want to see whether or not there is any real, significant debate on climate change, we should be focusing on the high impact journals, so using the Web of Science would likely give a more accurate representation of whether or not there is any serious debate. As with the different search terms, however, I would be very surprised if using a different database gave substantially different results. In other words, you might get 95% or 99% rather than 97%, but any of those values still represents an overwhelming consensus.

There are several other things to note about Tol’s paper. First, there are several mathematical mistakes and irregularities which you can find explanations of here and here (the latter is a published response by Cook et al.). Second, realize that Tol does not disagree with the notion that there is a scientific consensus, he simply disagrees with the methods used in Cook et al. To quote Tol’s paper,

“There is no doubt in my mind that the literature on climate change overwhelmingly supports the hypothesis that climate change is caused by humans.”

So even if you want to blindly side with Tol, that still doesn’t give you support for the notion that there isn’t a consensus.

 

The use of self-ratings strongly supports the consensus
The single most common criticism that I hear about Cook et al. is that their review system may have been biased. In other words, the people rating the abstracts were biased towards climate change, and therefore rated the papers in a way that favored their bias. Also, there is admittedly some evidence that the authors were setting out to prove that there was a consensus, which is a huge taboo in science and gave me some personal reservations when I first stated looking into this paper; however, the argument that the authors biased the results ignores one of the most important components of the Cook et al. study. Namely, they received self-ratings from 1,200 authors, and the self ratings showed a stronger consensus than the Cook et al. ratings! There was very little difference among papers that disagreed with climate change (0.6% by Cook et al., 1.8% by self rating), but there was a huge difference in the number of papers that were rated as agreeing that humans were causing climate change. Cook et al. only rated 36.9% of the papers as agreeing that humans are at fault, but among the self ratings, 62.7% agreed. In other words, the reviewers who rated the papers for Cook et al., were conservative and actually classified many papers as “no opinion” when they should have been listed as “endorses anthropogenic climate change.”

This brings me back to my central point: I’m not arguing that Cook et al. is utterly infallible. I have no doubt that slightly different methods would have yielded slightly different results, but there is no reason to think that different methodology would have produced substantially different results, and the results of the self-ratings show that the assessment ratings being used by Cook et al. did not bias things in the favor of anthropogenic climate change.

 

It is valid to calculate agreement only among papers that expressed a view
We need to talk for a minute about the large number of papers that neither endorsed nor rejected anthropogenic climate change. First, I often encounter people who think that Cook et al. simply threw those papers out because they didn’t agree with them, but that argument is clearly untrue. Those papers were included in the analyses of publications over time (see the next point), and a subset of them were re-analyzed, which revealed that only a tiny portion of them (0.3% of papers) actually expressed uncertainty (i.e., the authors stated that they were unsure about climate change), and the vast majority of them simply made no statement about whether or not climate change was caused be humans.

The next argument that I often encounter is that Cook et al. didn’t actually find a 97% consensus, but rather they found that 97% of papers that expressed a view agreed with anthropogenic climate change. In other words, 7,930 papers made no statement on anthropogenic climate change, so those papers were excluded when calculating the percent agreement.  Although this argument is technically true, it is extremely shoddy and ignores basic math. First, it’s important to stress that not stating an opinion and not having an opinion are two very different things. If the abstract didn’t state an opinion, then you cannot conclude anything about those authors’ views, and, as a result, you cannot include them when trying to calculate the level of agreement. That should be intuitively obvious, but I’ll use an example to try to illustrate this. Suppose that I surveyed 12,000 people and asked them if the earth was round, and 33% of them agreed (3960 people), 1% disagreed (120 people), and 66% of them did not reply (7920 people). How would I calculate the level of agreement? Would I only use the people that responded, and say that 97.1% agreed, or would I use all of the people (even those who didn’t respond) and say that 33% agreed? Obviously I would do the former. It would be absurd to include people who didn’t even express an opinion, but that situation is no different from what Cook et al. did. They “surveyed” roughly 12,000 abstracts, and roughly 66% of them “didn’t respond” (i.e., didn’t express an opinion), therefore they only included those that did express an opinion in their calculations. This is standard practice for surveys.

 

The percentage of papers that didn’t express an opinion increased over time
This may seem counter intuitive at first, but the large percentage of papers that didn’t express a view, and the fact that the percentage increased over time actually provides support for a consensus. I say that because when something is well established, there is no need to state your position or argue for it; whereas when something is highly contested, it is important to state where you stand and defend your position. For example, I have written several papers on evolution (or that discuss the evolution of a trait), but if you were to assess their abstracts using the criteria of Cook et al. (modified for acceptance of evolution), they would get put into the “no opinion” category because I did not affirm that I accepted the theory of evolution by natural selection. Importantly, I didn’t affirm that simply because the theory is accepted by virtually all scientists. There is no debate on it, and, therefore, there is no need for me to affirm that it is correct. In contrast, if I am writing a paper on a controversial position, I am going to state my view and defend it. So a large number of papers that neither endorse nor reject climate change is actually what you would expect from a strong consensus, and the increase in those papers over time suggests that the consensus is growing.

 

The results are important and useful
A final criticism that I frequently encounter is that Cook et al. is worthless because the disagreement among scientists is about the extent of climate change, not whether or not humans are having some impact on it, and Cook et al. only showed agreement that we are influencing climate change, without clarifying the extent of the influence. Bloggers and authors such as Montford and Legates et al. ramble on endlessly about this as if it is a significant critique of the paper (you can find a response to Legates here); however, arguing that Cook et al. are wrong because they didn’t document agreement about the extent of climate change is a strawman fallacy, because Cook et al. made no claims of having documented such an agreement. Rather, Cook et al. simply claimed to (and indeed succeed at) documenting widespread agreement that humans are causing the climate to change. Despite the many strawman fallacies, this is actually an important result because there are still many people in the general public who deny any suggestion that humans are causing the climate to change.

Addendum 13-11-15: Several people have been critical of both Cook et al. and my assessment of that paper because Cook et al. only found 65 papers which explicitly stated that humans are causing 50% or more of the warming. There are two problems with this criticism. First, the Cook et al. paper made no pretense about having document agreement on the amount of warming being caused by humans, it only stated that there is agreement that we are having an impact. Second, the fact that only 65 abstracts specified that humans are causing at least 50% of the warming does not mean that only the authors of those particular studies support that position. In science, anytime that you give a quantification, you have to back it up. In other words, for any paper to include an estimate of the amount of warming being caused by humans, it would have to include a rigorous analysis of that question in the paper, but such an analysis is well beyond the scope of most climate change papers. Most papers on climate change deal with one particular aspect of the problem, not the overarching picture. Therefore, we would expect very few papers to give an actual estimate of the total amount of warming being caused by humans. So you cannot misconstrue those 65 papers as evidence that there is little agreement on the extent of the warming. To be clear, you also can’t use it as evidence of agreement. In other words, the data provided in this paper simply cannot be used to address the question of whether or not there is agreement about the extent of the warming that is being caused by humans. That is a very real limitation of the paper, but it doesn’t make the paper a fraud.

 

Other lines of evidence
Beyond the Cook et al. paper, there are multiple other lines of evidence which show that there is a strong scientific consensus on climate change, and I will briefly discuss several of them (there are others which all have essentially the same results, but these are the most prominent ones).

 

Anderegg et al. 2010 “Expert Credibility in Climate Change.”
In this study (which was published in PNAS, a very prestigious journal), the authors identified 908 expert climatologists (defined as those that have published at least 20 papers on climate change) and rated them as agreeing or disagreeing with the idea that the climate is changing and humans are “very likely” responsible for “most” of the warming (ratings were based on signed statements about climate change). They then subset the data to look at the consensus among different levels of expertise (again, expertise was defined based on the number of relevant publications). They found that agreement with anthropocentric climate change increased as expertise increased, and among the the highest levels of expertise, 97-98% of scientists agreed that humans were the largest factor causing the climate to change.

This study is admittedly dependent on the authors’ use of publications as a proxy for expertise, and its results have to be presented very carefully to avoid a strawman. It would not, for example, be fair to use this paper as evidence that 97% of all scientists agree on climate change. Indeed, when looking at a broader data set, the authors found that roughly 80% of all the scientists that disagreed with climate change had fewer than 20 publications and were thus eliminated from the analysis. So all that this paper shows is that there is strong agreement among the most well published climatologists. In my opinion, that is a really useful result, because publication record is one of the most common metrics against which researchers are judged, and if there is significant debate on an issue, you would surely expect it to be represented among the top researchers, rather than just among minor players who rarely publish.

In the interest of openness, Bodenstein published an article criticizing Anderegg’s use of publication records as a proxy of expertise, and Anderegg et al. wrote a response to that response. You can read Bodenstein’s argument here, and Anderegg’s response here.

 

James Lawrence Powell’s literature searches
Powell combed through the scientific literature from 1991-2012 looking for any papers that rejected the idea that humans are changing the climate. Out of 13,950 papers on climate change, he only found 24 that rejected anthropogenic climate change. Later, he followed up that survey by looking at papers from November 2012 to December 2013, and out of 2,258 articles on climate change, only one rejected the idea that humans are causing it. To be clear, his survey was not peer-reviewed, but you are welcome to repeat his methods yourself (you’ll get the same result).

 

American Association for the Advancement of Science (AAAS) survey
A recent survey of AAAS scientists found that 87% of them thought that “climate change is mostly due to human activity.” This number is clearly lower than the one calculated by Cook et al., but it still represents a significant consensus. Also there are two things that should be noted about it. First, it is asking a different (and more specific) question than Cook et al. asked. Second, this survey was conducted across all AAAS scientists, not just climatologists, whereas Cook et al. looked at actual publications on climate change. It is not at all uncommon to have a lower consensus among non-experts than among those who are actively conducting research in a given field.

Doran and Zimmerman 2009 “Examining the Scientific Consensus on Climate Change”
This survey polled 10,257 “Earth scientists” and asked them two questions:

1. When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?
2. Do you think human activity is a significant contributing factor in changing mean global temperatures?

The authors received 3,146 responses. As with the AAAS survey, this survey was not specific to climatologists, and only 79 individuals both listed climatology as their area of expertise and had strong publication records in climate change (i.e., at least 50% of their recent publications were on climate change).

Among the general body of scientists, 90% answered “risen” for question 1, and 82% answered “yes” for question two. Among the climatologists, 96.2% answered “risen” for question 1, and 97.4% answered “yes” for question 2.

The biggest criticism of this survey is obviously sample size. Seventy-nine climatologists is obviously a very small sample, and it is admittedly difficult to get a meaningful representation from such a small sample size. So in isolation, this paper can’t tell us much about the views of climatologists, but when included as part of the larger body of literature, it has some value. It is also informative about the views of scientists more generally.

 

Fansworth and Lichter 2011 “The Structure of Scientific Opinion on Climate Change”
This study polled 998 members of the American Geophysical Union and American Meteorological Society and received 489 replies. 97% agreed that global temperatures have risen during the past century, and 84% agreed that humans are currently causing the climate to change via greenhouse gases. When asked to rate what they expected the future impacts of climate change to be, 44% thought that they would be moderated, and 41% thought that they would be server/catastrophic.

This study admittedly had a small sample size and wasn’t truly representative of climatologists more generally, but among those sampled, there was still a large consensus that we are causing the climate to change. There was, however, disagreement over the extent of the change, which is something that I have never denied.

 

Conclusions

In conclusion, the majority of the arguments presented against the Cook et al. paper misrepresent what the authors did or what they found. The reality is that they surveyed a massive body of literature, controlled for their personal biases by getting authors to rate their own papers, and found a roughly 97% agreement that humans are changing the climate. Is Cook et al. a perfect study? No. If I was conducting it, there are certainly things that I would have done differently, but none of the problems with the study are serious enough to expect the true level of consensus to be far from 97%.

Further, numerous other studies and surveys have looked at the same basic question from several angles, and all of them paint the same picture: there is a strong consensus that humans are causing the climate to change. Among the general scientific community, the consensus is generally reported in the 80s, and among actively publishing climatologists, it’s probably in the high 90s. Granted, some of these studies were small, and some of them used very specific, focused criteria, but others were quite extensive (such as Powell’s), and they all found the same thing. So when taken together, we have numerous lines of evidence which all point towards a strong consensus that humans are changing the climate. Indeed, the only study to find a noticeably different result (at least to my knowledge), was one that specifically surveyed scientists who worked for the petroleum industry (most of whom weren’t climatologists), and it’s hardly a surprise that they often rejected the idea that we are responsible for climate change.

Inevitably, there is going to be someone who is very unhappy with this article, so I want to make several things clear. First, if you want to say that there isn’t a consensus on climate change, then you must completely defeat all of these lines of evidence. Second, simply showing that these studies are invalid would not automatically show that there isn’t a consensus (that would be a fallacy fallacy [yes, that’s its name, not a typo]). In other words, showing that these studies were conducted incorrect would only mean that these studies could not be trusted. It would not mean that there isn’t a consensus. So if you want to argue that there isn’t a consensus, you must provide evidence for that position. In other words, you must find or conduct your own survey, literature review, etc. and show that there is strong disagreement on the issue. Nothing else will suffice.

In short, there is extremely strong agreement among experts that we are in fact causing the climate to change. Based on the available evidence, the agreement is roughly 97% among climatologists, but it may be slightly higher or slightly lower. Regardless of exactly what it is, however, it is clear that a strong consensus exists. The amount of change that we will cause is still debated, but the simple idea that we are causing the climate to change is “settled.”

 

Note: there have been numerous accusations of fraudulent behavior among the authors of the Cook et al. study, but none of those arguments stand up against the facts and basic logic, so I haven’t bothered to go through them here (you can’t say that someone committed fraud just because you don’t like what they have to say). The authors of the study have, however, written numerous posts explaining the study in more detail and responding to critics. You can find examples of them here, here, here, here, and here. Please give the authors a chance to defend themselves before you believe conspiracy theorist websites (ever heard of innocent until proven guilty?).