I spent my afternoon reviewing a paper for a scientific journal and making a recommendation about whether or not the paper should be published. As a scientist, this is not an uncommon task for me, but it is a process that is largely foreign to the general public. Indeed, the peer-review system often seems to be a mystery to those who don’t participate in it, and, as a result, it is a frequent topic for this blog. For example, I have previously written about what it takes to publish a paper. However, I have not yet written a post specifically about what it is like to be a reviewer or even who reviewers are. So, I thought I would take this opportunity to explain the process from a reviewer’s point of view and offer you a window into the system that determines which papers get published.

Who are reviewers and how are they chosen?
In short, reviewers are scientists. The peer-review system is rather interesting because essentially everyone involved with it acts as both an author and a reviewer. In other words, reviewers are themselves scientists who also submit papers for review. This is a good system, because it means that scientific papers are being reviewed by other scientists, not by politicians, corporations, etc.

Nevertheless, reviewers obviously aren’t chosen at random from within the scientific community. Rather, journal editors choose them based on relevant experience and expertise. This can happen in several ways. Often, journals require you to recommend reviewers when you submit your paper. In other words, as part of the submission process, you have to nominate several people to serve as reviewers, as well as providing contact information and (often) a justification for why they would make suitable reviewers. As a general rule, you want to suggest people who have published similar papers (often papers that you cited in the paper you are submitting) and who aren’t in anyway affiliated with you (this insightful post provides more details about how to select reviewers). Ultimately though, the editor has the final say, and he/she will consider your suggestions and make the final call.

Another common method that journals use to select reviewers is simply to recruit reviewers during the paper submission process (or via society memberships). In other words, the submission form often includes an optional check-box that says something to the effect of “I am willing to be a reviewer for this journal.” Generally, this is also accompanied by a section where you list your areas of expertise. Thus, if you are submitting to a herpetology journal (i.e., reptiles and amphibians) and you check the box and list yourself as a turtle expert, then you will go on the list of potential reviewers for turtle papers.

A third mechanism that editors use is simply to contact authors who recently published similar research in the journal that they edit. Indeed, after publishing a paper in a particular journal, it is quite common for an editor of that journal to contact you about reviewing a different paper for that journal (this has happened to me several times).

Finally, editors may choose simply to send the paper to someone who they know does research in a similar area or who was recommended to them by someone else. For example, the editor may have recently read a similar paper and contact one of the authors of that paper. Indeed, the paper that I reviewed today was in a journal that I have never published in or signed up to be a reviewer for, and the paper did not cite my previous work, so I doubt that the authors recommended me. However, I have previously published extremely similar and relevant research, so I suspect that I was selected by the editor because of my previous papers on this topic.

Regardless of how they are chosen, the point is that reviewers are generally experts on the topic of the paper that they are being asked to review. In other words, editors select scientists who have the necessary skills, knowledge, and experience to assess the paper and determine whether or not it is worthy of publication. Usually, at least two reviewers are selected per paper, but it is not uncommon to have three reviewers, and some journals use four or more.

Who can’t be a reviewer?
Most publishing scientists also serve as reviewers, but not all scientists are eligible to be a reviewer for a given paper. First, to be a potential reviewer, you have to have expertise on the particular field that the paper is about. I, for example, am predominately a herpetologist, so I would never get asked to review a paper on physics. For that matter, I wouldn’t even get asked to review a paper on botany. I might, however, be asked to review papers on other areas of zoology, but only if the topic of the paper was closely aligned with topics that I study (e.g., one of my areas of research is population ecology, and although I study the population ecology of reptiles and amphibians, the same concepts, statistics, etc. apply to other taxa, so I have the necessary skills and knowledge to assess a population paper on birds, for example).

A second criteria is a lack of conflicts of interest. Exactly how that is defined is variable, but as a general rule, reviewers should not be in anyway associated with the paper in question, and they should not be institutionally linked to any of the authors. I could not, for example, serve as a reviewer for a paper that was written by another graduate student in my lab, because that would be a conflict of interest (i.e., even though I would try to be objective, I would be less likely to criticize the paper because I work with and like the authors).

Finally, in many cases, authors can recommend people who should not serve as reviewers. Editors are under no obligation to follow these recommendations, but if there is someone who you really don’t want reviewing your paper, you can make that case. For example, if you have a long standing rivalry with someone and, as a result, that person likely would not be objective, then you can argue that he/she should not be selected as a reviewer. To be clear though, there needs to be a legitimate reason why that person would not make a good reviewer, and you can’t include someone in that list simply because you are concerned that they will find problems with your paper.

What happens when you receive a request to review a paper?
Each journal is different, but this is generally how things play out. First, you receive an email from an editor asking if you would be willing to review a given paper. This email usually includes the abstract for the paper, or, at the very least, the topic that the paper is on. It may also include other information such as the number of pages, words, figures, and/or tables in the paper. Sometimes this email will have information about the reviewing standards of that journal such as whether or not the review is anonymous and the time frame in which you are expected to provide the review.

As a reviewer, you then look at the information that you have been given and decide whether or not the paper is on a topic that you are sufficiently knowledgeable about, whether or not you have any conflicts of interest, and whether or not you currently have the time to review it. You then respond to the editor to tell them whether or not you are willing to be a reviewer. If you reject the request, it is generally considered good form to suggest an alternative reviewer (which is another way that editors identify potential reviewers).

Once you accept your duty as a reviewer, you generally receive a copy of the paper as well as paperwork on confidentiality. Some journals also provide you with a review template that you are supposed to fill out, whereas others let you provide feedback in whatever way you see fit. Generally, with either system, the final review will consist of several summary statements about your views on the paper, a list/explanations of your major criticisms, and a list of comments on specific lines (these are either annotations to the original document, or a list with corresponding line numbers).

What do reviewers look for?
Imagine that you are a reviewer, and you have just been given a paper to read. What are you going to look for? There are actually lots of things that you should look for, but the key thing to keep in mind is that the job of a reviewer is to assess the quality of the research rather than acting as an editor. In other words, reviewers do look at things like grammar, readability, and the presentation of the data (e.g. good/appropriate figures and tables), but their main duty is to act as a filtering mechanism that blocks bad science from getting published and helps authors to improve potentially useful research.

As such, the most important thing that reviewers look at is the methodology. They check things like sample size, experimental design, the statistics that were used, how the statistical models were set up, etc. All of this is intended to identify poor methodology and ensure that the study was done correctly. Following that, reviewers will check to see if the results were reported and discussed properly. It is not at all uncommon for authors to jump to conclusions that are not supported by the data, and it is the reviewers’ job to reign them back in and make sure that all of the conclusions are merited.

Additionally, reviewers are tasked with making sure that the paper is well grounded in the scientific literature. All research inevitably builds on previous research. As such, papers are supposed to cite and discuss relevant papers, especially if similar studies reached different conclusions. Thus, reviewers check to make sure that this was done appropriately. This is another reason why it is important for reviewers to be experts on the given field. As experts, they know the literature, so if important papers are missing, they will be able to point them out. Indeed, it is exceptionally common for reviewers to suggest specific papers that the authors should have cited (I made several such suggestions on the paper that I reviewed today).

Finally, it is the reviewers’ job to actually be helpful to the authors. Many otherwise nice people become utter jerks when reviewing, but the idea is actually for reviewers to provide constructive criticism that will help the authors improve their paper. Thus, reviewers are asked not only to point out problems with the paper, but also to suggest ways to fix those problems (e.g., “your statistical method is inappropriate, and you should use method X instead”).

What do reviewers recommend?
You have now read the paper and made a list of comments, which means that it is time for you to make a decision. Does this paper deserve to be published? You generally have 4–5 options to choose from (some journals don’t use option 2, and some journals use slight variations of these).

1. Reject without the option to resubmit — This means that the paper is seriously flawed and will rejected without further consideration.
2. Reject with the option to resubmit — This means that the paper has serious flaws, but it also has merit if those flaws can be correct. The authors can then revise the manuscript based on your comments and resubmit it back to that journal. At that point it goes back out for review, and you will often (but not always) be asked to review the revised paper.
3. Accept pending major revisions — This means that the study has merit, but there are still some substantial issues that need to be addressed. If the authors can correct those errors to the editor’s satisfaction, then it will be accepted for publication without further review.
4. Accepted pending minor revisions — This means that the paper is solid, but there are some minor issues that need to be dealt with before it will be published.
5. Accepted in its current form — This is a theoretical state in which a paper is accepted without any changes being required. I’m not convinced that it ever actually happens to real papers (it definitely hasn’t in my experience as either an author or reviewer).

After you make and justify your decision, the editor will look at your comments as well as the comments made by the other reviewer(s), then make the final decision about the fate of the paper. This system of having several reviewers is another strength of the peer-review process, because even if one reviewer does a crappy job and misses major flaws, the other reviewer(s) are there to pick up the slack.

Why do scientists serve as reviewers?
It is worth mentioning that you don’t get paid to be a reviewer. It is entirely a volunteer service, and it is quite time consuming. I, for example, had a large data set that I was hoping to analyze this afternoon, but instead I spent six hours reviewing a paper, and I never got around to the data (and this was a fairly short paper). So you may be wondering why on earth do scientists do it. Why don’t we always just reject requests to review papers? I obviously can’t speak for every scientist, but I can tell you my views and why I do it and take it seriously, and I know many other scientists who feel the same way. So I can’t state anything statistical, but I suspect that this would be a common response.

First, it is simply reciprocity. Every time that I submit a paper, I am imposing on other scientists to take time out of their busy schedules and review my research. As such, it is only fair that I then take time out of my schedule to review other scientists’ research. If everyone tried to pass the buck, then all of the reviews would be being conducted by an increasingly small and disgruntled group of scientists, and that would be bad for everyone. So acting as a reviewer is really just paying my dues as a member of the scientific community.

Second, as a reviewer, I get to play an active role in ensuring the quality of the research in my field, and that is a duty that I take very seriously. I obviously care greatly about my field and the advancement thereof, and I want the papers in my area of research to be of the highest quality possible. So, by serving as a reviewer, I get to block flawed research, promote high quality research, and make recommendations about how to improve research. This lets me extend my influence and impact on my field far beyond my own publications, and I see that as both a duty and a privilege.

To put this another way, I think that being a reviewer is an enormous responsibility. I know what it takes to do original research. I know the amount of work and effort that is involved, and I know what it is like to have all of your work torn to shreds by a reviewer. So, when I receive a paper, I always want it to be good and publishable, because I want those scientists to be rewarded for their extraordinary effort. At the same time though, when I agree to review a paper, I accept responsibility for preventing bad research from being published, and that responsibility motivates me to make absolutely sure that the research is solid before I give it my stamp of approval.

Summary
In short, reviewers are simply other scientists who have been matched with a paper based on their experience and expertise. It is their duty to carefully examine the paper, determine whether or not it was conducted correctly, and recommend if it should be published or rejected. Scientists do not get paid for this service, but it is an important task that researchers tend to take seriously.

Note: Journals vary widely with regards to anonymity. In some cases, reviewers don’t know who the authors are and authors don’t know who the reviewers are. In other cases, reviewers have the authors names, but authors don’t know who the reviewers are, and in yet other cases, both parties know the identities of the other party, and some reviews are even made public. There is an interesting debate about which system is best, and I had planned on going into it, but this post became longer than intended, so I will save that topic for a later post.

This flowchart summarizes the steps required to publish a peer-reviewed paper. See this post for details.