Who reviews scientific papers and how do reviews work?

I spent my afternoon reviewing a paper for a scientific journal and making a recommendation about whether or not the paper should be published. As a scientist, this is not an uncommon task for me, but it is a process that is largely foreign to the general public. Indeed, the peer-review system often seems to be a mystery to those who don’t participate in it, and, as a result, it is a frequent topic for this blog. For example, I have previously written about what it takes to publish a paper. However, I have not yet written a post specifically about what it is like to be a reviewer or even who reviewers are. So, I thought I would take this opportunity to explain the process from a reviewer’s point of view and offer you a window into the system that determines which papers get published.

 

Who are reviewers and how are they chosen?
In short, reviewers are scientists. The peer-review system is rather interesting because essentially everyone involved with it acts as both an author and a reviewer. In other words, reviewers are themselves scientists who also submit papers for review. This is a good system, because it means that scientific papers are being reviewed by other scientists, not by politicians, corporations, etc.

Nevertheless, reviewers obviously aren’t chosen at random from within the scientific community. Rather, journal editors choose them based on relevant experience and expertise. This can happen in several ways. Often, journals require you to recommend reviewers when you submit your paper. In other words, as part of the submission process, you have to nominate several people to serve as reviewers, as well as providing contact information and (often) a justification for why they would make suitable reviewers. As a general rule, you want to suggest people who have published similar papers (often papers that you cited in the paper you are submitting) and who aren’t in anyway affiliated with you (this insightful post provides more details about how to select reviewers). Ultimately though, the editor has the final say, and he/she will consider your suggestions and make the final call.

Another common method that journals use to select reviewers is simply to recruit reviewers during the paper submission process (or via society memberships). In other words, the submission form often includes an optional check-box that says something to the effect of “I am willing to be a reviewer for this journal.” Generally, this is also accompanied by a section where you list your areas of expertise. Thus, if you are submitting to a herpetology journal (i.e., reptiles and amphibians) and you check the box and list yourself as a turtle expert, then you will go on the list of potential reviewers for turtle papers.

A third mechanism that editors use is simply to contact authors who recently published similar research in the journal that they edit. Indeed, after publishing a paper in a particular journal, it is quite common for an editor of that journal to contact you about reviewing a different paper for that journal (this has happened to me several times).

Finally, editors may choose simply to send the paper to someone who they know does research in a similar area or who was recommended to them by someone else. For example, the editor may have recently read a similar paper and contact one of the authors of that paper. Indeed, the paper that I reviewed today was in a journal that I have never published in or signed up to be a reviewer for, and the paper did not cite my previous work, so I doubt that the authors recommended me. However, I have previously published extremely similar and relevant research, so I suspect that I was selected by the editor because of my previous papers on this topic.

Regardless of how they are chosen, the point is that reviewers are generally experts on the topic of the paper that they are being asked to review. In other words, editors select scientists who have the necessary skills, knowledge, and experience to assess the paper and determine whether or not it is worthy of publication. Usually, at least two reviewers are selected per paper, but it is not uncommon to have three reviewers, and some journals use four or more.

Who can’t be a reviewer?
Most publishing scientists also serve as reviewers, but not all scientists are eligible to be a reviewer for a given paper. First, to be a potential reviewer, you have to have expertise on the particular field that the paper is about. I, for example, am predominately a herpetologist, so I would never get asked to review a paper on physics. For that matter, I wouldn’t even get asked to review a paper on botany. I might, however, be asked to review papers on other areas of zoology, but only if the topic of the paper was closely aligned with topics that I study (e.g., one of my areas of research is population ecology, and although I study the population ecology of reptiles and amphibians, the same concepts, statistics, etc. apply to other taxa, so I have the necessary skills and knowledge to assess a population paper on birds, for example).

A second criteria is a lack of conflicts of interest. Exactly how that is defined is variable, but as a general rule, reviewers should not be in anyway associated with the paper in question, and they should not be institutionally linked to any of the authors. I could not, for example, serve as a reviewer for a paper that was written by another graduate student in my lab, because that would be a conflict of interest (i.e., even though I would try to be objective, I would be less likely to criticize the paper because I work with and like the authors).

Finally, in many cases, authors can recommend people who should not serve as reviewers. Editors are under no obligation to follow these recommendations, but if there is someone who you really don’t want reviewing your paper, you can make that case. For example, if you have a long standing rivalry with someone and, as a result, that person likely would not be objective, then you can argue that he/she should not be selected as a reviewer. To be clear though, there needs to be a legitimate reason why that person would not make a good reviewer, and you can’t include someone in that list simply because you are concerned that they will find problems with your paper.


What happens when you receive a request to review a paper?
Each journal is different, but this is generally how things play out. First, you receive an email from an editor asking if you would be willing to review a given paper. This email usually includes the abstract for the paper, or, at the very least, the topic that the paper is on. It may also include other information such as the number of pages, words, figures, and/or tables in the paper. Sometimes this email will have information about the reviewing standards of that journal such as whether or not the review is anonymous and the time frame in which you are expected to provide the review.

As a reviewer, you then look at the information that you have been given and decide whether or not the paper is on a topic that you are sufficiently knowledgeable about, whether or not you have any conflicts of interest, and whether or not you currently have the time to review it. You then respond to the editor to tell them whether or not you are willing to be a reviewer. If you reject the request, it is generally considered good form to suggest an alternative reviewer (which is another way that editors identify potential reviewers).

Once you accept your duty as a reviewer, you generally receive a copy of the paper as well as paperwork on confidentiality. Some journals also provide you with a review template that you are supposed to fill out, whereas others let you provide feedback in whatever way you see fit. Generally, with either system, the final review will consist of several summary statements about your views on the paper, a list/explanations of your major criticisms, and a list of comments on specific lines (these are either annotations to the original document, or a list with corresponding line numbers).

What do reviewers look for?
Imagine that you are a reviewer, and you have just been given a paper to read. What are you going to look for? There are actually lots of things that you should look for, but the key thing to keep in mind is that the job of a reviewer is to assess the quality of the research rather than acting as an editor. In other words, reviewers do look at things like grammar, readability, and the presentation of the data (e.g. good/appropriate figures and tables), but their main duty is to act as a filtering mechanism that blocks bad science from getting published and helps authors to improve potentially useful research.

As such, the most important thing that reviewers look at is the methodology. They check things like sample size, experimental design, the statistics that were used, how the statistical models were set up, etc. All of this is intended to identify poor methodology and ensure that the study was done correctly. Following that, reviewers will check to see if the results were reported and discussed properly. It is not at all uncommon for authors to jump to conclusions that are not supported by the data, and it is the reviewers’ job to reign them back in and make sure that all of the conclusions are merited.

Additionally, reviewers are tasked with making sure that the paper is well grounded in the scientific literature. All research inevitably builds on previous research. As such, papers are supposed to cite and discuss relevant papers, especially if similar studies reached different conclusions. Thus, reviewers check to make sure that this was done appropriately. This is another reason why it is important for reviewers to be experts on the given field. As experts, they know the literature, so if important papers are missing, they will be able to point them out. Indeed, it is exceptionally common for reviewers to suggest specific papers that the authors should have cited (I made several such suggestions on the paper that I reviewed today).

Finally, it is the reviewers’ job to actually be helpful to the authors. Many otherwise nice people become utter jerks when reviewing, but the idea is actually for reviewers to provide constructive criticism that will help the authors improve their paper. Thus, reviewers are asked not only to point out problems with the paper, but also to suggest ways to fix those problems (e.g., “your statistical method is inappropriate, and you should use method X instead”).

What do reviewers recommend?
You have now read the paper and made a list of comments, which means that it is time for you to make a decision. Does this paper deserve to be published? You generally have 4–5 options to choose from (some journals don’t use option 2, and some journals use slight variations of these).

  1. Reject without the option to resubmit — This means that the paper is seriously flawed and will rejected without further consideration.
  2. Reject with the option to resubmit — This means that the paper has serious flaws, but it also has merit if those flaws can be correct. The authors can then revise the manuscript based on your comments and resubmit it back to that journal. At that point it goes back out for review, and you will often (but not always) be asked to review the revised paper.
  3. Accept pending major revisions — This means that the study has merit, but there are still some substantial issues that need to be addressed. If the authors can correct those errors to the editor’s satisfaction, then it will be accepted for publication without further review.
  4. Accepted pending minor revisions — This means that the paper is solid, but there are some minor issues that need to be dealt with before it will be published.
  5. Accepted in its current form — This is a theoretical state in which a paper is accepted without any changes being required. I’m not convinced that it ever actually happens to real papers (it definitely hasn’t in my experience as either an author or reviewer).

After you make and justify your decision, the editor will look at your comments as well as the comments made by the other reviewer(s), then make the final decision about the fate of the paper. This system of having several reviewers is another strength of the peer-review process, because even if one reviewer does a crappy job and misses major flaws, the other reviewer(s) are there to pick up the slack.


Why do scientists serve as reviewers?
It is worth mentioning that you don’t get paid to be a reviewer. It is entirely a volunteer service, and it is quite time consuming. I, for example, had a large data set that I was hoping to analyze this afternoon, but instead I spent six hours reviewing a paper, and I never got around to the data (and this was a fairly short paper). So you may be wondering why on earth do scientists do it. Why don’t we always just reject requests to review papers? I obviously can’t speak for every scientist, but I can tell you my views and why I do it and take it seriously, and I know many other scientists who feel the same way. So I can’t state anything statistical, but I suspect that this would be a common response.

First, it is simply reciprocity. Every time that I submit a paper, I am imposing on other scientists to take time out of their busy schedules and review my research. As such, it is only fair that I then take time out of my schedule to review other scientists’ research. If everyone tried to pass the buck, then all of the reviews would be being conducted by an increasingly small and disgruntled group of scientists, and that would be bad for everyone. So acting as a reviewer is really just paying my dues as a member of the scientific community.

Second, as a reviewer, I get to play an active role in ensuring the quality of the research in my field, and that is a duty that I take very seriously. I obviously care greatly about my field and the advancement thereof, and I want the papers in my area of research to be of the highest quality possible. So, by serving as a reviewer, I get to block flawed research, promote high quality research, and make recommendations about how to improve research. This lets me extend my influence and impact on my field far beyond my own publications, and I see that as both a duty and a privilege.

To put this another way, I think that being a reviewer is an enormous responsibility. I know what it takes to do original research. I know the amount of work and effort that is involved, and I know what it is like to have all of your work torn to shreds by a reviewer. So, when I receive a paper, I always want it to be good and publishable, because I want those scientists to be rewarded for their extraordinary effort. At the same time though, when I agree to review a paper, I accept responsibility for preventing bad research from being published, and that responsibility motivates me to make absolutely sure that the research is solid before I give it my stamp of approval.

Summary
In short, reviewers are simply other scientists who have been matched with a paper based on their experience and expertise. It is their duty to carefully examine the paper, determine whether or not it was conducted correctly, and recommend if it should be published or rejected. Scientists do not get paid for this service, but it is an important task that researchers tend to take seriously.

Note: Journals vary widely with regards to anonymity. In some cases, reviewers don’t know who the authors are and authors don’t know who the reviewers are. In other cases, reviewers have the authors names, but authors don’t know who the reviewers are, and in yet other cases, both parties know the identities of the other party, and some reviews are even made public. There is an interesting debate about which system is best, and I had planned on going into it, but this post became longer than intended, so I will save that topic for a later post.

flowchart diagram how to publish scientific peer-reviewed paper blog

This flowchart summarizes the steps required to publish a peer-reviewed paper. See this post for details.

Advertisements
This entry was posted in Nature of Science and tagged . Bookmark the permalink.

10 Responses to Who reviews scientific papers and how do reviews work?

  1. phambichha says:

    Reblogged this on Phambichha's Blog and commented:
    Food for thoughts on peer-reviews publication

    Liked by 1 person

  2. So…in other words, the reviewers are like the jesters in medieval times, they show to the community that they are concerned about them because they do volunteer reviewing for free the scientific papers, But on the other hand they do not have any authority to change or publish the papers because it is the king who has the ring that rules them all!

    Like

    • Fallacy Man says:

      No, that is not at all how it works. The editor simply weighs the arguments made by the reviewers. They don’t veto them. In other words, when two reviewers disagree, then the editor will arbitrate, but when the reviewers are in agreement, the editor simply defers to them. In other words, if all of the reviewers said that a paper was terrible, the editor is not going to veto them and publish the paper anyway. That would completely defeat the point of having independent reviewers. Similarly, if the reviewers said that the paper should be published, then, unless the editor spots a glaring mistake that the reviewers missed, the paper will be published. In situations where the reviewers disagree, the editor will look at the arguments made by the reviewers and if one reviewer did actually find significant problems with the paper, then it will be rejected (this is usually the case with disagreements); whereas if one reviewer was simply being a jerk and making mountains out of mole hills, the paper will be published (pending suitable revisions).

      In short, the reviewers determine whether the paper is published or rejected. The editor simply conveys their decision and arbitrates when there is disagreement. So your jester/king analogy is utterly ridiculous.

      Like

      • You are not telling the true, you are mixing disinfo with trues! The reviewers don’t have any authority to publish a scientific paper because they are reviewers not editors! It’s like WordPress, what you are telling is that a subscriber or a contributer selected by the Author have the same capabilities to publish posts when in truth they don’t! I suspect that you are a nice piece of propaganda, like you nickname says: a Fallacy Man!

        Like

        • Fallacy Man says:

          You are, once again, not reading carefully and, as a result, making strawman arguments. Yes, the editors are the ones that actually say, “this paper will be published,” but they only do that based on the reviewers’ decisions. The reviewers are the ones who actually evaluate the papers and decide whether or not they are worthy of publishing. The editor simply carries out their decision and arbitrates when there is disagreement. That is how the system works. Imagine the following scenario: you have three people who are asked to make a decision about what to do. Person 1 and Person 2 get to vote on what should be done, and Person 3 merely carries out their decisions, unless there is a tie. In that case, Person 3 then gets to vote as well. In that situation, Person 1 and Person 2 obviously are making decisions and do have power over the outcome, and that is the same basic thing that happens in peer-review. Editors don’t just go around vetoing reviewers. That would completely defeat the purpose of having reviewers and it would force editors to carefully review every single paper that comes across their desk. So, the editor simply carries out the reviewers decision.

          You seem to be focusing on the technicality that the editor enters the final decision while ignoring the fact that the decision is reached on the basis of the reviewers’ recommendations (i.e., if all reviewers say that the paper should be published, it is published, and if all reviewers say that it should not be published, then it is not published).

          PS I find it fascinating that you feel justified in attempting to lecture me on a system that you clearly have never participated in and have no first hand knowledge of. Where are you getting your information from? What is your basis for saying that I am wrong about how the system works? The fact that you would compare the system to WordPress clearly demonstrates just how little you understand about it, because it is nothing like WordPress.

          Like

          • “but they only do that based on the reviewers’ decisions”
            You are using the bandwagon and the black-or-white fallacies because you presented two alternative states (publish or not publish depending of the reviewers decisionss) as the only possibilities, when in fact more possibilities exist! And the bandwagon fallacy because you appealed to popularity or the fact that many people do something as an attempted form of validation.

            Editors know very well how to play the game, they can easily hire the reviewers they want until the revisions are pleased to what they want it to be published! In addition, Editors may very well completely ignore the reviews because as I told you, it is they who have the final decision of publication. Revisions are a form of slave labor in scientific community, disguised as honored volunteer work, because they are free for Editors to have the revisions made without spending a penny and especially without having any authority over them to prevent them from publishing! That is why volunteering in this situation does not work properly because it lacks ethics and independence. The only way of revisions work with full transparency, independence and professionalism is being paid and carried out by independent and decentralized entities completly unrelated to publishers/editors!

            Like

            • Fallacy Man says:

              What!? None of the makes any sense. First, if you read the original post, I made it very clear that there are sub-categories within the overarching two options of reject or accept, but stating all of the subcategories every single time in the comments is a silly waste of time.

              Second, bandwagon fallacies happen when you say, “many people are doing X, therefore X is right/good.” That is not even remotely what I said. Rather, this is just how the peer-review system works. Editors reject or accept papers based on the reviewers decisions. That’s not a judgement call, that is just how the system works.

              The rest of your comment derails into nonsensical and paranoid crap that once again just illustrates that you don’t have the foggiest clue how the system works. You are making things up with no evidence or basis for them.

              For example, you said “they can easily hire the reviewers they want until the revisions are pleased to what they want it to be published!” Except as explained in the post, reviewers don’t get paid. They are scientists who are independent of the paper being reviewed.

              Similarly, you said, “because as I told you, it is they who have the final decision of publication.” Again, you are claiming that editors go around vetoing reviewers, and that simply isn’t how it works. You can believe that if you want, but it’s not true (again, you don’t actually participate in this system so I’m not sure where you are getting these ideas from). You are aware that authors get to see all of the reviews right? So, let’s imagine for a second that I get a paper back and all of the reviewers said that it was a good paper that should be published, but the editor rejected it. First I would contact the editor to see what was going on. If he/she refused to give an adequate explanation, I can then appeal to the broader board of editors. Editors do have accountability believe it or not. Further, even if all of that fails because all of the editors at that journal are corrupt and conspiring together for some unknown and sinister purpose, then I will never submit there again, and I will tell all of my colleges to do likewise. Similarly, everyone else who has that experience will do likewise, and the journal will very quickly find itself in hot water.

              “The only way of revisions work with full transparency, independence and professionalism is being paid and carried out by independent and decentralized entities completly unrelated to publishers/editors!” That is exactly how it works! For goodness sake that is why we use independent reviewers! The system is deliberately set up for transparent, independent, and professional reviews of papers. Again, the editors simply carry out the verdict of the reviewers. Further, the editors themselves are scientists. You know that right? They are experts in their fields, so why exactly would they corrupt their own fields of research? Additionally, most of the best journals operate out of professional scientific societies and are run by a board of independent scientists. Although there are some for profit journals, a large portion are independent non-profits.

              Think of the situation like a judge and a jury. The judge is ultimately in charge and he orchestrates things, but the jury decides whether the accused is guilty or not guilty. The judge doesn’t get to veto the jury’s decision (note: that analogy is admittedly imperfect, because judges don’t get to step in when there is a split vote among the jury members, whereas editors do get to make the final call when reviewers disagree, but even there, they do so by carefully considering the arguments made by the reviewers).

              At this point, we have once again carried this conversation as far as it is worth having (probably further). So I’m going to end it. Otherwise, you will inevitably respond by continuing to insist that editors have totally autonomy and do whatever they want. I will then respond by telling you that it simply doesn’t work that way and you have no evidence for that. Rather you are making baseless conjectures about a system that you clearly have never participated in and know nothing about, and we will go back and forth like this for eternity. I just don’t have the time for that. If you want to continue to hold your ridiculous and fundamentally flawed views, whatever, I don’t care. You’re dead wrong, but there is obviously nothing that I can say that will convince you otherwise.

              Like

  3. Missylulu says:

    Thanks so much for this thorough explanation of the peer review process. I have a Masters of Library Science, and I wish someone had given me such a thorough explanation of this process when I was in graduate school. I now work as a librarian in an academic library, and am studying to become a chemical engineer, and all of this information is tremendously helpful. I’m embarrassed I wasn’t more familiar with this process before, but I have lots of students ask me why they are required to use peer-reviewed articles, and this is a great explanation of the rigor of the peer-review process vs. simply publishing any content to the internet. The chart is amazing!

    Do you have any other articles on here about how to determine the quality of a scholarly journal article? I struggle myself finding quality articles for people in fields that aren’t related to my own, which is unfortunate because students expect librarians to really know everything about finding articles. I make sure to never recommend an article I am unsure about, and in unrelated fields I tend to recommend resources over specific articles with the assumption that the person looking has a better background in the field than I do. However, knowing more generally how to determine an article’s quality would be so helpful to me and make me more confident in my recommendations. Peer-reviewed articles in library school are sort of treated as a panacea source, and while they are probably a good bet due to the rigorous review process (as opposed to blogs or popular periodicals), I imagine that they are not all perfect, or at least are not necessarily perfect to use in particular cases. Students tend to also believe they can just throw a dart at a wall covered in peer-reviewed sources and, as long as they are vaguely on the same topic, will be fine to use in their work.

    Thanks so much for the wonderful, informative post!

    Like

Comments are closed.