Peer-reviewed literature: What does it take to publish a scientific paper?

Posted on March 4, 2015 by Fallacy Man

Modern science publishes research through a careful peer-review system, and it is the peer-reviewed literature that scientists rely on for their information. Nevertheless, the peer-review system is very poorly understood among the general public, and opponents of science tend to be very critical and dismissive of it. Indeed, anti-scientists’ default position is usually to blindly reject all peer-reviewed data (unless of course it is one of the handful of studies that seems to support their position). When I ask them why they are so distrustful of the peer-review system, they generally say something along the lines of, “its biased and you can only publish if you agree with the mainstream view.” The fascinating thing about claims like this is that they are nearly always made by people who have no personal experience with the scientific literature (i.e., people who have never written or reviewed a paper). Therefore, as someone who actually participates in the peer-review system (as an author, reviewer, and editor) I want to explain how the system actually works, what it takes to get published, and why it is a pretty good system.

The flowchart below summarizes everything in the post, but keep reading for more details (click the image and magnify to view it more easily).

flowchart diagram how to publish scientific peer-reviewed paper blog

This flowchart summarizes the steps required to publish a peer-reviewed paper and the steps required to publish a blog post. Take a careful look at this difference, then honestly tell me that you think that blogs are a better source of information about scientific topics.

Planning and conducting research
The first step of scientific inquiry is always observation. You make some observation about the universe around you, then you try to understand that observation, usually by making a testable hypothesis. Forming the hypothesis is relatively easy, figuring out exactly how to test it is, however, extremely difficult. Before you can start the experiment, you need to review all the literature on the topic so that you know what has already been found, and you need to design an experiment that follows ethical guidelines, has proper controls, will generate a large sample size, etc. All of this becomes very technical and, generally speaking, it is more than one person can do. So, most studies involve several scientists who collaborate together and share authorship on the final product. This is very important because the more people who are involved, the less likely it is that any one individual will bias the study. Also, different scientists have different specific areas of expertise (even within a single field), so bringing multiple scientists together gives you access to a large body of collective knowledge and experience, thus maximizing the odds that you will design a robust experiment.

Even with a group of various scientists collaborating together, however, it is still never a bad idea to consult with an outside expert. For example, all scientists have a working knowledge of statistics, but most of them are not statisticians in the truest sense. So it is very common for scientists to design their statistical analysis, then run it by an actual statistician just to make sure that there is nothing that they missed. Similarly, if the study involves a complex method that none of the authors have used before, it is a good idea to talk to someone who has used that method and make sure that you fully understand its intricacies.

Finally, once all of the collaborators agree on the design of the project, you can conduct the study and collect your data. This can take anywhere from a few days to a few years (usually at least a few months) depending on what the project is. Hopefully, the data will adhere to the structure that you anticipated, but that is often not the case. Without going into unnecessary detail, the type of statistical test you use depends on the type of data you are working with. So, for example, if you had planned on using a parametric test, but your data turn out to be strongly skewed, you may have to use a non-parametric test instead. Please realize, this is not a manipulation of the data. Scientists are not cherry-picking their statistics. Rather, there are mathematical limitations to how you can analyze data and each test has a specific set of requirements that have to be met before that test will give an accurate result. The point is that you may not be able to use the statistics you had originally planned on using. This often means that you will need to consult with a statistician again to determine the appropriate test given the data that you actually obtained.

Now, you can finally run your statistics and analyze your data. With any luck, you got good, reliable results that provide some useful and novel insight into the world around you, but sometimes your data don’t come out cleanly. Sometimes you failed to get a large enough sample size to accurately test your hypothesis, or your data may simply not add anything useful to our ever growing body of scientific knowledge. The point is that for various reasons many studies die at this stage before they even get submitted for peer-review. For example, I personally have two data sets sitting on my computer at the moment that I cannot in good conscience try to publish because, for various reasons, the experiments did not go as planned and I cannot trust my results.

Preparing and submitting a paper
If your data appear to give useful and reliable results, you can then write your paper. This tends to be a very time consuming process and usually involves many drafts being passed among your co-authors until eventually you all agree on a final product. At that point, however, it is not uncommon for you or some of your coauthors to want an outside opinion before submitting for formal review. Because you all worked on the project, you are all biased to think that the study is good (no matter how hard you try to avoid those biases). So, it is often a good idea to have a friend who works in a related field read the paper and give you some feedback. Depending on what he/she says, you may be ready to submit it, or you and your coauthors may go through several more revisions, or, in some cases, they may point out a fatal flaw that you missed and the paper dies there.

Assuming that your friend did not find a critical error, then once you have made the suggested revisions and you and your coauthors are content, you can finally submit it to a journal for formal peer-review. The first stage of the review process is generally a quick read by one of the editors. At this stage, they are trying to see if it is fairly well written, follows ethical guidelines, gives novel and potentially interesting results, appears to be a potentially valid study, and is the type of research that is published in that particular journal. During submission, most journals also require that you declare any conflicts of interest (e.g., you have a patent pending for the vaccine that you were testing).

If your paper passes all of the initial checks, it gets sent out to reviewers. If it doesn’t pass, it gets rejected and sent back to you. What you do at that point depends on why it was rejected. Often, the editor simply didn’t think it was a suitable topic for that journal. In which case, you can just submit it to another journal. Other times, there are mistakes that need to be fixed before submitting elsewhere (thus sending you back through the revision loops), and sometimes, there is a serious flaw (such as a very small sample size) that will prevent you from publishing anywhere.

If your paper goes out for review, it gets sent to 2-3 (occasionally 4) other scientists who are experts in the field that your paper address. For example, a few months ago, I was asked to review a paper on frogs’ diets because I am a herpetologist, and I have published several diet studies. Thus, I am familiar with the methodologies, literature, etc., and I am in a good position to tell whether or not a diet study on frogs was conducted properly. Importantly, reviewers generally have to be people who are not institutionally linked to the authors. In other words, your friend in the lab next to you cannot review your paper. It needs to be someone who isn’t from the same institution or company as you.

Reviewers look for several things. At the most basic level, they see if the paper is well written, easy to understand, etc. More importantly, however, they scrutinize the methods, statistics, conclusions, etc. to ensure that the study was done correctly, the proper statistics were used, the conclusions are valid, etc. They then send their comments and recommendations back to the editor. The editor then considers their recommendations, often consults with another editor, then sends you their decision.

At this stage, there are several possibilities. The best one is that it was accepted in its current state. In other words, they will publish your paper as is. More often, it gets accepted with either major or minor revisions. In other words, they think that there is merit to your study, but there are some concerns about certain parts of your paper (perhaps details of one of the methods you used). So, you and your coauthors make the revisions, then send it back to them. The editor(s) and sometimes the reviewers then look at your changes and decide whether or not they are acceptable. Usually, papers go through at this point, but sometimes additional changes are still required. Also, journals are increasingly shying away from saying “accepted with revisions” and instead are simply asking for revisions before saying anything about whether it will be accepted after the revisions are made (a situation which is extremely frustrating for researchers).

A third possibility is that your paper gets rejected with the option to resubmit. In this situation, they had very serious concerns about your paper (perhaps they think your statistics were completely inappropriate), but they still think that your paper has good potential. So, you and your coauthors get to make major changes to the paper and analyses. Once those changes are made, you can resubmit back to the same journal, at which point your paper goes back out for review. At this stage, your reviewers may or may not be the same reviewers that you had the first time.

The final possibility, is that your paper gets rejected without the option to resubmit. You are, however, given the reviewers’ comments. Sometimes they found critical flaws that truly render your paper unpublishable. Other times, however, there are serious flaws that you need to fix, but once those have been taken care of, you can submit it to a different journal, at which point you start this whole process over again.

This process is extremely complicated and time consuming, and most papers don’t make it through on their first round. Rejection rates vary among journals, but they are often 70% or higher (in other words, 70% of the papers that get submitted to that journal are rejected). Most importantly, by the time that a paper completes this process and actually gets published, many different scientists from different institutions and companies have looked at your work and given their input. As a result, the final product is usually of high quality, but bad papers do sometimes make it through. Fortunately, the peer-review process does not end with publication.

Dealing with bad papers
Scientists are very critical of each others’ work. In fact, we are trained to be critical of the peer-reviewed literature and to carefully examine a paper’s methodologies before accepting its conclusions. As a result, many papers sit quietly without ever being cited because other scientists are skeptical of their claims. Sometimes, however, a paper contains serious flaws, at which point, scientists can write to the editor of the journal explaining the problems, or they can write and publish a rebuttal paper. Depending on the problems that they point out, this may result in the journal retracting the paper. A very public example of this occurred last year when the journal Translational Neurodegeneration published a paper that supposedly found a link between autism and vaccines. The paper was rife with problems, and the journal quickly retracted it after multiple scientists expressed their concerns about its accuracy.

Other times, it may take years for the problems with a paper to surface. A famous example of this is Wakefield’s 1998 study that first proposed a link between autism and vaccines. This was an extraordinary claim, so scientists did what they always do with a claim like this: they tested it over and over again. The problem was that none of them could replicate Wakefield’s results. This culminated in a formal investigation which found serious ethical and methodological problems with the study, as well as a major financial conflict of interest that Wakefield failed to declare (a huge taboo in academic publishing). This resulted in the paper being retracted and Wakefield loosing the privilege of being allowed to practice medicine.

Important points
There are several important take home messages here. First, the idea that you cannot publish anything that is contrary to the mainstream view or that there are several “big wigs” who are pulling the strings and deciding what to publish and what not to publish is absurd. For any of these notions to work, all of the reviewers, editors, etc. that look at your paper would need to be corrupt and/or biased, but given the number of people involved and the fact that they are from multiple institutions, it is highly unlikely that all of them would be corrupt and/or biased. Further, if you get a biased editor who rejects your paper for absurd reasons, you can appeal to the editor and try to reason with them, or you can just submit to another journal. In other words, if you wanted to suppress any papers that were contrary to the mainstream view, you would need every editor in the world to agree not to publish any controversial papers. This is clearly absurd. Evidence that opposes the mainstream view can be published if you have good data. The reason that there are so few papers opposing evolution, climate change, vaccines etc. is not that there is some global conspiracy, but rather that there is no evidence to support those positions.

The notion that papers supporting a mainstream position are easy to publish is similarly absurd. Scientists are an extremely critical, ornery, argumentative bunch. We love nothing more than to prove each other wrong, and most papers get shredded during review. Publishing is hard, and no matter what your topic is, you are going to have to pass a careful review by objective scientists before you get published. Further, even if you pass the review system, your work will then be scrutinized by thousands of scientists from all over the world.

blogs vs scientific peer-reviewed literature meme

The problem with blogs is not that they are all faulty, but rather that they have no quality control mechanisms, so you have no reason to think that they are correct.

Finally, yes, the peer-review system is not perfect. No one is saying that it is an infallible system, but it is a good system, and it’s better than the alternative. To those of you who insist on trusting blogs rather than the peer-reviewed literature, think about the difference in what it takes to publish via each medium. Scientific papers are evaluated by numerous different experts before they are published, a huge number of papers never get published, and even after being published, papers can be retracted if flaws in them are found. In contrast, blogs are reviewed by no one, and at worst, someone might troll you. So which one do you actually think is more trustworthy: an admittedly imperfect review system that still manages to block a tremendous amount of bad research and ensure high quality in the majority of published papers, or a complete and total lack of review in which absolutely any piece of garbage can be published? The peer-review system may not be perfect, but it’s the best we have, and it’s still several orders of magnitude better than blogs (or Youtube videos).

Note: please read this post before commenting about Ioannidis’s work suggesting that most published papers are wrong.

Related posts

This entry was posted in Nature of Science and tagged , . Bookmark the permalink.