The peer-reviewed literature is where scientists publish their research, and it is the source for scientific information. As a result, I spend a lot of time on this blog talking about it. I have explained how the peer-review system works (also here). I have provided advice on how to evaluate studies and how not to evaluate studies. I have explained the hierarchy of evidence. I’ve explained P values and false positives. I’ve explained why many studies are unreliable and why it is important not to cherry-pick studies. I have provided worked examples of how to dissect studies (e.g., here, here and here), and I do my best to cite studies to back up all the claims I make on this blog. Nevertheless, it was recently pointed out to me that I have utterly failed to explain something important and fundamental: how and where to find peer-reviewed studies. So I am going to remedy that by providing a brief primer on how to go about finding articles on topics you are interested in, and how to get free copies of them.

Where to look

Let’s start with where to look. You can try simply doing a standard Google search, but odds are that you will get flooded with tons of blogs and websites, and it is a pretty inefficient way to find what you are after. A much better option is to use a database specifically tailored to peer-reviewed literature. There are two major ones that are freely available that I’m going to talk about: Google Scholar and PubMed (there are many others that are behind paywalls, but I am going to assume that most people reading this are not academics and don’t have access to those).

Let’s start with Google Scholar. First, I need to make it absolutely clear that this is not the same thing as a regular Google search. Literally anyone can get a blog, write an article, and it will show up in a Google search. In contrast, Google Scholar is tailored for academic articles, and you cannot manually add articles to it*. Instead, Scholar pulls from several academic databases (e.g., JSTORE) and employs bots to scour the internet for DOIs, abstracts, titles, etc., which it uses to identify peer-reviewed articles and add them to its repository. It’s not a perfect system; some articles get missed by the bots, and occasionally they pick up a non-peer-reviewed article that has the trappings of a peer-reviewed article (e.g., a non-reviewed report). Nevertheless, it is an extremely useful tool. It is a massive database that is very easy to use (more on that later) and even though I have access to more well-curated databases, Scholar is usually what I default to for quick searches.

Scholar also has the advantage of being a generalist database. In other words, it is not topic specific, and articles on medicine, zoology, climate change, GMOs, evolution, physics, chemistry, archeology, etc. can all be found within its digital walls. Sometimes though, it is useful to use a more focused database, and that is where PubMed comes in.

As its name suggests, PubMed is a repository for medical papers. It gets its papers both directly from journals and from author submissions. These submissions are checked to ensure that they are scientific papers. As a result, it tends to be more curated than Scholar, and you don’t get as many results that aren’t actually peer-reviewed papers.

There are lots of other databases out there, and if someone reading this has one that they love, feel free to mention it in the comments, but these are the two I’m going to focus on. I will quickly mention though that Mendeley’s database is often a good place to find more obscure articles. It is another generalist, but it allows author submissions, and on multiple occasions I have found papers there that didn’t show up elsewhere. So, while I wouldn’t use it as a primary database, it can be useful (you have to make an account, but it is free).

*If you are a research and have an account, you can manually add the bibliographic information for an article to Scholar, which may help Scholar to locate it if it hadn’t done so already, but you cannot simply upload an article.

How to search

Now let’s move on to how to actually find the papers you are after. For both PubMed and Scholar you can use them like a standard internet search and type in “vaccines autism,” for example, but that is going to return a ton of studies, so it is usually best to be as specific as possible. For example, if you specifically want to see results from randomized controlled trials, include that in your search terms.

Both databases also have very helpful advanced search settings. For PubMed, there is an “advanced” tab under the main search bar, and this returns a screen with a bunch of pretty self-explanatory options. For example, you can limit results to a specific author, specific journal, specific date range, specific word in the title, etc. Google Scholar is similar, but with fewer options (to get to it, click on the three lines on the left-hand side indicating a drop-down menu, then select “Advanced search”).

It can also be useful to either include or exclude specific words or phrases. PubMed and Scholar both let you include specific words or phrases by simply putting the word or phrase that you care about in quotes, at which point they will limit the searches to articles that contain that quote. This can be very useful if you are getting a lot of irrelevant results that include some parts of your search terms, but not exact phrases you are after. Conversely, there may be times when it is useful to eliminate a word. For example, if you are only interested in studies on humans, you might want to exclude a word like “mice” or “in vitro.” In PubMed, this has to be set in the advanced search option, but in Scholar, you can just ad a minus sign to the beginning of the word (or quoted phrase) that you want to exclude. This should be done cautiously, however, as you may inadvertently exclude relevant studies. For example, if you exclude the word “mice” you may accidentally exclude a study on humans that discussed rodent studies in the introduction or discussion, or even just cited a study with the word “mice” in the title. So, while this feature can be useful, it should be used carefully, and it is often better to put quotes around a word you care about, rather than eliminating a word. For example, you could put “human” in quotes, to force the search to give you more human trials. Having said that, quotes can bias search results and make it easier to cherry pick results (particularly when using long phrases). So, use these tools carefully.

Another really useful approach is to find one relevant study, then look both at the studies it cited and the studies cited by it. To my knowledge, PubMed does not have a “cited by” tab, but Scholar does under each article, thus allowing you to see which articles cited it. Also, both databases have a “related articles” or “similar articles” link under each article, which you can use to find other relevant research.

Personally, I find the citations within a paper to be the most useful. If you really want to understand a topic, then as you go through a paper, you should note the references to related studies that are worth reading. Then, you can use the literature cited section of the paper and Scholar or PubMed to look up those articles and read them. As you read them, you should find yet more articles. As you can well imagine, the number of articles you need to read balloons out pretty quickly, and it is why scientists have to spend so much time reading. This can, however, also provide a useful check for how well you have covered a topic. After reading a large number of papers, you should start to notice that the number of new, relevant papers being cited decreases. You should start to see a lot of familiar citations to papers you’ve already read. In other words, at first, the number of new citations to papers you need to read should be quite large after each paper you read, and that number will continue to grow until you start to get a good grasp on the literature. Then, it will slowly start to decrease as you read more and more of the relevant studies (i.e., it becomes harder and harder to find papers you haven’t read yet). This doesn’t mean that you are an expert and have read all relevant studies, of course, but it is a useful proxy for assessing your thoroughness.

How to get papers for free

Now comes the critical question, how do you actually get the paper without paying for it? In many cases, you can do so directly though Google Scholar or PubMed (Scholar is particularly good at finding and including links to free copies if they are available). Failing that, you have several options.

The first, is to do a standard Google search for the title of the paper. Sometimes, this brings up copies that Scholar missed. You can also check Research Gate and Mendelely, but usually Scholar picks those up. For papers on “physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics,” you can also try arXiv.org, which is run by Cornell and offers free, legal, open access to many papers in those fields.

The second option (which is often the best) is simply to contact the author and ask for a copy. In almost every case, they will be more than happy to send one to you. I want to pause here for a moment to make a brief point. Scientists do not get paid for their publications. Those fees to access papers go directly and entirely to the publishers. Scientists do not get one cent from them. So, don’t feel bad about asking a scientist for their research, because you aren’t costing them anything, and they will be thrilled to know that someone is interested in their work.

To actually get a hold of an author, email is usually the best option. At least one author always includes an email address on the paper. If that address doesn’t work, they may have switched universities, but a Google search will usually bring up their current position with their current email. Failing that, you can try to contact them via Research Gate, but at least for me personally, I find that to be an inefficient way for people to get in touch with me. I don’t get notifications from my Research Gate (because they were obnoxious) nor do I check it often, so when people ask me for my papers via Research Gate, it often takes me a long time to respond. In contrast, emailing me usually results in a response is a few hours. I think this is probably true for most academics, so I’d start with email.

One final note about emailing scientists, sometimes people feel like they are inconveniencing scientists by asking for a paper (particularly people who are not academics or students) so they write them a lengthy story about what they are interested in and why they want the paper. Don’t do that. You don’t need to justify your desire for knowledge and you are just wasting their time. All you need to say is, “Dear Dr X, could you please send me a copy of your paper titled, “Y.” Thank you very much, Your Name” or something to that effect. It doesn’t have to be quite that terse, but academics often get hundreds of emails a day, so keeping your message short is appreciated.

If all of that has failed, you can go old school and drive to a University, go to the periodical room of their library, and read the actual physical journal. It sounds antiquated, but periodical rooms are pretty neat, and some older papers haven’t been digitalized.

Finally, there’s always Sci-Hub. Sci-Hub has often been called the “Pirate’s Bay of academia,” and that is pretty apt. I don’t pretend to know all the details of how it works, but basically, the people who run it got access to a bunch of log-in credentials for journals and have used them to make those journals available to everyone. So, you can go to the site, drop a URL, title, or DOI for a paper, and 99% of the time, a free PDF will open. Is it legal? That is questionable. It has been sued several times, and it has had to switch domain names more than once. In my opinion, however, a more relevant question is, “is it ethical?” and as far as I’m concerned, the answer is, “yes.”

For obvious reasons, I cannot tell you that you should be using Sci-Hub, but I will tell you my personal view on the situation. I think that information should be available to anyone who wants it, and I think that it is wrong for data to be locked behind paywalls (particularly given how much research is publicly funded via tax dollars). I also think that the current publishing system is an unethical scam. Without going too much into the details, scientists have to pay “page charges” to publish in most journals, ostensibly to cover the cost to the journal for their editorial staff (see section later on predatory journals). Then, the journals sell the papers, and, as mentioned earlier, the scientists get no money back. Every single year, millions (probably billions) of dollars of grant money are paid by scientists for the privilege of being allowed to publish our work. Meanwhile, the journals rake in billions of dollars in profit from selling the articles, and in turn, stopping many people from having access to them.

To put all of that another way, the money flow goes like this:

1. You pay the government via taxes
2. The government gives a tiny portion of that to scientists to do research
3. Scientists have to spend a good chunk of that money to publish their research
4. Journals make billions of dollars in profit by charging you (the public) to access the results of the research that you already paid for via taxes.

It is an insane system that robs scientists of countless amounts of precious research funding that we could be using to actually test new questions, all while preventing many from reading the research that, in many cases, they funded with their taxes. Sadly, scientists are trapped in this system. We have to publish our research, and if we want good jobs, we have to publish in high-ranking journals, which means we have to publish in journals that charge us. Publishers know this and exploit it. Papers often cost $3,000 or more to publish. So, if you want to know my personal opinion about academic publishing companies and whether or not it is ethical to bypass their fees via Sci-Hub, I say screw them. It’s a stupid, unethical system that should be overthrown. Read up me hearties, it’s a pirate’s life for me (here endeth my rant).

Organizing your papers

This is somewhat tangential, but I think it is important. As you read papers, you should be taking notes and organizing your papers in a way that makes it easy for you to find the papers again in the future. There are several reference organizing programs specifically for this purpose, with Mendeley and Endnote being the two front runners. I started using Mendeley years ago (before it was bought by one of the massive publishers I just ranted about) and moving to a new system now would be too difficult to be worth it. Having said that, I’m really happy with Mendeley. It is free unless you need to store an ungodly number of pdfs, and it lets you organize papers in a lot of useful ways. You can create folders in the program to store different categories of papers, highlight the text, and write notes. Most usefully of all (IMO) you can “tag” papers with custom tags, then subset within a folder (or your whole collection) by those tags. For example, you could have a folder called “climate change” and tags such as: models, hurricanes, and heat waves. Then, if you need to look at a paper on hurricanes, for example, you can just subset by that tag. On top of that, you can then sort by title, author, journal, etc., or do a search for text in your notes or the papers themselves. Additionally, Mendeley backs up to the cloud, so you can access your files from any computer with an internet connection. It is very useful, and I highly recommend it (or EndNote or some other program) if you plan on reading lots of papers.

Predatory journals and reading critically

Finally, I need to make an important point about critically assessing the results you get from your searches. First, as mentioned earlier, databases like Scholar may return results other than peer-reviewed articles. So just because it showed up in the results, doesn’t automatically make it valid research.

Second, there are, unfortunately, a large number of “predatory journals.” These are, to a large extent, “pay-to-publish” journals that lack an actual peer-review system. I need to explain what I mean by this carefully, because this is not the same thing as the page charges I mentioned earlier. For real journals, you submit your paper for review with the acknowledgement that you are willing to pay the charges if the paper is accepted. Then, the paper goes out for review by other scientists, and if it is accepted you have to pay the charges. These journals care greatly about their reputation and at least try to keep shoddy research from being published (though see the next two paragraphs). In contrast, predatory journals are not real journals. They don’t actually do proper peer-review. You pay them just to publish any junk paper without critically assessing it. They are frauds and should not be treated as if they are real journals. Sometimes proper scientists get duped by them, but an awful lot of the papers in them are there because no legitimate journals would take them. Spotting predatory journals can be hard, but Beale’s List has a pretty good collection of journals and publishers to watch out for.

Beyond predatory journals, there is a wide range in quality for journals. Some journals aren’t technically predatory, but also aren’t really legitimate. To give a really extreme example, a while ago, a Bigfoot “researcher” was tired of actual journals rejecting their nonsense paper, so they started their own journal (de Novo) and published their “paper” there. I’m sure they reviewed their own paper with the highest of standards (sarcasm). That’s obviously the far end of the spectrum, but there are many journals out there that appear reputable, but actually have a strong bias towards fringe positions and tend to have pretty lax standards for review (looking at journals’ editorial boards, their scope, and their impact factor can be helpful for evaluate them).

Further, even really good journals sometimes publish bad papers. As I have said repeatedly on this blog, the peer-review system is good, but it is far from perfect, so you always have to read critically and look for a consensus of studies. The fact that a study found X doesn’t mean that X is automatically true. Scrutinize the study. Ask questions like, was this published in a reputable journal? Was the sample size large enough? Did they control confounding factors? Did they use appropriate statistics? Then, look at what other studies have found. Look at the entire body of literature rather than cherry-picking a handful of studies that agree with you. If there actually is good evidence that X is true, then you should find multiple large studies that used good methodologies and were published in reputable journals, and you should find few studies that disagree (or the dissenting studies should have small sample sizes, be published in questionable journals, etc.).

In short, databases like Google Scholar and PubMed are wonderful, powerful tools, but with great power comes great responsibility. It is extremely easy to do a quick search, find a paper that confirms your biases, then ignore all other studies and claim that you are right and everyone else is wrong, but it is your responsibility to avoid that temptation. It is your responsibility to be intellectually honest, read papers critically, and carefully examine the entire body of research, not just the studies that confirm your biases.

Key points

• Google Scholar and PubMed are great databases for scientific research
• Their advanced search options are very useful for wading through a mountain of literature
• Citations within papers are also very useful for finding other relevant research
• Papers that are behind paywalls can be obtained for free by either contacting authors (totally legal) or using Sci-Hub (questionably legal)
• Some journals are “predatory” and do not conduct a proper peer-review
• Journals and papers range widely in quality and you should avoid blindly believing the first study that agrees with you. Read critically and look at the entire body of literature.