The peer-reviewed literature is where scientists publish their research, and it is the source for scientific information. As a result, I spend a lot of time on this blog talking about it. I have explained how the peer-review system works (also here). I have provided advice on how to evaluate studies and how not to evaluate studies. I have explained the hierarchy of evidence. I’ve explained P values and false positives. I’ve explained why many studies are unreliable and why it is important not to cherry-pick studies. I have provided worked examples of how to dissect studies (e.g., here, here and here), and I do my best to cite studies to back up all the claims I make on this blog. Nevertheless, it was recently pointed out to me that I have utterly failed to explain something important and fundamental: how and where to find peer-reviewed studies. So I am going to remedy that by providing a brief primer on how to go about finding articles on topics you are interested in, and how to get free copies of them.
Where to look
Let’s start with where to look. You can try simply doing a standard Google search, but odds are that you will get flooded with tons of blogs and websites, and it is a pretty inefficient way to find what you are after. A much better option is to use a database specifically tailored to peer-reviewed literature. There are two major ones that are freely available that I’m going to talk about: Google Scholar and PubMed (there are many others that are behind paywalls, but I am going to assume that most people reading this are not academics and don’t have access to those).
Let’s start with Google Scholar. First, I need to make it absolutely clear that this is not the same thing as a regular Google search. Literally anyone can get a blog, write an article, and it will show up in a Google search. In contrast, Google Scholar is tailored for academic articles, and you cannot manually add articles to it*. Instead, Scholar pulls from several academic databases (e.g., JSTORE) and employs bots to scour the internet for DOIs, abstracts, titles, etc., which it uses to identify peer-reviewed articles and add them to its repository. It’s not a perfect system; some articles get missed by the bots, and occasionally they pick up a non-peer-reviewed article that has the trappings of a peer-reviewed article (e.g., a non-reviewed report). Nevertheless, it is an extremely useful tool. It is a massive database that is very easy to use (more on that later) and even though I have access to more well-curated databases, Scholar is usually what I default to for quick searches.
Scholar also has the advantage of being a generalist database. In other words, it is not topic specific, and articles on medicine, zoology, climate change, GMOs, evolution, physics, chemistry, archeology, etc. can all be found within its digital walls. Sometimes though, it is useful to use a more focused database, and that is where PubMed comes in.
As its name suggests, PubMed is a repository for medical papers. It gets its papers both directly from journals and from author submissions. These submissions are checked to ensure that they are scientific papers. As a result, it tends to be more curated than Scholar, and you don’t get as many results that aren’t actually peer-reviewed papers.
There are lots of other databases out there, and if someone reading this has one that they love, feel free to mention it in the comments, but these are the two I’m going to focus on. I will quickly mention though that Mendeley’s database is often a good place to find more obscure articles. It is another generalist, but it allows author submissions, and on multiple occasions I have found papers there that didn’t show up elsewhere. So, while I wouldn’t use it as a primary database, it can be useful (you have to make an account, but it is free).
*If you are a research and have an account, you can manually add the bibliographic information for an article to Scholar, which may help Scholar to locate it if it hadn’t done so already, but you cannot simply upload an article.
How to search
Now let’s move on to how to actually find the papers you are after. For both PubMed and Scholar you can use them like a standard internet search and type in “vaccines autism,” for example, but that is going to return a ton of studies, so it is usually best to be as specific as possible. For example, if you specifically want to see results from randomized controlled trials, include that in your search terms.
Both databases also have very helpful advanced search settings. For PubMed, there is an “advanced” tab under the main search bar, and this returns a screen with a bunch of pretty self-explanatory options. For example, you can limit results to a specific author, specific journal, specific date range, specific word in the title, etc. Google Scholar is similar, but with fewer options (to get to it, click on the three lines on the left-hand side indicating a drop-down menu, then select “Advanced search”).
It can also be useful to either include or exclude specific words or phrases. PubMed and Scholar both let you include specific words or phrases by simply putting the word or phrase that you care about in quotes, at which point they will limit the searches to articles that contain that quote. This can be very useful if you are getting a lot of irrelevant results that include some parts of your search terms, but not exact phrases you are after. Conversely, there may be times when it is useful to eliminate a word. For example, if you are only interested in studies on humans, you might want to exclude a word like “mice” or “in vitro.” In PubMed, this has to be set in the advanced search option, but in Scholar, you can just ad a minus sign to the beginning of the word (or quoted phrase) that you want to exclude. This should be done cautiously, however, as you may inadvertently exclude relevant studies. For example, if you exclude the word “mice” you may accidentally exclude a study on humans that discussed rodent studies in the introduction or discussion, or even just cited a study with the word “mice” in the title. So, while this feature can be useful, it should be used carefully, and it is often better to put quotes around a word you care about, rather than eliminating a word. For example, you could put “human” in quotes, to force the search to give you more human trials. Having said that, quotes can bias search results and make it easier to cherry pick results (particularly when using long phrases). So, use these tools carefully.
Another really useful approach is to find one relevant study, then look both at the studies it cited and the studies cited by it. To my knowledge, PubMed does not have a “cited by” tab, but Scholar does under each article, thus allowing you to see which articles cited it. Also, both databases have a “related articles” or “similar articles” link under each article, which you can use to find other relevant research.
Personally, I find the citations within a paper to be the most useful. If you really want to understand a topic, then as you go through a paper, you should note the references to related studies that are worth reading. Then, you can use the literature cited section of the paper and Scholar or PubMed to look up those articles and read them. As you read them, you should find yet more articles. As you can well imagine, the number of articles you need to read balloons out pretty quickly, and it is why scientists have to spend so much time reading. This can, however, also provide a useful check for how well you have covered a topic. After reading a large number of papers, you should start to notice that the number of new, relevant papers being cited decreases. You should start to see a lot of familiar citations to papers you’ve already read. In other words, at first, the number of new citations to papers you need to read should be quite large after each paper you read, and that number will continue to grow until you start to get a good grasp on the literature. Then, it will slowly start to decrease as you read more and more of the relevant studies (i.e., it becomes harder and harder to find papers you haven’t read yet). This doesn’t mean that you are an expert and have read all relevant studies, of course, but it is a useful proxy for assessing your thoroughness.
How to get papers for free
Now comes the critical question, how do you actually get the paper without paying for it? In many cases, you can do so directly though Google Scholar or PubMed (Scholar is particularly good at finding and including links to free copies if they are available). Failing that, you have several options.
The first, is to do a standard Google search for the title of the paper. Sometimes, this brings up copies that Scholar missed. You can also check Research Gate and Mendelely, but usually Scholar picks those up. For papers on “physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics,” you can also try arXiv.org, which is run by Cornell and offers free, legal, open access to many papers in those fields.
The second option (which is often the best) is simply to contact the author and ask for a copy. In almost every case, they will be more than happy to send one to you. I want to pause here for a moment to make a brief point. Scientists do not get paid for their publications. Those fees to access papers go directly and entirely to the publishers. Scientists do not get one cent from them. So, don’t feel bad about asking a scientist for their research, because you aren’t costing them anything, and they will be thrilled to know that someone is interested in their work.
To actually get a hold of an author, email is usually the best option. At least one author always includes an email address on the paper. If that address doesn’t work, they may have switched universities, but a Google search will usually bring up their current position with their current email. Failing that, you can try to contact them via Research Gate, but at least for me personally, I find that to be an inefficient way for people to get in touch with me. I don’t get notifications from my Research Gate (because they were obnoxious) nor do I check it often, so when people ask me for my papers via Research Gate, it often takes me a long time to respond. In contrast, emailing me usually results in a response is a few hours. I think this is probably true for most academics, so I’d start with email.
One final note about emailing scientists, sometimes people feel like they are inconveniencing scientists by asking for a paper (particularly people who are not academics or students) so they write them a lengthy story about what they are interested in and why they want the paper. Don’t do that. You don’t need to justify your desire for knowledge and you are just wasting their time. All you need to say is, “Dear Dr X, could you please send me a copy of your paper titled, “Y.” Thank you very much, Your Name” or something to that effect. It doesn’t have to be quite that terse, but academics often get hundreds of emails a day, so keeping your message short is appreciated.
If all of that has failed, you can go old school and drive to a University, go to the periodical room of their library, and read the actual physical journal. It sounds antiquated, but periodical rooms are pretty neat, and some older papers haven’t been digitalized.
Finally, there’s always Sci-Hub. Sci-Hub has often been called the “Pirate’s Bay of academia,” and that is pretty apt. I don’t pretend to know all the details of how it works, but basically, the people who run it got access to a bunch of log-in credentials for journals and have used them to make those journals available to everyone. So, you can go to the site, drop a URL, title, or DOI for a paper, and 99% of the time, a free PDF will open. Is it legal? That is questionable. It has been sued several times, and it has had to switch domain names more than once. In my opinion, however, a more relevant question is, “is it ethical?” and as far as I’m concerned, the answer is, “yes.”
For obvious reasons, I cannot tell you that you should be using Sci-Hub, but I will tell you my personal view on the situation. I think that information should be available to anyone who wants it, and I think that it is wrong for data to be locked behind paywalls (particularly given how much research is publicly funded via tax dollars). I also think that the current publishing system is an unethical scam. Without going too much into the details, scientists have to pay “page charges” to publish in most journals, ostensibly to cover the cost to the journal for their editorial staff (see section later on predatory journals). Then, the journals sell the papers, and, as mentioned earlier, the scientists get no money back. Every single year, millions (probably billions) of dollars of grant money are paid by scientists for the privilege of being allowed to publish our work. Meanwhile, the journals rake in billions of dollars in profit from selling the articles, and in turn, stopping many people from having access to them.
To put all of that another way, the money flow goes like this:
- You pay the government via taxes
- The government gives a tiny portion of that to scientists to do research
- Scientists have to spend a good chunk of that money to publish their research
- Journals make billions of dollars in profit by charging you (the public) to access the results of the research that you already paid for via taxes.
It is an insane system that robs scientists of countless amounts of precious research funding that we could be using to actually test new questions, all while preventing many from reading the research that, in many cases, they funded with their taxes. Sadly, scientists are trapped in this system. We have to publish our research, and if we want good jobs, we have to publish in high-ranking journals, which means we have to publish in journals that charge us. Publishers know this and exploit it. Papers often cost $3,000 or more to publish. So, if you want to know my personal opinion about academic publishing companies and whether or not it is ethical to bypass their fees via Sci-Hub, I say screw them. It’s a stupid, unethical system that should be overthrown. Read up me hearties, it’s a pirate’s life for me (here endeth my rant).
Organizing your papers
This is somewhat tangential, but I think it is important. As you read papers, you should be taking notes and organizing your papers in a way that makes it easy for you to find the papers again in the future. There are several reference organizing programs specifically for this purpose, with Mendeley and Endnote being the two front runners. I started using Mendeley years ago (before it was bought by one of the massive publishers I just ranted about) and moving to a new system now would be too difficult to be worth it. Having said that, I’m really happy with Mendeley. It is free unless you need to store an ungodly number of pdfs, and it lets you organize papers in a lot of useful ways. You can create folders in the program to store different categories of papers, highlight the text, and write notes. Most usefully of all (IMO) you can “tag” papers with custom tags, then subset within a folder (or your whole collection) by those tags. For example, you could have a folder called “climate change” and tags such as: models, hurricanes, and heat waves. Then, if you need to look at a paper on hurricanes, for example, you can just subset by that tag. On top of that, you can then sort by title, author, journal, etc., or do a search for text in your notes or the papers themselves. Additionally, Mendeley backs up to the cloud, so you can access your files from any computer with an internet connection. It is very useful, and I highly recommend it (or EndNote or some other program) if you plan on reading lots of papers.
Predatory journals and reading critically
Finally, I need to make an important point about critically assessing the results you get from your searches. First, as mentioned earlier, databases like Scholar may return results other than peer-reviewed articles. So just because it showed up in the results, doesn’t automatically make it valid research.
Second, there are, unfortunately, a large number of “predatory journals.” These are, to a large extent, “pay-to-publish” journals that lack an actual peer-review system. I need to explain what I mean by this carefully, because this is not the same thing as the page charges I mentioned earlier. For real journals, you submit your paper for review with the acknowledgement that you are willing to pay the charges if the paper is accepted. Then, the paper goes out for review by other scientists, and if it is accepted you have to pay the charges. These journals care greatly about their reputation and at least try to keep shoddy research from being published (though see the next two paragraphs). In contrast, predatory journals are not real journals. They don’t actually do proper peer-review. You pay them just to publish any junk paper without critically assessing it. They are frauds and should not be treated as if they are real journals. Sometimes proper scientists get duped by them, but an awful lot of the papers in them are there because no legitimate journals would take them. Spotting predatory journals can be hard, but Beale’s List has a pretty good collection of journals and publishers to watch out for.
Beyond predatory journals, there is a wide range in quality for journals. Some journals aren’t technically predatory, but also aren’t really legitimate. To give a really extreme example, a while ago, a Bigfoot “researcher” was tired of actual journals rejecting their nonsense paper, so they started their own journal (de Novo) and published their “paper” there. I’m sure they reviewed their own paper with the highest of standards (sarcasm). That’s obviously the far end of the spectrum, but there are many journals out there that appear reputable, but actually have a strong bias towards fringe positions and tend to have pretty lax standards for review (looking at journals’ editorial boards, their scope, and their impact factor can be helpful for evaluate them).
Further, even really good journals sometimes publish bad papers. As I have said repeatedly on this blog, the peer-review system is good, but it is far from perfect, so you always have to read critically and look for a consensus of studies. The fact that a study found X doesn’t mean that X is automatically true. Scrutinize the study. Ask questions like, was this published in a reputable journal? Was the sample size large enough? Did they control confounding factors? Did they use appropriate statistics? Then, look at what other studies have found. Look at the entire body of literature rather than cherry-picking a handful of studies that agree with you. If there actually is good evidence that X is true, then you should find multiple large studies that used good methodologies and were published in reputable journals, and you should find few studies that disagree (or the dissenting studies should have small sample sizes, be published in questionable journals, etc.).
In short, databases like Google Scholar and PubMed are wonderful, powerful tools, but with great power comes great responsibility. It is extremely easy to do a quick search, find a paper that confirms your biases, then ignore all other studies and claim that you are right and everyone else is wrong, but it is your responsibility to avoid that temptation. It is your responsibility to be intellectually honest, read papers critically, and carefully examine the entire body of research, not just the studies that confirm your biases.
- Google Scholar and PubMed are great databases for scientific research
- Their advanced search options are very useful for wading through a mountain of literature
- Citations within papers are also very useful for finding other relevant research
- Papers that are behind paywalls can be obtained for free by either contacting authors (totally legal) or using Sci-Hub (questionably legal)
- Some journals are “predatory” and do not conduct a proper peer-review
- Journals and papers range widely in quality and you should avoid blindly believing the first study that agrees with you. Read critically and look at the entire body of literature.
Reblogged this on DAVEBOOK and commented:
I thought Beall’s work was continued/updated by others here:
Thanks for sharing. I was only aware of archival versions of the list. I’ll update the post
Thank you for this
Is there a place to comment on something other than this post? I couldn’t find it.
I recently read this article ( https://www.thedailybeast .com/why-your-carbon-footprint-is-meaningless ), which makes a lot of good points, but contains a huge fallacy, which I would call an “invalid induction fallacy” — maybe there is a better or more common name for it.
The author states that changing one light bulb provides a meaningless improvement, so does changing 2, etc. Then he extrapolates to say we should stop wasting time on this, because no number of changed light bulbs will help. Maybe true, but then he extends it to saying (the title of the article) that your “carbon footprint is meaningless”, implying that everyone’s carbon footprint is meaningless, by induction, one presumes. You could make the same argument about closing coal-fired utility plants. Closing 1 won’t make a dent. Closing 2 won’t make a dent. Therefore, by the invalid induction fallacy, closing all coal-burning utility plants won’t help. Wrong!
I thought that you or your readers may be interested in this, even though off-topic.
Keep up the good and interesting work!
On Mon, Sep 16, 2019 at 7:35 AM The Logic of Science wrote:
> Fallacy Man posted: “The peer-reviewed literature is where scientists > publish their research, and it is the source for scientific information. As > a result, I spend a lot of time on this blog talking about it. I have > explained how the peer-review system works (also here). I ha” >
“The author states that changing one light bulb provides a meaningless improvement, so does changing 2, etc. Then he extrapolates to say we should stop wasting time on this, because no number of changed light bulbs will help.”
This is wrong way of using Mathematical induction: https://en.wikipedia.org/wiki/Mathematical_induction
I will show it on clear example first.
Using the method proposed in article for numbers would let you prove, that there is no number bigger than (let say) 1 milion.
1 < 1 000 000
If number n < 1 000 000, then n+1 < 1 000 000. (this is wrong reasoning).
Now in your case. Let us assume that "meaningless improvement" is less than 50% improvement from all emission from all bulbs. You take any number here, athe the proof will bee similarly wrong. Higher than 50% is not meaningless.
Changing one bulb is meaningless. That's obvious.
If changing n bulbs is meaningles, than changing (n+1) bulbs is meaningless.
The step case here is wrong, because it does not matter, how you define the "meaningless level", you can always find such n, that n+1 is more than meaningless.
This is error analogous to the number example.
You can make up any such examples. One more example: if one atom has meaningless mass, then any number of atom has meaningless mass.
Do you think it is good science to claim a vaccine is safe and effective when no double-blind, placebo-controlled studies have been done?
First, your claim is blatantly false. There are, in fact, quite a few double-blind placebo-controlled trials on vaccines. If you actually had an interest in science and evidence, then you would have gotten on one of the data bases I talked about in this post, entered the phrase “double-blind placebo-controlled trial vaccine” and seen that there are many of these studies. https://scholar.google.com.au/scholar?hl=en&as_sdt=0%2C5&q=double-blind+placebo-controlled+trial+vaccine&btnG=
This entire post was literally about how to find the answer to questions like this, yet you persisted with making an inane argument instead of simply looking up evidence.
Second, even if you were correct that those studies don’t exist (which they do), scientists have other tools beyond randomized-controlled trials that can also provide accurate information. Indeed, in cases where an effect of interest is rare (as is the case of vaccine side-effects), cohort and case control studies can actually be more powerful than randomized-controlled trials. I talked about this in detail in the two posts below
So yes, countless scientific studies have confirmed that vaccines are safe. See, for example, this post specifically about the evidence concerning vaccines and autism https://thelogicofscience.com/2019/03/12/vaccines-and-autism-a-thorough-review-of-the-evidence-2019-update/
How do you reconcile the growing evidence that gut health (or the microbiome) is implicated in seemingly every cognitive disorder (not just autism)? If vaccines attack the immune system and kill-off bacteria in the gut microbiome (ground-zero for the immune system), and science is now acknowledging there is a gut-brain relationship that we are only beginning to understand (with the human microbiome only maturing around 5 years of age), then perhaps the research on vaccines and (not just) autism has been looking in the wrong place?
“If vaccines attack the immune system and kill-off bacteria in the gut microbiome”
That is not even remotely what vaccines do. Vaccines neither attack the immune system, nor do they kill bacteria. They work by training the immune system. You really should make sure that you understand the science before you make claims like this. I explained what vaccines actually do here https://thelogicofscience.com/2018/10/24/vaccines-dont-bypass-the-immune-system/
Further, if they caused disorders like autism via altering the gut microbiome, then the studies testing vaccines would have found this. If this hypothesis was true, the unvaccinated or less vaccinated kids should have lower autism rates than full vaccinated kids, but they don’t. I explained the extensive literature on autism and vaccines here https://thelogicofscience.com/2019/03/12/vaccines-and-autism-a-thorough-review-of-the-evidence-2019-update/
Finally, there is still a ton that we don’t know about how the microbiome affects health, and there is a substantial amount of evidence that autism is largely genetic. You need actual evidence before you can implicate vaccines
Your “thorough” review from March of this year missed on a couple of studies that wouldn’t have met your selection criteria but seem to be of critical importance to the topic nonetheless; they were written about in a May, 2019 article from The Economist (why The Economist? I don’t know) titled “More Evidence That Autism Is Linked to Gut Bacteria”. These studies are on ‘Microbiota Transfer Therapy’ (or MTT).. essentially, fecal matter from a healthy microbiome in pill-form (or “poo-in-a-pill” as I like to call it). The first study, out of Arizona State University — titled “Microbiota Transfer Therapy alters gut ecosystem and improves gastrointestinal and autism symptoms: an open-label study” on PubMed (2017) and “Long-term benefit of Microbiota Transfer Therapy on autism symptoms and gut microbiota” to more depth in an April, 2019 Nature.com article — only studied 18 children with autism and would’ve missed your radar due to sample size. The second study – titled “Human Gut Microbiota from Autism Spectrum Disorder Promote Behavioral Symptoms in Mice” on PubMed – replicated the results from the first study with a larger sample of rodents, but that study concluded a couple months after your relevant post and wouldn’t have met your criteria due to being an animal study (albeit one that used MTT from human donors to alter the behaviors of another species).
There’s another study by the Arizona State crew that has been put on “fast-track” status by the FDA due to its promising results; it is titled “Microbiota Transfer Therapy for Adults With Autism Spectrum Disorder (ASD) Who Have Gastrointestinal Disorders” on the ClinicalTrials.gov website and isn’t scheduled to be completed until the fall of 2021 (with 84 adult participants).
Those studies are about managing autism symptoms, not causing autism. Those are totally different. Further, again, vaccines don’t alter the gut microbiome. That’s not what they do. The fundamental premise of your entire argument is flawed. Finally, as explained in my post on autism and vaccines, hypotheses are only good until they have been tested. If a hypothesis like yours was true, then the tests on autism would have found effects of vaccines. They did not, therefore your hypothesis must be rejected.
Also.. you didn’t acknowledge a June, 2018 US District Court order in which the government acknowledged that they’ve been violating Federal Law as it relates to the 1986 “National Childhood Vaccine Injury Act” that is widely acknowledged as giving immunity to vaccine manufacturers. In that law, the Department of Health and Human Services is required to submit to Congress a biennial report regarding vaccines (42 U.S.C. subsection 300aa-27, “Mandate for safer childhood vaccines”). It took a lawsuit by a non-profit group co-founded by RFK, Jr to reveal that for over 30 years, there has been zero governmental oversight on vaccines.
Also… it should be acknowledged somewhere how much money the CDC makes as the chief market-maker for vaccines and how many “pro-vaccine” studies are underwritten by the CDC, shouldn’t it?
Also.. please read the article “L’affaire Wakefield: Shades of Dreyfus & BMJ’s Descent into Tabloid Science” on the website for the ‘Alliance for Human Research Protection’ (AHRP) before contributing to that smear campaign. Issues with the gut microbiome as it related to autism was all that Andrew Wakefield was trying to say with his paper (not study) documenting his analysis of 12 ASD diagnosed children.
At this point your going down the conspiracy rabbit hole, and I have no time for that nonsense. The majority of your claims here are simply false, but if you are honestly delusional enough to trust Wakefield, then there is no point in even bother to explain the problems with your claims. He was found guilty of fraud and unethically tests on children. He falsified his data and should never be trusted on any topic relating to science or health. You really need to objectively study this issue using good sources, and stop getting information from garbage sources. Your opening statement made it it clear that you have absolutely no understanding of even the basic concepts of how vaccines work. You need to understand the science before you can criticize it.
The simple reality is this: numerous large independent studies have found that vaccines are safe, no high quality studies have found that they are dangerous. End of story.
Given that none of these comments have anything to do with the topic of the post, I am ending these threads, as per the comment rules