Understanding abstracts: Does the study say what you think it says?

I spend a lot of time on this blog talking about scientific studies and how to analyze them, but there is a very important topic that, until now, I have only mentioned briefly: abstracts. Abstracts are intended to be useful tools to help scientists decide whether or not to read a given paper, but they are often misused and abused. Indeed, it is extremely common for people to base an argument entirely on an abstract. Even more disturbingly, I often see people citing conference abstracts as evidence. On numerous occasions, I have seen someone claim that, “A new study found X,” and when I follow the link, the study that they are referring to was presented at a professional conference, but has yet to be peer-reviewed.

scientific abstracts are not the same as papers meme funny peer-review

There is a huge difference between reading a study and reading the study’s abstract

To be clear, this is in no way a problem that is specific to those who routinely deny well established science. Rather, this is a problem that is rampant among staunch supporters of science as well. Therefore, in this post, I want to explain what abstracts are actually intended for, how you should use them, and how you shouldn’t use them. I also want to spend some time specifically on conference abstracts, because they are a special case that deserves extra attention.

The real purpose of abstracts
To begin with, I want to briefly explain what abstracts are actually intended for. Their purpose is simply to help researchers decide whether or not it is worth their time to locate and read the entire paper. This is a very useful function because critically reading scientific papers takes a long time. I’m a fairly fast reader, but if I want to really understand a paper, and go through all of the proper checks to make sure that it was done correctly, it will take me a minimum of one hour to read it (usually a lot more). So, given the millions of papers that have been published, a single scientist clearly can’t read all of them. Even within their chosen discipline, there will generally be far more papers than they could realistically read.

This is where the abstract comes in. A well written abstract will contain a very brief explanation of the background and importance of the study, a brief explanation of how the study was conducted, and the key results of the study. Scientists will then use that information to determine whether or not that paper is relevant for the question that they are working on. That’s it. That is the purpose of an abstract. It is to help scientists maximize their efficiency by focusing on papers that are relevant to them. It is not intended to be a through summary that you can read instead of reading the actual paper.

Think of abstracts like the descriptions on the back of a DVD case. You aren’t supposed to read the description instead of watching the movie. Rather, you are supposed to use the description to decide whether or not to watch the movie, and it would be silly to enter a debate about a movie for which you have only read the description. Similarly, you should use abstracts to determine which papers are worth reading, but you shouldn’t read the abstracts instead of reading the papers themselves.

Why you shouldn’t trust abstracts
The common tendency to read abstracts instead of the actual studies is extremely problematic because you can’t do any of the normal quality checks with just an abstract. Unfortunately, not all scientific studies are conducted properly. There is a lot of junk science out there. So, whenever you read a scientific study you need to very carefully look at the experimental design to make sure that it was set up correctly (e.g., was it properly randomized, did it use proper controls, etc.), you need to look at the statistics to make sure that they were done correctly, you need to look at the conclusions to make sure that they are justified by the results, etc. None of that is information that you can get from an abstract. So the abstract may look great, but the study itself might be fundamentally flawed. For example, last week I dissected a recent study that claimed that Splenda causes cancer in mice, and most of the things that I commented on were things that I could not have assessed from just the abstract. If all that I read was the abstract, I never would have seen things like the questionable statistics or inconsistent results.

This brings me to my next major point: abstracts often misrepresent studies. It is very easy to write a misleading abstract. Because abstracts are brief, they generally only highlight key results, but this means that authors have the opportunity (whether intentionally or unintentionally) to only present the results that support their conclusion while ignore results that challenge it. Similarly, abstracts generally include a general conclusion statement from the authors, but I have seen many, many cases where that conclusion statement really wasn’t supported by the data in the paper. To put this another way, abstracts often reflect the authors’ opinions about their study rather than the actual results of the study.

Finally, abstracts, like DVD cases, are designed to make you want to read the rest of the study (or watch the movie). They are scientists’ opportunity to advertise their research, and this sometimes results in abstracts that oversell the significance of the research or gloss over serious problems. To make another movie analogy, it would be silly to judge a movie by its trailer. Even so, you shouldn’t judge a paper by its abstract.

All of this simply means that you need to read the study itself before you either cite it as evidence or conclude that it is not good evidence. Reading the abstract is not enough to tell you what the study actually found or if you should even accept its results.

Understanding conference abstracts
Scientific conferences are lots of fun. I always look forward to them because I get to meet my fellow scientists (often including the great minds of my field), as well as presenting my own research and learning about other people’s research (btw, you can go to these meetings even if you aren’t a scientist, and if you are really interested in science, I highly recommend it). In a situation that is similar to the scientific literature, however, most meetings have more talks than one person can attend (there are usually several sessions running concurrently). So, much like the abstracts for papers, conference abstracts are simply intended to help people figure out which talks they want to see. Thus, they are very similar to the abstracts for papers, but there are a few key differences that make them even more dubious as sources of evidence.

First, conference abstracts generally aren’t peer-reviewed (there are a few exceptions, and some conferences even require a short paper, but that is not the norm). In other words, conference abstracts are usually accepted or rejected entirely based on whether or not they fit within the scope of the conference. This is extremely important because it means that other scientists have not meticulously examined the research to make sure that it is of a high quality. In other words, having an abstract accepted to a conference is in no way an affirmation that the research is solid or that the results are trustworthy. Even for the few conferences that review the abstracts, the review process is generally much less stringent than it is for publishing a paper in a journal.

Indeed, it is extremely common for the research in presentations/posters to never see the glorious light of publication, and if I am going to be totally honest, this was the case for one of my very first research projects as a lowly undergraduate. This is, in fact, commonplace for conference presentations. I have sat through some truly horrible talks, but the abstracts for those presentations generally looked great.

This brings me to me next major point. Remember earlier how I said that you need to critically analyze papers? Well you can’t do that with conference abstracts. This is the primary reason why they should not be used as evidence. When someone cites a conference abstract to back up their claim, all of that you have is that abstract. You have no way of checking the researchers’ methods, statistics, detailed results, etc. Which means that you have absolutely no way of knowing whether or not study was done correctly.

Further, talks and posters at conferences are often on projects that are currently in progress. As a result, they often include incomplete sample sizes and preliminary results. Thus, even when a talk does go on to be published in a reputable journal, the final data and conclusions in the paper often differ from the conclusions in the talk.

Similarly, another important function of conferences is actually to get feedback on your research prior to submitting it for peer-review. It is extremely common for the question and answer section of talks to include comments from other researchers such as, “Have you tried X?” or “You need to make sure that you account for Y.” These comments can be extremely helpful in refining your analyses and improving your research, but because they are always made during the conference, they do not get included in the abstract. In other words, the abstract is on the unrefined version of the data that has yet to receive casual feedback from your peers, not alone formal peer-review.

Advice on getting copies of papers
At this point, you may be thinking, “This is all well and good, but half the time the paper is behind a paywall, so all that I have is the abstract.” That is admittedly a real problem, but there are ways around it.

First, for conference abstracts you need to try to figure out whether or not the study has actually been published in a proper peer-reviewed journal. If it has, then you should read that study, and ignore the conference abstract. If it has not been published, the you should not be citing the abstract as evidence (sometimes a lack of publication is actually a sign that the paper was flawed).

As far as the actual papers, getting them can be challenging. The first thing to do is check both Google and Google Scholar. You can often get free copies of papers from them (note: most journals allow authors to self-archive their papers, so those pdfs often are legal). If you can’t get them there, you might be able to get them via your local library, especially if you have access to an academic library on a college campus. Even if you aren’t a student, many universities have a library that members of the public can visit, and those libraries usually contain a large periodical room where you can read and photocopy/scan articles from lots of different scientific journals.

Another option is simply to email the authors. You can usually get the lead author’s email address without actually paying for the paper (or if nothing else, Google them), and scientists will usually be more than happy to send you a free copy of their papers. So don’t be afraid to send them a brief, polite email asking for a copy of the paper. Preferably, include something like, “paper copy request” in the subject, and include the full title of the paper you want in your email. Don’t just say something like, “I want a copy of your paper on vaccines” because odds are that they have many papers on vaccines.

Conclusion
The take home message from this post should be self-evident, but I will state it anyway. Abstracts should not be used as evidence. They do not give you enough information to properly assess a study and they often reflect the authors’ biases rather than the actual results of the paper. Abstracts from conferences are even worse because they are often on preliminary data and generally have not undergone peer-review. Therefore, you need to read the actual study itself, not its abstract.

 Other posts on understanding scientific papers

Advertisements
This entry was posted in Nature of Science and tagged , , . Bookmark the permalink.

7 Responses to Understanding abstracts: Does the study say what you think it says?

  1. J.I. says:

    “scientists will usually be more than happy to send you a free copy” – unless they have something to hide and prefer people to rely on the abstract (or to use the paper only to pad their publications list). The issue of (not) being able to access scientific papers and how to address it is also discussed on this Wikipedia page: https://en.wikipedia.org/wiki/Sci-Hub

    Like

    • Fallacy Man says:

      I don’t really understand your argument. If the scientists had something to hide, why would they publish in the first place? Doing research and writing papers takes a tremendous amount of work. Why on earth would they do that then deliberately try to make their research inaccessible? Also, as far as padding your publication list, quality is often more important than quantity, and quality is generally judged by the number of citations. So a paper that is almost never cited because no one can get a hold of it doesn’t really help your publication record much.

      Finally, although members of the public have difficulty accessing papers, a very large portion of academicians have free access to them through their universities. I, for example, very rarely have to contact an author for a copy of their paper because I have access to a large university library which subscribes to tons of journals. Also, scientists themselves usually subscribe to the journals that are most relevant for their fields, and they receive copies of them every time that a new issue comes out. So even if you publish in a journal that uses a paywall, numerous other scientists who are in the same field as you are going to receive a copy of your research.

      So if you really want to prevent other scientist from reading your research, the only way to do that would be to publish in an extremely minor journal that uses a paywall and is so minor that very few scientists and academic libraries subscribe to it; however, having a paper that is almost never cited in a journal that no one has ever heard of actually looks bad and can hurt your publication list more than helping it. So again, why would anyone do that? Why would anyone knowingly publish a fraudulent paper, but try to make sure that no one ever reads it?

      Liked by 1 person

  2. J.I. says:

    Why would an academic enquire about who you are, then tell you he wouldn’t have a copy of his own paper (how likely is that?), but that he would tasks his assistant to locate it and send it to you (which then never happened). I assume because he wrote agenda-driven technophobic nonsense in that paper (the journal of which was targeted at his own peers in social sciences), and he was irritated when an outsider asked for a copy…

    Like

    • Fallacy Man says:

      Knowing academics, my guess would be that he was super disorganized and forgot to pass the request onto his assistant. I know quite a few professors whose offices are giant mountains of papers, and their computer files are similarly disorganized. So he probably had the paper somewhere, but that doesn’t mean that he knew where it was. I also know quite a few that need to be reminded frequently before even simple tasks get done. In many cases, it’s not their fault. Academics generally are incredibly busy people and it’s easy to forget things like sending an email. So I wouldn’t jump to the conclusion that the paper was fraudulent.

      Like

  3. J.I. says:

    As you wish. IMHO it’s very unusual to do an interrogation of somebody who is simply asking for a copy of your paper. (Who are you, do I know you, why do you want the paper, what do you want it for.) Especially if that person is so very incredibly busy, how comes they have time for grilling you virtually, but not for answering your actual question? And not finding a published paper, which is an important output of one’s work, is a bit strange. I wouldn’t trust somebody who is so disorganized to do a better job elsewhere in their work. If I get a request, it takes me two minutes to attach the paper and send it back, without even wondering who the persons are or why they need it.

    Like

  4. Marty says:

    Hey, can you (or any authors/books) explain where the increase in cancer and autoimmune disorders come from?

    Like

    • Fallacy Man says:

      The answers to that are complex. First, for cancer, it’s really hard to give a single explanation because there are so many different types of cancer each of which has its own causes. So some types of cancer have gone up, other have gone down, and most have stayed the same. Further, you have to account for the fact that the average life expectancy has been steadily rising, and that rise is actually a big part of the reason for cancer increases. In other words, on average, more people will develop cancer over the course of their life than they would have a few decades ago, but that is at least partly just because they are living longer now, which means that there are more opportunities to develop cancer. You can find a more detailed explanation and a very relevant study here
      http://www.skepticalraptor.com/skepticalraptorblog.php/cancer-prevalent-usa-another-myth-debunked/

      For autoimmune disorders, we aren’t completely sure, but it seems likely that it has to with the fact that in our modernized world, we do not get exposed to correct types of beneficial bacteria and other micro-organisms that we would have historically. This may also be why people who live in very rural areas often have fewer autoimmune disorders than those from very urbanized areas. I explained this in more detail under point 5 of this post
      https://thelogicofscience.com/2016/02/22/6-reasons-why-anti-vaccers-are-wrong-that-being-sick-is-good/

      Like

Comments are closed.