Skip to main content

A Beginner’s Guide to Systematic Review and Meta-Analysis

5 Literature Searches

Let’s make one thing clear before we start talking about the literature search:

If your literature search is flawed, your entire systematic review may not be publishable or otherwise useful.

So that means we need to get this step right. Well, if we’re honest we need to get every step right in the systematic review process, but if you mess up your literature search, it doesn’t matter what else is done properly, because the literature search underlies everything else.

5.1 Create a Search String

Arguably the most important part of your entire review is creating a good search string. This means you need to include all the commonly used keywords in the field. The absolute scariest feedback I can imagine from a reviewer is something like, “Your search was not broad enough. You did not consider any of these important keywords….”. That means you get to start over again. Sounds fun right? No… not even remotely fun.

So, what can we do to make sure we have a good search string? First, make sure you actually know the field you’re conducting a review in. Read some papers, gain some knowledge. You’ll likely learn the keywords that should be used through this process. Another important step is to look at existing systematic reviews and meta-analyses in the field. What keywords did they use?

Keep in mind you can use all sorts of strategies to search within databases, such as parentheses, quotation marks, AND, OR, and asterisks. How these types of strategies function may vary between databases but they generally have documentation available that tells you what types of operators their search supports.

5.1.1 How broad or narrow should the search be?

This totally depends on your study. Some fields are big. Some fields are small. That will, in part, dictate how many studies you are likely to locate. In my field, it is common to review 800-2000 abstracts, but some reviews are smaller and some are larger. It just depends what the specific topic is and how much literature is available.

5.1.1.1 Ways people try to reduce the number of studies.

Should we restrict our studies to only those published in journal articles? This is quite common, but personally I do not prefer this practice. This greatly limits what will be located, especially in certain fields. One may argue, “Well journal articles are the highest quality because they’re peer-reviewed”. Well, I’ve read quite a few peer-reviewed articles I would call low-quality, so that argument doesn’t hold much weight for me. In addition, many conferences, especially those with printed proceedings, are peer-reviewed. So, I do not prefer the approach of only reviewing journal articles.

Should we limit the studies by publication date? Maybe - but you should have a good reason, such as a major breakthrough in the field that changed the nature of the field. Otherwise, this is, in my opinion, a weak rationale for reducing the scope of your search.

Should I search the whole article or only titles and abstracts when I search the database(s)? Now this is a more important question. Personally, I prefer to search broader when possible. However, sometimes this is not practical because even though we have a great search string, we get tens of thousands of results. In these cases, where limiting your search string would needlessly exclude relevant studies, I do often support searching the abstracts of studies rather than their full-text within the database. However, in these cases it is important that the search string is intentionally designed for searching abstracts rather than full-text. For example, you may find you need to use slightly different search terms to really capture the relevant studies.

5.2 Pick Databases

You should pick the databases you search intentionally, and the databases you choose will be dictated by your field of study. How many should you search? Well, that also depends, so I can’t give you a good answer. Lately in my work in educational technology, we have been searching eight or more databases.

5.2.1 What about Google Scholar?

Google Scholar is a bit difficult to search in a way others can replicate because, at the time of writing (January, 2024), you cannot export all the citations at once. So, I typically do not formally search Google Scholar, but I will use various combinations of keywords to informally search Google Scholar, and I’ll add relevant studies to my database that were not located through other searches. I usually categorize this as something like, “Additional studies located through informal searches”.

5.3 Add Studies from Existing Reviews

It is good practice to add the studies included in relevant systematic reviews and meta-analyses in the field to your database.

5.4 Record Everything

As noted on the PRISMA page, you should record a lot of details about your literature search. Again, not having this information can be a significant problem during peer-review. Some items to record include:

  • The exact date of your literature search

  • The exact search terms you used in each database, including any limiters or filters

  • The exact databases you searched

  • The exact number of items located in each database

  • The total number of abstracts located for review

  • The total number of duplicates removed

  • The total number of studies excluded during title and abstract screening

  • The total number of studies reviewed during full-text screening

  • The total number of studies excluded during full-text screening and the reasons why, aligned with the inclusion/exclusion criteria

5.5 Exporting Studies from Databases

Nearly every database I’ve ever used has a way to mass export your located studies. I export my citations, with abstracts, as a .ris file. That is so I have a record of the search (I keep the .ris file) and so I can upload the citations into a citation management system like Zotero, or into software to aid with screening like ASReview, Rayyan, Covidence, or MetaReviewer. There is more information about these types of software in the study screening chapter.

5.6 Building Your Project’s Citation Database

Now that you have all your .ris files from the databases you searched, you are likely going to want to combine them into your own citation database for your project. You can do this by uploading your files into a citation management system like Zotero, or into software to aid with screening like ASReview, Rayyan, Covidence, or MetaReviewer.

5.6.1 Removing Duplicates

When you search for studies across multiple databases, you are going to have duplicate entries, by which I mean, there will be studies that come up in multiple databases. Removing duplicates can be incredibly time consuming if you are doing that yourself with no software assistance. Fortunately, there are now tools that can help, such as Zotero, ASReview, Rayyan, and Covidence.

I’ve used Zotero for years for duplicate removal but (at the time of writing) it can be tedious and time consuming because you have to individually approve each duplicate merge. I have not used duplicate removal in Rayyan yet. I am currently (January, 2024) collaborating on a review using Covidence to remove duplicates, and so far it seems helpful. I do not have an opinion as to if one software is “best” because what is “best” will vary by your use case. I’ll just say you should explore all of the options and see what your preference is, because there are certainly pros and cons to each.