1 Introduction to Systematic Reviews
This chapter serves as a surface level introduction to systematic reviews. It is not meant to replace a proper course of learning about what systematic reviews are and all the different types.
1.1 Types of Systematic Reviews
Let’s get this out of the way early: systematic review is both an umbrella term, containing many types of systematically-conducted reviews, as well as a specific methodology. I know that’s confusing. If you think it is confusing, you’re not alone. You know how I know? I know it’s confusing because I see a lot of scoping reviews published that are called systematic reviews. While scoping reviews are conducted systematically, they are more properly labeled as scoping reviews. Imagine my disappointment when I expect to be reading really well-synthesized findings about the efficacy or design of an intervention and instead I end up reading a high-level overview of the field.
So, what types of reviews are we talking about when we talk about systematic reviews in the educational sciences? We’re typically referring to scoping reviews (relatively new to the field (in January, 2024), but not new as a method), systematic reviews, and meta-analyses. Yes, meta-analyses are actually a type of systematic review. In fact, the methods are identical until the data extraction and data analysis. In systematic reviews we (usually) qualitatively analyze the data, while in meta-analysis we quantitatively analyze the data. There are a lot of different types of systematic reviews you should be aware of though. I recommend reading this classic piece by Grant and Booth (2009)6.
Let’s take a quick look at the different types of systematic reviews we typically see in education.
1.1.1 Scoping Reviews
Scoping reviews tend to be surface-level overviews of a field. They’re meant to evaluate the nature of a field and the amount of evidence available in the field. Essentially, they help us identify research questions, see gaps in the literature, and help us identify where more research synthesis is needed. Notably - study quality is not typically examined in scoping reviews.
Since scoping reviews provide a surface-level overview of the field, are they helpful? Well, personally I think it depends on the field. For new fields, certainly. For established fields, maybe. It depends on what you want to review. Perhaps you are looking to see if specific methods have been used, or if certain aspects of an intervention have been investigated. In these cases scoping reviews can be quite helpful. However, if you’re seeking to find out the efficacy of an intervention, or how to design an intervention based on the literature, it is the systematic review or meta-analysis that you’re likely looking for.
For more about scoping reviews, please review the PRISMA extension for scoping reviews (Tricco et al., 2018)7.
1.1.2 Systematic Reviews
Systematic reviews tend to answer more specific questions than scoping reviews. For example, you may look at the efficacy of an intervention or how to design an intervention to be most effective. Typically, systematic review, when used as the method rather than an umbrella term, refers to a qualitative analysis of the data from primary studies.
Importantly, if you are going to conduct a systematic review, you should review the PRISMA guidelines (Page et al., 2021)8.
1.1.3 Umbrella Reviews
Umbrella reviews are systematic reviews of reviews. In other words, we collect all of the reviews in the field and synthesize them. Pretty cool right? You can read more about them in Aromataris et al. (2015)9.
1.1.4 Meta-Analysis
Meta-analysis is an extension of systematic review methods, with the primary difference being that data are analyzed quantitatively rather than qualitatively. There are various types of meta-analysis, including conventional (covered in this book), three-level (covered in this book), as well as structural equation modeling meta-analysis (which you can read about here). There are other types as well (e.g., Bayesian) but they’re not common in education fields at the time of writing (January, 2024).
As I noted, in this book we don’t explore the statistical aspects of meta-analysis, it’s a how-to book. But you can read about the statistics in other sources, such as Doing Meta-Analysis in R: A Hands-On Guide.10 I felt Harrer et al. did a great job making meta-analysis accessible in that book, so if you are interested in an introduction to the statistical side of meta-analysis, it’s a great resource in my opinion.
Importantly, if you are going to conduct a meta-analysis, you should review the PRISMA guidelines (Page et al., 2021)11. The guidelines apply to meta-analysis as well as systematic review.
1.2 Non-Systematic Reviews
Non-systematic reviews are typically referred to as narrative reviews, critical reviews, overviews, etc. For the purposes of this book, we’re ignoring all of this literature because they are not systematic reviews. I’m not saying they’re not important. Rather, we’re just not talking about them in this book.