2 Getting Started
You’ve decided to conduct a systematic review or meta-analysis. Cool!
Now where do we start?
2.1 Research Questions
First we need to set up our research questions. The research questions we have will guide the type of review we are conducting. To be honest, if you’ve found this book on systematic review and meta-analysis, I’m guessing you already know the basics of research questions. So, let’s look at some examples of the types of questions that may fit for different types of systematic reviews:
Broad questions fit for scoping reviews:
What are the publication trends in the field?
What is the nature of the evidence we see in the field?
What type of research exists in this research space?
Specific questions fit for systematic reviews:
How do we best design virtual humans?
What has been the impact of virtual humans on learning?
Specific questions fit for meta-analyses:
What has been the impact of virtual humans on learning?
What is the impact of virtual humans on motivational outcomes?
Wait - there is the same question for systematic review and meta-analysis!
Yes there is. That is because both types of reviews can be used to answer this question, they just do it differently. A systematic review will typically qualitatively analyze the results, talking about trends in the field. Whereas, a meta-analysis will quantitatively aggregate the results of the included studies.
2.2 Inclusion and Exclusion Criteria
Now that we have our research questions, we need to establish our inclusion and exclusion criteria. These define how we are selecting what studies are included in our analysis. They should be specific and clear to a reader. A reader should be able to apply these same criteria if they wanted to replicate your study. Here is an example:
Research Question: What is the influence of virtual humans on learning?
Methodological Approach: Three-level meta-analysis.
Inclusion criteria: To be included in this meta-analysis, studies must:
include a comparison of a virtual human to a non-virtual human condition.
measure a quantified learning outcome.
report enough data for effect size calculation.
be publicly available.
Exclusion criteria: Studies will be excluded if they:
used a physical robot or hologram as their virtual human.
were not conducted in authentic educational settings.
What does this accomplish?
These criteria should include studies of computer-generated virtual humans that are not physically embodied outside of a computerized interface, as well as only include studies that took place in authentic educational settings. All studies would also have enough data around learning outcomes to compute an effect size.