Skip to Main Content

Systematic Review Tutorial

What's the Best Review for the Job?

Comparison of review types

Description & Purpose

Search Methods

Types of Articles included

Time to Completion

Number of Authors

Meta-Analysis

Technique that statistically combines the results of quantitative studies to provide a more precise effect of the results. Takes individual studies and assimilates them into a composite evidence base
  • Comprehensive systematic search and literature collection.
  • Search should be peer reviewed.

  • Article selection reported with PRISMA flow diagram.
  • Quantitative Research Studies. Priority placed on Randomized Controlled Trials.
  • No Reviews or Qualitative research.

12-18+ months

3+

Systematic Review

A systematic search for, appraisal and synthesis of research evidence, often adhering to guidelines on the conduct of a review. It seeks to draw together all available knowledge on a topic area and sum up best available research on a topic or area.
  • Comprehensive systematic search and literature collection. Search should be peer reviewed.
  • Article selection reported with PRISMA flow diagram.
  • Quantitative and Qualitative research studies.
  • Priority placed on Randomized Controlled trials. No reviews.

12-18+ months

3+

Umbrella Review

Specifically refers to review compiling evidence from multiple reviews into one accessible and usable document. Focuses on broad condition or problem for which there are competing interventions and highlights reviews that address these interventions and their results
  • Comprehensive systematic search and literature collection.
  • Peer review recommended.

  • Article selection reported with PRISMA flow diagram.

Reviews, but only those without theoretical studies or published opinion as primary evidence in them.

Typically an Umbrella review is a review of Systematic Reviews & Meta-Analyses.

~12 months

3+

Scoping Review

Aims to rapidly map the crucial points, arguments, and evidence from an area of research. Scoping reviews can serve to (1) examine the extent, range and nature of research activity, (2) determine the value of undertaking a full systematic review, (3) summarize and spread research findings, and/or (4) identify gaps in the research literature.
  • Comprehensive systematic search and literature collection.
  • Peer review recommended.

  • Article selection reported with PRISMA flow diagram.

It is generally acceptable to include all available literature regardless of study design that is germane to the research question(s).

3-6 months

Rapid Review
  • Assessment of what is already known about a policy or practice issue, by using modified systematic review methods to search and critically appraise existing research
  • The emphasis on Rapid indicates the time this review is completed in but reducing time can be accomplished by deviating from several systematic reviews such as limiting the scope of the question, limiting database searching, article inclusion, etc
  •  Search may not necessarily need to be comprehensive—various methods (such as hand searching, grey literature databases) can be eliminated in the interest of time.
  • Limitations of the search strategy or shortcuts taken must be acknowledged.

No recommended methodology for study selection.

< 4 months

1+

Literature Review

Generic term: published materials that provide examination of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness. May include research findings

Search should use at least one electronic database and identify their methods used.

No recommended methodology for study selection.

several weeks

1+

Other Types of Reviews

Meta-Analysis

A Systematic Review that uses a technique to statistically combine the results of quantitative studies to provide a more precise effect of the results. In order for a successful Meta-Analysis the characteristics beings studied--such as population, intervention, and comparison--must be sufficiently similar, and that the same outcome or measure be calculated the same way at the same time intervals.

Pros

  • Allows researchers to take individual studies and aggregate them into a composite evidence base.
  • Offers an opportunity for smaller studies--which may be statistically insignificant on their own--to make a research contribution.
  • Present a time-efficient summary of combined evidence for stakeholders.

Cons

  • Can suffer from 'apples and oranges' combinations of studies that are not sufficiently similar.
  • A Meta-analysis is only as good as the studies included.
  • Time consuming to complete, requires the use of many experts in search, screen, and statistical methods.

Helps Analyze

  • Numerical measures of effect assuming absence of heterogeneity

 

Deeks JJ, Higgins JPT, Altman DG (editors). "Chapter 10: Analysing data and undertaking meta-analyses." In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. www.training.cochrane.org/handbook.

Rapid Review

An assessment of what is already known about a policy or clinical practice issue that uses systematic review methodology to search and critically appraise evidence, but may modify stages to shorten the timescale to completion.

Pros

  • "Quick but not Dirty" - legitimate techniques to shorten timescale of completion (only some, not all may be used)
    • Carefully focusing question
    • Use broader, less sophisticated search strategies
    • restrict amount of grey literature
    • Extract only key variables and perform only the simplest quality appraisal
  • Stages are explicitly reported, and reports the effects of shortening methods on outcomes.
  • Less time-consuming than a systematic review.
  • Provides a thoroughly systematic search and screen process for reduced bias in that stage of the literature search.

Cons

  • Shortening the duration of the review process may introduce bias.
  • Rapid Reviews are particularly susceptible to providing precise answers to the wrong questions or providing inconclusive answers to ill-conceived questions their rush to publish.

Helps Analyze

  • Quantities of Literature available on a focused research question
  • Overall quality/direction of the effect of literature
  • current state of knowledge and priorities for future investigation and research
Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science : IS, 5, 56. https://doi.org/10.1186/1748-5908-5-56
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: The evolution of a rapid review approach. Systematic Reviews, 1, 10. https://doi.org/10.1186/2046-4053-1-10

Scoping Review

An assessment of the potential size and scope of available research literature. A Scoping Review aims to identify the nature and extent of research evidence.

Pros

  • Helps inform researchers, policymakers, and other stakeholders if a full Systematic Review is necessary
  • Less time-consuming than a systematic review.
  • Provides a thoroughly systematic search and screen process for reduced bias in that stage of the literature search.

Cons

  • limitations in the process of quality analysis.
  • Due to the limitations of Scoping reviews, they often do not provide substantial evidence to use in policymaking or in guideline creation.

Helps Analyze

  • Characteristics, quantity, and quality of available literature
  • Future attempts to specify viability for systematic review
  • current state of knowledge and priorities for future investigation and research
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32.
Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science : IS, 5, 69. https://doi.org/10.1186/1748-5908-5-69
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 143. https://doi.org/10.1186/s12874-018-0611-x

Umbrella Review

A review that focuses on a broad condition or problem for which there are two or more potential interventions and highlights reviews that address these potential interventions and their results.

Pros

  • Accessible review format allowing readers quick overviews of reviews relevant to a decision at hand
  • Response to the "lumping" versus "splitting" dilemma in reviews - i.e. it allows reviews to cover multiple interventions while covering an overview of the topic at hand.

Cons

  • Only logistically possible when narrower component reviews are available.
  • Only includes reviews, not primary research studies.

Helps Analyze

  • What is known
  • Recommendations for practice
  • What remains unknown
  • recommendations for future research
Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015). Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach. International Journal of Evidence-Based Healthcare, 13(3), 132–140. https://doi.org/10.1097/XEB.0000000000000055

Literature* Review

A work that provides an examination of recent or current published literature. While definitions are usually quite broad, most literature (narrative) reviews involve locating published literature, identifying materials for inclusion, and synthesizing the results of the literature as a whole, making some analysis of their contribution or value.

Pros

  • Successful Literature/Narrative Review:
    • Identify previous accomplishments in research literature
    • Allow for consolidating the collected work
    • Allows authors to build on previous work and expertise
    • Creates a great review tool for identifying omissions and gaps in the research literature.
  • Offers flexibility with searching, appraisal, and synthesis process.
  • Can be completed by one person.

Cons

  • Lacks an explicit intent to maximize scope or analyze data collected.
  • Conclusions are easily open to bias from the open potential to omit without much questioning w/o systematic methods in place.
  • Authors could potentially only select literature that confirms their hypothesis.

Organizational Frameworks

  • Can provide chronological analysis of a topic
  • A method-based review may provide more details on how researchers used certain methods
  • A theoretical or concept-based review may be appropriate for organizing a review around theories and concepts used by researchers

* - Sometimes literature reviews are called Narrative Reviews

Cronin P, Ryan F, & Coughlan M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), 38–43. https://doi.org/10.12968/bjon.2008.17.1.28059
Gasparyan, A. Y., Ayvazyan, L., Blackmore, H., & Kitas, G. D. (2011). Writing a narrative biomedical review: Considerations for authors, peer reviewers, and editors. Rheumatology International, 31(11), 1409. https://doi.org/10.1007/s00296-011-1999-3
 
Maggio, L. A., Sewell, J. L., & Artino, A. R. (2016). The Literature Review: A Foundation for High-Quality Medical Education Research. Journal of Graduate Medical Education, 8(3), 297–303. https://doi.org/10.4300/JGME-D-16-00175.1