Handbook of Meta-Analysis

Christopher H. Schmid, Theo Stijnen, Ian White, Christopher H. Schmid, Theo Stijnen, Ian White

• Publisher: Chapman and Hall/CRC
• Year: 2020
• ISBN:
  9781351645713
• Edition: 1
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

Meta-analysis is the application of statistics to combine results from multiple studies and draw appropriate inferences. Its use and importance have exploded over the last 25 years as the need for a robust evidence base has become clear in many scientific areas, including medicine and health, social sciences, education, psychology, ecology, and economics.

Recent years have seen an explosion of methods for handling complexities in meta-analysis, including explained and unexplained heterogeneity between studies, publication bias, and sparse data. At the same time, meta-analysis has been extended beyond simple two-group comparisons of continuous and binary outcomes to comparing and ranking the outcomes from multiple groups, to complex observational studies, to assessing heterogeneity of effects, and to survival and multivariate outcomes. Many of these methods are statistically complex and are tailored to specific types of data.

Key features

• Rigorous coverage of the full range of current statistical methodology used in meta-analysis
• Comprehensive, coherent, and unified overview of the statistical foundations behind meta-analysis
• Detailed description of the primary methods for both univariate and multivariate data
• Computer code to reproduce examples in chapters
• Thorough review of the literature with thousands of references
• Applications to specific types of biomedical and social science data
• Supplementary website with code, data, sample chapters, and errata

This book is for a broad audience of graduate students, researchers, and practitioners interested in the theory and application of statistical methods for meta-analysis. It is written at the level of graduate courses in statistics, but will be of interest to and readable for quantitative scientists from a range of disciplines. The book can be used as a graduate level textbook, as a general reference for methods, or as an introduction to specialized topics using state-of-the art methods.

1

Introduction to Systematic Review and Meta-Analysis

Christopher H. Schmid, Ian R. White, and Theo Stijnen

CONTENTS

1.1 Introduction

1.2 Topic Preparation

1.3 Literature Search

1.4 Study Screening

1.5 Data Extraction

1.6 Critical Appraisal of Study and Assessment of Risk of Bias

1.7 Analysis

1.8 Reporting

1.9 Using a Systematic Review

1.10 Summary

References

1.1 Introduction

The growth of science depends on accumulating knowledge building on the past work of others. In health and medicine, such knowledge translates into developing treatments for diseases and determining the risks of exposures to harmful substances or environments. Other disciplines benefit from new research that finds better ways to teach students, more effective ways to rehabilitate criminals, and better ways to protect fragile environments. Because the effects of treatments and exposures often vary with the conditions under which they are evaluated, multiple studies are usually required to ascertain their true extent. As the pace of scientific development quickens and the amount of information in the literature continues to explode (for example, about 500,000 new articles are added to the National Library of Medicine’s PubMed database each year), scientists struggle to keep up with the latest research and recommended practices. It is impossible to read all the studies in even a specialized subfield and even more difficult to reconcile the often-conflicting messages that they present. Traditionally, practitioners relied on experts to summarize the literature and make recommendations in articles that became known as narrative reviews.

Over time, however, researchers began to investigate the accuracy of such review articles and found that the evidence often did not support the recommendations (Antman et al., 1992). They began to advocate a more scientific approach to such reviews that did not rely on one expert’s idiosyncratic review and subjective opinion. This approach required documented evidence to back claims and a systematic process carried out by a multidisciplinary team to ensure that all the evidence was reviewed.

This process is now called a systematic review, especially in the healthcare literature. Systematic reviews use a scientific approach that carefully searches for and reviews all evidence using accepted and pre-specified analytic techniques (Committee on Standards, 2011). A systematic review encompasses a structured search of the literature in order to combine information across studies using a defined protocol to answer a focused research question. The process seeks to find and use all available evidence, both published and unpublished, evaluate it carefully and summarize it objectively to reach defensible recommendations. The synthesis may be qualitative or quantitative, but the key feature is its adherence to a set of rules that enable it to be replicated. The widespread acceptance of systematic reviews has led to a revolution in the way practices are evaluated and practitioners get information on which interventions to apply. Table 1.1 outlines some of the fundamental differences between narrative reviews and systematic reviews.

TABLE 1.1
Key Differences between Narrative and Systematic Review

Narrative review	Systematic review
Broad overview of topic	Focus on well-formulated questions
Content experts	Multidisciplinary team
Not guided by a protocol	A priori defined protocol
No systematic literature search	Comprehensive, reproducible literature search
Unspecified selection of studies	Study selection by eligibility criteria
No critical appraisal of studies	Quality assessment of individual studies
Formal quantitative synthesis unlikely	Meta-analysis often performed when data available
Conclusions based on opinion	Conclusions follow analytic plan and protocol
Direction for future research rarely given	States gaps in current evidence

Systematic reviews are now common in many scientific areas. The modern systematic review originated in psychology in a 1976 paper by Gene Glass that quantitatively summarized all the studies evaluating the effectiveness of psychotherapy (Glass, 1976). Glass called the technique meta-analysis and the method quickly spread into diverse fields such as education, criminal justice, industrial organization, and economics (Shadish and Lecy, 2015). It also eventually reached the physical and life sciences, particularly policy-intensive areas like ecology (Järvinen, 1991; Gurevitch et al., 1992). It entered the medical literature in the 1980s with one of the earliest influential papers being a review of the effectiveness of beta blockers for patients suffering heart attacks (Yusuf et al., 1985) and soon grew very popular. But over time, especially in healthcare, the term meta-analysis came to refer primarily to the quantitative analysis of the data from a systematic review. In other words, systematic reviews without a quantitative analysis in health studies are not called a meta-analysis, although this distinction is not yet firmly established in other fields. We will maintain the distinct terms in this book, however, using meta-analysis to refer to the statistical analysis of the data collected in a systematic review. Before exploring the techniques available for meta-analysis in the following chapters, it will be useful first to discuss the parts of the systematic review process in this chapter. This will enable us to understand the sources of the data and how the nature of those sources affects the subsequent analysis of the data and interpretation of the results.

Systematic reviews generally involve six major components: topic preparation, literature search, study screening, data extraction, analysis, and preparation of a report (Figure 1.1). Each involves multiple steps and a well-conducted review should carefully attend to all of them (Wallace et al., 2013). The entire process is an extended one and a large, funded review may take over a year and cost hundreds of thousands of dollars. Fortunately, several organizations have written standards and manuals describing proper ways to carry out a review. Excellent references are the Institute of Medicine’s Standards for Systematic Reviews of Comparative Effectiveness Research (Committee on Standards, 2011), the Cochrane Collaboration’s Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2019) and Handbook for Diagnostic Test Accuracy Reviews (Cochrane, https://methods.cochrane.org/sdt/handbook-dta-reviews), and the Agency for Healthcare Research and Quality (AHRQ) Methods Guide for Effectiveness and Comparative Effectiveness Reviews (Agency for Healthcare Research and Quality, 2014). We briefly describe each component and reference additional sources for readers wanting more detail. Since the process is most fully developed and codified in health areas, we will discuss the process in that area. However, translating the ideas and techniques into any scientific field is straightforward.

FIGURE 1.1
Systematic review process.

1.2 Topic Preparation

The Institute of Medicine’s Standards for Systematic Review (Committee on Standards, 2011) lists four steps to take when preparing a topic: establishing a review team, consulting with stakeholders, formulating the review topic, and writing a review protocol.

The review team should have appropriate expertise to carry out all phases of the review. This includes not only statisticians and systematic review experts, but librarians, science writers, and a wide array of experts in various aspects of the subject matter (e.g., clinicians, nurses, social workers, epidemiologists).

Next, for both the scientific validity and the impact of the review, the research team must consult with and involve the review’s stakeholders, those individuals to whom the endeavor is most important and who will be the primary users of the review’s conclusions. Stakeholders may include patients, clinicians, caregivers, policy makers, insurance companies, product manufacturers, and regulators. Each of these groups of individuals will bring different perspectives to ensure that the review answers the most important questions. The use of patient-reported outcomes provides an excellent example of the change in focus brought about by involvement of all stakeholders. Many older studies and meta-analyses focused only on laboratory measurements or clinical outcomes but failed to answer questions related to patient quality of life. When treatments cannot reduce pain, improve sleep, or increase energy, patients may perceive them to be of little benefit even if they do improve biological processes. It is also important to address potential financial, professional, and intellectual conflicts of interest of stakeholders and team members in order to ensure an unbiased assessment (Committee on Standards, 2011).

Thoroughly framing the topic to be studied and constructing the right testable questions forms the foundation of a good systematic review. The foundation underlies all the later steps, especially analysis for which the proper approach depends on addressing the right question. Scope is often motivated by available resources (time, money, personnel), prior knowledge about the problem and evidence. Questions must carefully balance the tradeoff between breadth and depth. Very broadly defined questions may be criticized for not providing a precise answer to a question. Very narrowly focused questions have limited applicability and may be misleading if interpreted broadly; there may also be little or no evidence to answer them.

An analytic framework is often helpful when developing this formulation. An analytic framework is a graphical representation that presents the chain of logic that links the intervention to outcomes and helps define the key questions of interest, including their rationale (Anderson et al., 2011). The rationale should address both research and decision-making perspectives. Each link relating test, intervention, or outcome represents a potential key question. Stakeholders can provide important perspectives. Figure 1.2 provides an example from an AHRQ evidence report on the relationship between cardiovascular disease and omega-3 fatty acids (Balk et al., 2016).

FIGURE 1.2
Analytic framework for omega-3 fatty acid intake and cardiovascular disease (Balk et al., 2016).

For each question, it is important to identify the PICOS elements: Populations (participants and settings), Interventions (treatments and doses), Comparators (e.g., placebo, standard of care or an active comparator), Outcomes (scales and metrics), and Study designs (e.g., randomized and observational) to be included in the review. Reviews of studies of diagnostic test accuracy modify these components slightly to reflect a focus on tests, rather than treatments. Instead of interventions and comparators, they examine index tests and gold standards (see Chapter 19). Of course, some reviews may have non-comparative outcomes (e.g., prevalence of disease) and so would not have a comparator. Table 1.2 shows potential PICOS components for this study to answer the question posed in the omega-3 review “Are omega-3 fatty acids beneficial in reducing cardiovascular disease?”

TABLE 1.2
Potential PICOS Criteria for Addressing the Question: “Are Omega-3 Fatty Acids Beneficial in Reducing Cardiovascular Disease?”

As with primary studies, it is also important to construct a thorough prot...