Statistical Approaches to Causal Analysis

Matthew McBee, Author

• Publisher: SAGE Publications Ltd
• Year: 2022
• Print ISBN: 9781526424730
• eBook ISBN: 9781529711110
• Edition: 1
• Topic: Social Sciences
• Subtopic: Research & Methodology in Psychology
• Index: Social Sciences

This book provides an up-to-date and accessible introduction to causal inference in quantitative research. Featuring worked example datasets throughout, it clearly outlines the steps involved in carrying out various types of statistical causal analysis. In turn, helping you apply these methods to your own research.

It contains guidance on:

• Selecting the most appropriate conditioning method for your data.
• Applying the Rubin's Causal Model to your analysis, a mathematical framework for understanding and ensuring accurate causation inferences.
• Utilising various techniques and designs, such as propensity scores, instrumental variables analysis, and regression discontinuity designs, to better synthesise and analyse different types of data.

Part of The SAGE Quantitative Research Kit, this book will give you the know-how and confidence needed to succeed on your quantitative research journey.

1 Introduction

Chapter Overview

• Internal validity 2
• External validity 2
• Threats to validity 3
• Randomisation 6
• Non-experimental research 7
• A pragmatic definition of causation 9
• Prediction versus explanation 10
• Causal inference requires external information 11
• Estimation versus hypothesis testing 13
• Prerequisites 13
• Notation 14
• The R statistical programming environment 14
• Structure of this book 17

In my experience, students of the social sciences receive much more instruction regarding what causation is not than what it is. When asked ‘what is causation?’ the first answer given by the graduate students in my linear modelling course was, ‘Correlation does not imply causation!’ This is correct as a general principle, of course. But causation nearly always implies correlation. There are conditions under which a correlation, slope coefficient or group mean difference measures the magnitude or strength of the causal relationship between variables. But there are many others in which these summary statistics imply nothing of the sort. In some conditions, these statistical values become fine instruments allowing us to peer into the heart of phenomena; in others, they are nearly as useless as the astrological signs or creases on a palm. One of the great challenges of empirical science is the proper understanding of when observed statistics can be imbued with meaning and when they should be ignored. Correlation does not imply causation, except for when it does.

In psychology and education, the fields I was trained in, the dominant framework for understanding causal inference and its connection to research design comes primarily from Campbell and Stanley’s (1966) classic, Experimental and Non-Experimental Designs for Research, and the later work that it inspired, such as Shadish et al. (2002). Campbell and Stanley’s short and readable monograph is freely available online¹ and is well worth careful study, as it introduced a number of concepts that are now central to social research.

¹ www.sfu.ca/~palys/Campbell&Stanley-1959-Exptl&QuasiExptlDesignsForResearch.pdf

Internal validity

Campbell and Stanley (1966, p. 5) described internal validity as ‘the basic minimum without which any experiment is uninterpretable: Did in fact the experimental treatments make a difference in this specific experimental instance?’ They went on to describe it as the sine qua non of research. To conclude that a study exhibits internal validity is equivalent to determining that it supports causal inference. As this book is primarily interested in the question when and how researchers can draw causal conclusions from data, one could argue that this book is primarily concerned with internal validity.

External validity

External validity describes the degree to which a causal effect is generalisable to other populations, contexts or settings. In social research, controlled laboratory experiments are often portrayed as high in internal validity but low in external validity, because it is unclear to what degree behaviours exhibited in the artificial environment of a lab mirror those exhibited in real-life contexts. On the other hand, field experiments may have high external validity but suffer from low internal validity due to the lack of control researchers have over extraneous factors in such settings (Shaughnessy et al., 2015). Thus, internal validity and external validity are in tension. Studies relying on animal models may have the highest baseline internal validity imaginable due to the researcher’s ability to control the environment, diet, exercise, social exposure and even the genetics of the animals. However, the external validity of such studies is questionable due to the obvious differences between animal and human physiology and behaviour.

External validity concerns appear in this book in several places. Propensity score analysis (which is introduced in Chapter 5) allows researchers to estimate causal effects that apply to different subgroups of the population. But the most important concerns about external validity arise in the context of instrumental variables analysis (Chapter 6) and regression discontinuity designs (Chapter 7). These techniques estimate causal effects for very specific subgroups. The exact identity of those groups may be unclear, or they may not be exactly the population to which the researcher hopes to generalise. These subpopulation-specific causal effects are called local average treatment effects. The general inability of instrumental variables and regression discontinuity design techniques to estimate global causal effects is a limitation of those methods.

Threats to validity

A threat to validity is an alternative, non-causal explanation for apparent differences between group means or other suggestive quantitative indications of potential causal link between variables. Campbell and Stanley (1966) described eight threats to internal validity, which are displayed in Table 1.1. They also described four threats to external validity.

Table 1.1

From Campbell and Stanley’s perspective, the quality or rigour of a study is proportional to the number of these threats that can be logically eliminated or rendered improbable by virtue of its design. Causation is provisionally accepted after counter explanations have been considered and rejected. Shadish (2010) described Campbell and Stanley’s (1966) perspective as rooted in the Popperian falsificationist philosophy of science. In Popper’s philosophy, an experimental result can only falsify or fail to falsify a theoretical proposition. It is not possible for an empirical finding to verify that a theory is correct, or even to directly support a theory per se. It can only show that the theory is incorrect or incomplete in some way when it fails to account for an empirical observation (Dienes, 2008). The best experimental outcome that a theorist can hope for is survival. As Campbell and Stanley (1966) wrote,

In a very fundamental sense, experimental results never ‘confirm’ or ‘prove’ a theory – rather, the successful theory is tested and escapes being disconfirmed. The word ‘prove’, by being frequently employed to designate deductive validity, has acquired in our generation a connotation inappropriate both to its older uses and to its application to inductive procedures such as experimentation. The results of an experiment ‘probe’ but do not ‘prove’ a theory. An adequate hypothesis is one that has repeatedly survived such probing – but it may always be displaced by a new probe. (p. 35)

In a similar fashion, Campbell and Stanley do not affirmatively define the conditions under which a true causal relationship can be concluded. Instead, they described conditions, circumstances and events that would create the illusion of causation when it does not exist. Consider their description of the ‘One-Group Pretest–Post-test Design’, in which subjects are measured on some variable, exposed to a ‘treatment’ and then measured again. Suppose that the outcome variable is the severity of depression symptoms and the treatment is cognitive behavioural therapy. Further, suppose that the mean depression symptoms at post-test are substantially (and statistically significantly) reduced from their mean value at pretest. Shall we conclude that the therapy caused a reduction in symptoms?

Campbell and Stanley (1966, Table 1, p. 8) indicate that this design is weak regarding the following threats to validity:

• History. It is possible that some outside event took place between the pretest and post-test measurements besides exposure to treatment which affected the subject’s depression symptoms. For example, perhaps the pretest took place in the winter and the post-test in the summer, and the increase in light, pleasant weather and outdoor activity affected the subjects.
• Maturation. Depression symptoms may spontaneously resolve without treatment for some subjects. This could be mistaken for an effect of treatment. More generally, natural change over time as a result of development or other processes can alter post-test scores independent of the treatment.
• Testing. The act of completing the first depression assessment may itself alter responses to the second assessment even if the subjects’ underlying level of depression has not changed. This threat may seem to be a remote possibility in this example, but if the assessment was performance based, the subjects may do better on the post-test assessment due to their experience of taking the pretest.
• Instrumentation. Calibration of measurement instruments may drift over time, or the instruments may fail, creating an illusory apparent change in scores from pretest to post-test. This once happened to me when a device for recording heart rate began to fail in the midst of data collection. This may seem unlikely if the instrument is a self-reported symptoms rating scale that cannot fail or lose calibration, but it is possible for even simple survey instruments to exhibit problems with longitudinal measurement invariance, in which the meaning or value anchoring of scores drift over time (Meade et al., 2005).
• Regression. When subjects are selected on the basis of extreme scores at baseline, it is likely that follow-up measurements will be less extreme. This is an unavoidable consequence of measurement error (Crocker & Algina, 1986). Thus, the post-test depression scores may improve even if the treatment is ineffective.
• Mortality. ‘Mortality’ refers to dropping out of the study. If there is an association between depression severity and the probability of dropping out, or an association between response to treatment and the risk of dropping out, then a comparison of pretest and post-test means will not necessarily represent the causal effect of the treatment. For example, if the most severely depressed subjects quit the study, the post-test mean symptoms score will improve simply because the subset of subjects remaining is less depressed, even if the treatment had no effect.

Campbell and Stanley (1966) describe several alternative research designs that are much more robust to these threats to validity than the ‘One-Group Pretest–Post-Test Design’ considered above. One of these is called the ‘Pretest–Post-Test Control Group Design’. In this design, subjects are randomly assigned to a treatment group and a control group. Both groups receive a pretest assessment. The treatment group is exposed to the active treatment condition and the control group to a control or placebo condition. Then both groups are assessed again at post-test. The treatment effect in this design is measured by the difference in pretest to post-test change for the treatment group versus the control group. Any change resulting from history, maturation, instrumentation or regression should affect both groups equally and will not systematically impact the estimated treatment effect. Mortality is still a potential problem (e.g. if the treatment is unpleasant and induces people to quit the study), but if there is little to no dropout, it becomes an implausible alternative expl...

Table of contents

• Cover
• Half Title
• Acknowledgements
• Title Page
• Copyright Page
• Acknowledgements
• Contents
• Illustration List
• About the Author
• Acknowledgements
• Preface
• 1 Introduction
• 2 Conditioning
• 3 Directed Acyclic Graphs
• 4 Rubin’s Causal Model and the Propensity Score
• 5 Propensity Score Analysis
• 6 Instrumental Variable Analysis
• 7 Regression Discontinuity Design
• 8 Conclusion
• Glossary
• References
• Index