1
Probability SurveyâBased Experimentation and the Balancing of Internal and External Validity Concerns1
Paul J. Lavrakas 1, Courtney Kennedy 2, Edith D. de Leeuw 3, Brady T. West 4, Allyson L. Holbrook 5, and Michael W. Traugott
1 NORC, University of Chicago, 55 East Monroe Street, Chicago, IL 60603, USA
2 Pew Research Center, Washington, DC, USA
3 Department of Methodology & Statistics, Utrecht University, Utrecht, the Netherlands
4 Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA
5 Departments of Public Administration and Psychology and the Survey Research Laboratory, University of Illinois at Chicago, Chicago, IL, USA
6 Center for Political Studies, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA
The use of experimental designs is1 an extremely powerful scientific methodology for directly testing casual relationships among variables. Survey researchers reading this book will find that they have much to gain from taking greater advantage of controlled experimentation with random assignment of sampled cases to different experimental conditions. Experiments that are embedded within probabilityâbased survey samples make for a particularly valuable research method as they combine the ability to more confidently draw causal attributions based on a true experimental design that is used with the ability to generalize the results of an experiment with a known degree of confidence to the target population which the survey has sampled (cf. Fienberg and Tanur 1987, 1989, 1996). For example, starting in the late 1980s, the rapid acceptance of using technology to gather data via computerâassisted telephone interviewing (CATI), computerâassisted personal interviewing (CAPI), and computerâassisted web interviewing (CAWI) has made it operationally easy for researchers to embed experimental designs within their surveys.
Prior to the 1990s, the history of experimentation in the social sciences, especially within psychology, essentially reflected a primary concern for maximizing the ability to identify empirically based causeâandâeffect relationships â ones with strong internal validity (Campbell and Stanley 1966) â with little regard for whether the study could be generalized with confidence â i.e. to what extent the study had strong external validity (Campbell and Stanley 1966) â beyond the particular group of subjects/respondents that participated in the experiment. This latter concern remains highly pertinent with the current âreplication crisisâ facing the health, social, and behavioral sciences and its focus on âreproducibilityâ as a contemporary criterion of good experimentation (Wikipedia 2018a,b).
This is not to say that prior to 1990 that all experimental social scientists were unconcerned about external validity (cf. Orwin and Boruch 1982), but rather that the research practices of most suggested that they were not. In contrast, practical experience within the field of survey research suggests that many survey researchers have focused on external validity, but failed to use experimental designs to enhance their research aims by increasing the internal validity of their studies. An example of this is in the realm of the political polling that is done by and for news media organizations. Far too often, political polls merely generate point estimates (e.g. 23% of the public approves of the job that the President is doing) without investigating what drives the approval and disapproval through the usage of question wording experiments (cf. Traugott and Lavrakas 2016).
A case in point: The results of a nonexperimental preelection study on the effects of political advertising were posted on the listâserver of the American Association for Public Opinion Research (AAPOR) in 2000. This surveyâbased research was conducted via the Internet and reported finding that a certain type of advertising was more persuasive to potential voters than another type. By using the Internet as the data collection mode, this survey was able to display the ads â which were presented as digitized video segments â in realâtime to respondents/subjects as part of the data collection process and, thereby, simulate the televised messages to which voters routinely are exposed in an election campaign. Respondents were shown all of the ads and then asked to provide answers to various questions concerning their reactions to each type of ad and its influence on their voting intentions. This was done in the individual respondent's own home in a room where the respondent normally would be watching television. Here, the Internet was used in a very effective and creative way to provide mundane realism 2 (and there by contribute to âecological validityâ) to the research study by having survey respondents react to ads in a context quite similar to one in which they would be exposed to real political ads while they were enjoying a typical evening at home viewing television. Unlike the many social science research studies that are conducted under conditions far removed from âreal life,â this study went a long way toward eliminating the potential artificiality of the research environment as a serious threat to its overall validity.
Another laudable design feature of this study was that the Internet sample of respondents was chosen with a rigorous scientific sampling scheme so that it could reasonably be said to represent the population of potential American voters. The sample came from a large, randomly selected panel of U.S. households that had received Internet technology (WebTV) in their homes, allowing the researchers to survey subsets of the sample at various times. Unlike most social science research studies that have studied the effects of political advertising by showing the ads in a research laboratory setting (e.g. a centralized research facility on a university campus), the validity of this study was not threatened by the typical convenience sample (e.g. undergraduates âvolunteeringâ to earn course credit) that researchers often rely upon to gather data. Thus, the results of this Internet research were based on a probability sample of U.S. households and, thereby, could reasonably be generalized to the potential U.S. electorate.
As impressive as these features of this research design were, the design had a serious, yet unnecessary, methodological limitation â one that caused it to miss a golden opportunity to add considerably to the overall validity of the conclusions that could have been drawn from its findings. The research design that was used displayed all the political ads to each respondent, one ad at a time. There were no features built into the design that controlled either for the possible effects of the order in which the respondent saw the ads or for having each respondent react to more than one ad within the same data collection session. As such, the causeâandâeffect conclusions that could be drawn from this nonexperimental study design about which ads âcausedâ stronger respondent reactions rested on very weak footing. Since no design feature was used to control for the fact that respondents viewed multiple ads within the same data collection session, the validity of the conclusions drawn about the causality underlying the results remained little more than speculations on the part of the researchers, because such factors as the order of the ads and number of ads were not varied in a controlled manner by the researchers. Unfortunately, this missed opportunity is all too common in many surveyâbased research studies in the social sciences.
This study could have lent itself to the use of various experimental designs whereby a different political ad (i.e. the experimental stimuli) or different subsets of ads could have been randomly assigned to different subsamples of respondents. Or, the order of the presentation of the entire set of political ads could have been randomly assigned across respondents. In either case, an experimental design with random assignment would have provided the researchers with a far stronger basis (i.e. a study with greater internal validity) from which to draw causal inferences. Furthermore, such an experimental approach would have had little or no cost implications for the research budget, and under a design where one and only one ad was shown to any one respondent, would likely have saved data collection costs.
Using this example as our springboard, it is the goal of this book to explain how experimental designs can and should be deployed more often in survey research. It is most likely through the use of a true experimental design, with the random assignment of subjects/respondents to experimental conditions, that researchers gain the strong empirical basis from which they can make confident statements about causality. Furthermore, because of the widespread use of computerâassisted technologies in survey research, the use of a true experiment within a survey often adds little or no cost at all to the budget. In addition, embedding an experiment into a survey most often provides the advantage that the cell sizes of the different experimental groups can be much larger than in traditional experimental designs (e.g. in social psychology), which has the benefit that...
