Handbook of Ethics in Quantitative Methodology

A. T. Panter, A. T. Panter, Sonya K. Sterba, A. T. Panter, Sonya K. Sterba

• Publisher: Routledge
• Year: 2011
• Print ISBN: 9781848728547
• eBook ISBN: 9781136888724
• Topic: Psychology
• Subtopic: Education General
• Index: Psychology

This comprehensive Handbook is the first to provide a practical, interdisciplinary review of ethical issues as they relate to quantitative methodology including how to present evidence for reliability and validity, what comprises an adequate tested population, and what constitutes scientific knowledge for eliminating biases. The book uses an ethical framework that emphasizes the human cost of quantitative decision making to help researchers understand the specific implications of their choices. The order of the Handbook chapters parallels the chronology of the research process: determining the research design and data collection; data analysis; and communicating findings. Each chapter:

• Explores the ethics of a particular topic
• Identifies prevailing methodological issues
• Reviews strategies and approaches for handling such issues and their ethical implications
• Provides one or more case examples
• Outlines plausible approaches to the issue including best-practice solutions.

Part 1 presents ethical frameworks that cross-cut design, analysis, and modeling in the behavioral sciences. Part 2 focuses on ideas for disseminating ethical training in statistics courses. Part 3 considers the ethical aspects of selecting measurement instruments and sample size planning and explores issues related to high stakes testing, the defensibility of experimental vs. quasi-experimental research designs, and ethics in program evaluation. Decision points that shape a researchers' approach to data analysis are examined in Part 4 – when and why analysts need to account for how the sample was selected, how to evaluate tradeoffs of hypothesis-testing vs. estimation, and how to handle missing data. Ethical issues that arise when using techniques such as factor analysis or multilevel modeling and when making causal inferences are also explored. The book concludes with ethical aspects of reporting meta-analyses, of cross-disciplinary statistical reform, and of the publication process.

This Handbook appeals to researchers and practitioners in psychology, human development, family studies, health, education, sociology, social work, political science, and business/marketing. This book is also a valuable supplement for quantitative methods courses required of all graduate students in these fields.

Ethics and Data Analysis Issues

DOI: 10.4324/9780203840023-13

Beyond Treating Complex Sampling Designs as Simple Random Samples: Data Analysis and Reporting

Sonya K. Sterba

Vanderbilt University

Sharon L. Christ

Purdue University

Mitchell J. Prinstein

University of North Carolina at Chapel Hill

Matthew K. Nock

Harvard University

DOI: 10.4324/9780203840023-14

This chapter addresses two issues: (a) how the method for selecting the sample ought to be reported in observational research studies, and (b) whether and when the sample selection method needs to be accounted for in data analysis. This chapter reviews available methodological and ethical guidelines concerning each issue and considers the extent to which these recommendations are heeded in observational psychological research. Discussion focuses on potential ethical implications of the gap between available methodological recommendations and current practice. A hypothetical case example and also a real world case example involving a daily diary study are used to demonstrate some alternative strategies for narrowing this gap.

It is important to note that both of the issues taken up in this chapter (reporting and accounting for sample selection in data analysis) arise after the sampling method has already been chosen. In contrast, a chapter on ethics and sampling in observational studies might have been expected to mainly concern the sample selection method itself—particularly whether a random (probability) or nonrandom (nonprobability) sample should be drawn.¹ The latter topic has long dominated informal discussions of ethics and sampling among social scientists, but has also often been misunderstood. Moreover, debate over choosing between probability versus nonprobability sampling has often led to an impasse, where observational researchers in particular fields (e.g., psychology) find only one sampling method pragmatically feasible (nonprobability sampling), and other fields (e.g., public health) find only one method statistically defensible (probability sampling; see Sterba, 2009). Our strategy is to begin with a brief overview of current and past perspectives on this controversial topic. The issues we address in this chapter are very general; they are relevant to whatever (probability or nonprobability) sample was selected. However, in discussing these issues in later sections, we periodically highlight relevant costs or benefits of using a probability versus nonprobability sampling method.

¹ Note that the issues that arise when deciding between random versus nonrandom assignment in treatment settings (e.g., Mark & Lenz-Watson, Chapter 7, this volume) are meaningfully different from those that arise when deciding between random versus non-random selection in observational (or experimental) settings, although there are certain parallels (Fienberg & Tanur, 1987).

Random and Nonrandom Sample Selection

When sampling was first proposed as an alternative to census taking, a distinction was drawn between two different methods for selecting samples from populations: probability (or random) sampling and nonprobability (or nonrandom) sampling (Bowley, 1906; Kaier, 1895). In probability sampling, the probability of selection for all units in the target population is known and nonzero. In nonprobability sampling, the probability of selection for some units is unknown, and possibly zero, and the finite, target population may be only loosely identified. Whereas early methodological debates sought to establish one method as superior and the other as uniformly unacceptable (Neyman, 1934; Stephan, 1948), such definitive conclusions were never reached despite extensive dialogues on the topic (see Royall & Herson, 1973; Smith, 1983, 1994; Sugden & Smith, 1984).

To summarize this debate briefly, collecting a probability sample by definition requires that key selection variables are observed and that the selection mechanism (i.e., the mechanism by which sampling units get from a finite population into the observed sample) is well understood. Both aspects in turn reduce the risk that selection on unmeasured, unobserved variables will bias results. Furthermore, the randomness entailed in a probability selection mechanism—specifically the fact that sampled and unsampled outcomes are assigned known probabilities—means that a distribution constructed from these probabilities can serve as the sole basis of inference to a finite population, without invoking strong modeling assumptions (e.g., Cassel, Sarndal, & Wretman, 1977). In contrast, nonprobability samples rely heavily on modeling assumptions to facilitate inference to a larger population, which is hypothetical. Nevertheless, should these modeling assumptions be met, there is a well-established statistical logic for inference from nonprobability samples (see Sterba, 2009, for a review of this logic). Hence both sampling methods have been recognized—initially at the 1903 Consensus Resolution of the International Statistical Institutes—and both are still frequently used.² Much attention has since turned to the two issues considered here: (a) what to report about sample selection, and (b) whether and when to account for sample selection in data analysis.

² This chapter pays specific attention to nonprobability (nonrandom) samples because they are most often used by psychologists.

Reporting About Sample Selection

Methodological Guidelines

For the issue of reporting about sample selection, our review necessarily takes a historical perspective because reporting guidelines have been in existence for a long time, yet have evolved considerably. The first methodological recommendations on reporting practices appeared almost immediately after the practice of sampling was first introduced. The International Statistical Institute’s 1903 Consensus Resolution called for “explicit account in detail of the method of selecting the sample” in research reports (Kish, 1996, p. 8). Similar recommendations were made in the proceedings of subsequent meetings, such as: “the universe from which the selection is made must be defined with the utmost rigour,” and “exactness of definition” is needed for “rules of selection” (Jensen, 1926, pp. 62–63). The nonspecificity of these guidelines, however, led to inconsistent reporting practices.

By the 1940s, mounting dissatisfaction over inconsistent reporting practices led the United Nations (UN) Economic and Social Council to convene a Subcommission on Statistical Sampling that met throughout the decade to develop a common terminology for such reporting (UN, 1946, 1947, 1948, 1949a). This Subcommission resulted in the formalized “Recommendations Concerning the Preparation of Reports of Sample Surveys” (UN, 1949b, 1949c). These recommendations highlighted the importance of reporting: (a) the sampling units; (b) the frame; and (c) the method of selecting (or recruiting) units—which may include (d) whether and how the frame was stratified before selection, (e) whether units were selected in clusters, (f) whether units were selected with equal or unequal probabilities of selection, and (g) whether units were selected in multiple phases. Also highlighted were reporting (h) sample size; (i) rates of refusals and attrition (see Enders & Gottschall, Chapter 14, this volume); (j) suspected areas of undercoverage of the frame; (k) methods undertaken after sample selection to gain insights into reasons for refusals and attrition; and (l) how the sample composition corresponds to preexisting survey data (e.g., census data). Table 10.1 provides definitions and brief examples of the italicized terms. Taken together, when a sample involves stratification, clustering, and/or disproportionate selection probabilities, it is conventionally called a complex sample, and those three key features are called complex sampling features. Sampling designs that lack all three features can be called simple (hence the term simple random sample).

Table 10.1 Some Terms Useful for Reporting About Sample Selection

Term	Definition	Examples
Sampling units	The physical units that were selected.	Persons, schools, divorce records, accident reports.
Sampling frame	All sampling units that had a nonzero probability of being selected into the sample.	A list of daycare centers in a community; all persons with registered university e-mail addresses; birth records from a particular county within a 2-month period.
Stratified sampling	Independently selecting sampling units from mutually exclusive groups, or strata, which may be preexisting or artificially defined.	Schools could be stratified into public vs. private; patients could be stratified into inpatient vs. outpatient; Alzheimer facilities could be stratified into nursing homes vs. assisted-living centers.
Cluster sampling	Using entire groups as sampling units, in lieu of individual elements, at one or more stages of selection.	Schools might be sampling units at a primary...

Table of contents

• Cover Page
• Half-Title Page
• Series Title Page
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• Preface
• Editors
• Contributors
• Software Notice
• Section I Developing an Ethical Framework for Methodologists
• Section II Teaching Quantitative Ethics
• Section III Ethics and Research Design Issues
• Section IV Ethics and Data Analysis Issues
• Section V Ethics and Communicating Findings
• Author Index
• Subject Index