Principals of Modern Psychological Measurement

A Festschrift for Frederic M. Lord

H. Wainer, S. Messick

• Publisher: Routledge
• Year: 2012
• Print ISBN:
  9780898592771
• eBook ISBN:
  9781136558436
• Topic: Education
• Subtopic: Education General

Published in 1983, Principals of Modern Psychological Measurement is a valuable contribution to the field of Education.

I	STATISTICS AND ITS APPLICATIONS

In addition to his path-breaking work on mental test theory, Fred Lord has done a number of important papers on various aspects of statistical theory and practice. His fable about the statistical theory of football numbers has been reprinted many times (see Turnbull's Appreciation in this volume for more details of this). He has worked on various innovative methods for hypothesis testing in nonstandard situations; he has also clearly presented one very confusing problem— what has become known as Lord's Paradox.

Twenty-five years have elapsed since Lord first put forth his perplexing paradox. Since that time there have been a number of discussions of it attempting to unravel the mystery. Paul Holland and Donald Rubin here provide the first full solution; done from the perspective of a formal model for causal inference. Viewed in this way the apparent contradictions disappear, leaving behind a clear resolution of this problem, as well as pointing the way toward a general solution for many derivitive problems.

Robert Linn examines one important such problem in his paper on predictive bias. He points out that Lord's conclusion about what may and what may not be inferred in his cafeteria situation applies directly when one is trying to estimate the predictive validity of an admissions procedure.

Melvin Novick amplifies these remarks, drawing a parallel between Lord's and Simpson's paradoxes. He suggests that the implication of these paradoxes and the methods devised for resolving them will remain central to future work in statistical inference.

The last paper in this section, by Henry Braun and John Tukey, provides a new method for making inferences about the means of several sub-populations. They discuss the general problem of pair-wise comparisons and show the area of preference for their new maximum subrange procedure.

1	On Lord's Paradox

Paul W. Holland

Donald B. Rubin

Educational Testing Service

ABSTRACT

Lord's Paradox is analyzed in terms of a simple mathematical model for causal inference. The resolution of Lord's Paradox from this perspective has two aspects. First, the descriptive, non-causal conclusions of the two hypothetical statisticians are both correct. They appear contradictory only because they describe quite different aspects of the data. Second, the causal inferences of the statisticians are neither correct nor incorrect since they are based on different assumptions that our mathematical model makes explicit, but neither assumption can be tested using the data set that is described in the example. We identify these differing assumptions and show how each may be used to justify the differing causal conclusions of the two statisticians. In addition to analyzing the classic “diet” example which Lord used to introduce his paradox, we also examine three other examples that appear in the three papers where Lord discusses the paradox and related matters.

1. INTRODUCTION

Lord's Paradox first appeared in a short, two-page article (Lord, 1967) in Psychological Bulletin. This article presents a remarkable contrast between two statisticians who draw widely different conclusions from the same set of data. The culprit appears to be that the analysis of covariance cannot be counted on to make a proper allowance for uncontrolled preexisting differences between natural groups. Much to the dismay of the editor of Psychological Bulletin, Lord did not resolve his paradox. This fact increases the interest in the questions it raises. The impact of the paper has been an extensive analysis and criticism of the use of the analysis of covariance that still continues (e.g., Games, 1976; Lindley & Novick, 1981). Lord wrote two additional short pieces on the paradox—a second article in Psychological Bulletin (Lord, 1969) and an entry in the Encyclopedia of Educational Evaluation (Lord, 1973 in Anderson, Ball and Murphy, 1973, page 233). We base our discussions on these three articles by Lord.

Lord uses examples to illustrate his points, and there are four examples discussed in the three papers. Our approach differs from Lord's in that we first present a mathematical framework that is complex enough to accommodate what we regard to be the important features of the examples treated by Lord, and we then apply this framework to each of his examples. As will become evident, we believe that there are several different issues that arise in these examples, and we feel that our mathematical framework provides the structure for a precise analysis.

Our paper is organized as follows. In section 2 we describe the general mathematical framework or model for causal inference. In section 3 we apply this general framework to each of the examples appearing in Lord's three papers. Section 4 gives our general conclusions regarding the nature of Lord's Paradox. We include an Appendix which indicates various related results that follow from our model.

2. A MODEL FOR CAUSAL INFERENCE

In this section we describe our model for causal inference and derive the results from it that we need for the examples that Lord discussed. More technical consequences of the model are derived in the Appendix.

2.1 The Elements of the Model

The chief issue that is of concern in Lord's Paradox is the attribution of cause. Much has been written about causation but our point of departure is the analysis of causal effects given in Rubin (1974, 1977, 1978, 1980). However, it will be sufficient for our purposes to deal with a simplified, population-level, version of Rubin's model. We have used this simplified model elsewhere (Holland & Rubin, 1980) to analyze causal inference in retrospective, case-control studies often used in medical research.

Our model is similar to those used to describe many simple statistical problems. However, we are absolutely explicit about certain distinctions and elements that are usually left implicit in other discussions. We believe that it is impossible to give a coherent analysis of causal inference without being at least as explicit as we are here.

The basic elements of our model are:

1. A population of units, P
2. An “experimental manipulation,” with levels t or c and its associated indicator variable, S
3. A subpopulation indicator variable, G
4. An outcome variable, Y
5. A concomitant variable, X

Each of these components to the model needs further specification and we do this in the next subs...

Table of contents

• Front Cover
• Half Title
• Title Page
• Copyright
• Dedication
• Contents
• Preface
• Frederic M. Lord: A Biographical Sketch Howard Wainer
• Publications of Frederic M. Lord
• To Fred Lord: An Appreciation
• PART I: STATISTICS AND ITS APPLICATIONS
• PART II: ITEM RESPONSE THEORY AND ITS APPLICATIONS
• PART III: FACTOR ANALYSIS
• PART IV: OTHER MODELS FOR PSYCHOLOGICAL MEASUREMENT
• PART V: COMMENTS ON THE MEASUREMENT OF TRAITS
• Author Index
• Subject Index