Handbook of Item Response Theory

Volume 3: Applications

Wim J. van der Linden

• Publisher: Chapman and Hall/CRC
• Year: 2017
• Print ISBN:
  9781466514331
• eBook ISBN:
  9781351643702
• Topic: Psychology
• Subtopic: Probability & Statistics
• Index: Psychology

Drawing on the work of internationally acclaimed experts in the field, Handbook of Item Response Theory, Volume 3: Applications presents applications of item response theory to practical testing problems. While item response theory may be known primarily for its advances in theoretical modeling of responses to test items, equal progress has been made in its providing innovative solutions to daily testing problems. This third volume in a three-volume set highlights the major applications.

Specifically, this volume covers applications to test item calibration, item analysis, model fit checking, test-score interpretation, optimal test design, adaptive testing, standard setting, and forensic analyses of response data. It describes advances in testing in areas such as large-scale educational assessment, psychological testing, health measurement, and measurement of change. In addition, it extensively reviews computer programs available to run any of the models and applications in Volume One and Three.

Features

• Includes contributions from internationally acclaimed experts with a history of advancing applications of item response theory
• Provides extensive cross-referencing and common notation across all chapters in this three-volume set
• Underscores the importance of treating each application in a statistically rigorous way
• Reviews major computer programs for item response theory analyses and applications.

Wim J. van der Linden is a distinguished scientist and director of research and innovation at Pacific Metrics Corporation. Dr. van der Linden is also a professor emeritus of measurement and data analysis at the University of Twente. His research interests include test theory, adaptive testing, optimal test assembly, parameter linking, test equating, and response-time modeling as well as decision theory and its applications to problems of educational decision making.

Section III

Test Design

9

Optimal Test Design

Wim J. van der Linden

CONTENTS

9.1Introduction

9.2Introductory Example

9.3Solving Test-Assembly Problems

9.4Test Specifications

9.5Formalizing Constraints

9.5.1Quantitative Constraints

9.5.2Categorical Constraints

9.5.3Logical Constraints

9.6Objectives for TIFs

9.7Item Sets

9.8Multiple Forms

9.9Formatted Test Forms

9.10Adaptive Test Assembly

9.10.1Performance of the Solver

9.10.2Alternative Adaptive Formats

9.11Concluding Remarks

References

9.1Introduction

Item response theory (IRT) lends itself perfectly to the application of optimal design principles to the problem of test assembly. The first to open our eyes to the opportunity was Birnbaum in his contributions on the three-parameter logistic (3PL) model to Lord and Novick (1968). His basic idea was to use the notion of the Fisher information on the test takers’ ability parameters in the responses to the test items to optimize the selection of a test form from a calibrated item pool.

In this chapter, we follow Birnbaum’s lead and review the principles of optimal test design for the same model, which explains the impact of the abilities of test takers p = 1,… , P and the properties of items i = 1,… , n on their success probabilities πpi as

πpi=ci+(1−ci)exp⁡[ai(θp−bi)]1+exp⁡[ai(θp−bi)],p=1,…,P;i=1,…,I,	(9.1)

where θp ∈ (−∞, ∞), ai ∈ (0, ∞), bi ∈ (−∞, ∞), and ci ∈ [0, 1] are parameters for the test takers’ abilities and the discriminating power, difficulty, and impact of guessing for the items (Volume One, Chapter 2). In the current context of optimal test design, θ is an unknown parameter but the item pool is assumed to be well calibrated, so its parameter estimates have enough precision to treat them as known.

Psychometrically, the test-assembly problem amounts to the selection of a combination of item parameter values from the pool that results in optimal estimation of the as-yet-unknown ability parameters of the future test takers. Birnbaum’s (1968) choice of criterion of optimality was maximization of the (expected) Fisher information in the item responses about the ability parameters, which for the current model can be shown to be equal to

Ii(θ)=ai2[1−pi(θ)][pi(θ)−ci)]2pi(θ)(1−ci).	(9.2)

Taken as a function of θ, Ii(θ) shows us how much information a response to item i is expected to deliver about each of the unknown θ parameters. As the Fisher information is known to be asymptotically equal to the inverse of the variance of the maximum-likelihood estimator of θ, optimization of it is also known as an application of the criterion of D-optimality. For an introduction to the Fisher information, see Chang et al. (Volume Two, Chapter 7), while Holling and Schwabe (Volume Two, Chapter 16) should be consulted for a review of statistical optimal design theory and the prominent position of the criterion of D-optimality in it.

A helpful feature of Equation 9.2, due to conditional independence of the responses given θ, is its additivity. For a test of n items, the test-information function (TIF) follows from its item-information functions as

I(θ)=∑i=1nIi(θ).	(9.3)

Figure 9.1 illustrates the additivity in Equation 9.3 for five different test items.

Figure 9.1

Information functions for five different test items along with their TIF.

More specifically, Birnbaum’s method of optimal test design consisted of the following three steps: (i) formulating a goal for the use of the test; (ii) translating the goal into a target for the TIF; and (iii) selecting a subset of items from the bank with the sum of information functions that matches the target best. Figure 9.2 shows possib...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• Contents for Models
• Contents for Statistical Tools
• Preface
• Contributors
• Section I: Item Calibration and Analysis
• Section II: Person Fit and Scoring
• Section III: Test Design
• Section IV: Areas of Application
• Section V: Computer Programs
• Index