Developing and Validating Multiple-choice Test Items

Thomas M. Haladyna

• Publisher: Routledge
• Year: 2004
• Print ISBN:
  9781138967472
  9780805846614
• eBook ISBN:
  9781136790652
• Edition: 3
• Topic: Education
• Subtopic: Education General

This book is intended for anyone who is seriously interested in designing and validating multiple-choice test items that measure understanding and the application of knowledge and skills to complex situations, such as critical thinking and problem solving. The most comprehensive and authoritative book in its field, this edition has been extensively

II

Developing MC Test Items

Thorndike (1967) noted that constructing good test items is probably the most demanding type of creative writing imaginable. Not only must the item writer understand content measured by the item but must determine whether the cognitive demand will involve recall, understanding, or application. Originality and clarity are key features of well-written test items. The set of four chapters in part II of this book is comprehensive with respect to writing MC items. Chapter 4 presents and illustrates many MC formats and discusses some important issues related to using these formats. Chapter 5 presents a validated list of guidelines to follow when writing MC items. These guidelines derive from past and current research (e.g., Haladyna et al., 2002). Chapter 5 contains many examples of MC items, most of which violate item-writing guidelines. Chapter 6 provides examples of test items taken from various sources. These items are exemplary because of their innovative format, content measured, mental processes represented, or some other feature. The purpose of chapter 6 is to give you a broad sampling of the effectiveness of the MC format for many types of content and cognitive processes, including the kind of thinking associated with the measurement of abilities. Chapter 7 is devoted to item generation. This chapter provides both older and newer ideas about how to prepare many items for different types of content and mental processes rapidly.

4

MC Formats

Overview

In this chapter eight MC formats are presented. Examples are given for each format. Claims are made about the types of content and cognitive processes that each format can elicit. This chapter shows the versatility of the MC format for measuring the recall or understanding of knowledge, some cognitive skills, and many types of complex mental behavior that we associate with abilities.

Contexts in which MC Formats are Used

Two main contexts apply to MC formats. The first is classroom testing, where the objective of a MC test is to obtain a measure of student learning efficiently. This measure is helpful when a teacher assigns a grade at the end of the grading period. This measure has value to teachers and students for giving students feedback and assistance in future learning or for reteaching and relearning content that has not been learned. The second context is a large-scale testing program. The purposes of this large-scale testing program might be graduation, promotion, certification, licensure, evaluation, placement, or admission. In this second context, MC is chosen because it is efficient and provides a useful summary of student learning of knowledge and cognitive skills.

MC Item Formats

This chapter presents a variety of recommended MC formats. One format is not recommended, and a recommended format serves as its replacement.

Conventional MC

The most common MC format is conventional. We have three variations. Each is shown in the Example 4.1. Each variation has three parts: (a) a stem; (b) the correct choice; and (c) several wrong answers, called foils, misleads, or distractors.

Question Format

Who is John Galt? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . stem

A. A rock star . . . . . . . . . . . . . . . . . . . . . . . . . . . . . foil or distractor

B. A movie actor . . . . . . . . . . . . . . . . . . . . . . . . . . foil or distractor

C. A character in a book . . . . . . . . . . . . . . . . . . . . . correct choice

Incomplete Stem (Partial Sentence)

John Galt is a character in an Ayn Rand novel who is remembered for his

A. integrity.

B. romantic tendencies.

C. courage.

Best Answer

Which is the most effective safety feature in your car?

A. Seat belt

B. Front air bag

C. Anti-lock braking system

Example 4.1. Three variations of conventional multiple-choice items.

Stem. The stem is the stimulus for the response. The stem should provide a complete idea of the knowledge to be indicated in selecting the right answer. The first item in Example 4.1 shows the question format. The second item shows the incomplete stem (a partial sentence) format. The third item shows the best answer format.

Correct Choice. The correct choice is undeniably the one and only right answer. In the question format, the correct choice can be a word, phrase, or sentence. In some rare circumstances, it can be a paragraph or even a drawing or photograph (if the distractors are also paragraphs, drawings, or photographs). However, the use of paragraphs, drawings, photographs, and the like make the administration of the item inefficient. With the incomplete stem, the second part of the sentence is the option, and one of these is the right answer. With the best-answer format, all the options are correct, but only one is unarguably the best.

Distractors. Distractors are the most difficult part of the test item to write. A distractor is an unquestionably wrong answer. Each distractor must be plausible to test takers who have not yet learned the knowledge or skill that the test item is supposed to measure. To those who possess the knowledge asked for in the item, the distractors are clearly wrong choices. Each distractor should resemble the correct choice in grammatical form, style, and length. Subtle or blatant clues that give away the correct choice should always be avoided.

The number of distractors required for the conventional MC item is a matter of some controversy (Haladyna & Downing, 1993). When analyzing a variety of tests, Haladyna and Downing (1993) found that most items had only one or two “working” distractors. They concluded that three options (a right answer and two distractors) was natural. Few items had three working distractors. In chapter 5, this issue is revisited. In this book, most of the examples contain three options because both theory and research suggest that for conventional MC three options works well.

Controversy About the Conventional MC Formats. Some controversy exists about the second variation, the incomplete stem (Gross, 1994). Statman (1988) provided a logical analysis of the issue. She asserted that with the completion format, one has to retain the stem in short-term memory while completing this stem with each option, evaluating the truthfulness of each option, or if short-term memory fails, the test taker has to range back and forth from the stem to each option, making a connection and evaluating the truth of that connection. Testing is anxiety provoking, and the added stress of the completion format may contribute to test anxiety, a problem that already troubles about one in four test takers according Hill and Wigfield (1984). The mental steps involved in answering a completion item also takes more time, which is undesirable. But research has not shown any appreciable difference when these two formats are compared (Rodriguez, 2003).

Test takers with limited English proficiency taking a test presented in the English language run a greater risk of having item format affect their performance. For this reason, the more direct question format seems better suited for these kinds of test takers.

Another issue is the use of blanks in the middle of the stem or question. The guideline to consider using is never to leave a blank in the middle or at the beginning of the stem. These blankety-blank formats are difficult for students to read. Such items also require more time to administer and reduce the time spent productively answering other items. For these many reasons, the use of internal or beginning blanks in completion-type items should be avoided. Example 4.2 shows the blankety-blank item format.

Child abuse is an example of ____________ violence, whereas sexism is an example of ____________ violence.

A. aggressive; structural

B. emotional; psychological

C. structural, emotional

Example 4.2. Embedded blank-type items.

Several creative innovations in conventional MC have added to the variety presented in this chapt...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright
• Contents
• Introduction
• I: A Foundation for Multiple-Choice Testing
• II: Developing MC Test Items
• III: Validity Evidence Arising From Item Development and Item Response Validation
• IV: The Future of Item Development and Item Response Validation
• References
• Author Index
• Subject Index