Criterion-referenced Test Development

Technical and Legal Guidelines for Corporate Training

Sharon A. Shrock, William C. Coscarelli

• Publisher: Pfeiffer
• Year: 2008
• Print ISBN: 9781118943403, 9780787988500
• eBook ISBN: 9780470410400
• Edition: 3
• Topic: Business
• Subtopic: Human Resource Management
• Index: Business

Criterion-Referenced Test Development is designed specifically for training professionals who need to better understand how to develop criterion-referenced tests (CRTs). This important resource offers step-by-step guidance for how to make and defend Level 2 testing decisions, how to write test questions and performance scales that match jobs, and how to show that those certified as?masters? are truly masters. A comprehensive guide to the development and use of CRTs, the book provides information about a variety of topics, including different methods of test interpretations, test construction, item formats, test scoring, reliability and validation methods, test administration, a score reporting, as well as the legal and liability issues surrounding testing. New revisions include:

• Illustrative real-world examples.
• Issues of test security.
• Advice on the use of test creation software.
• Expanded sections on performance testing.
• Single administration techniques for calculating reliability.
• Updated legal and compliance guidelines.

Order the third edition of this classic and comprehensive reference guide to the theory and practice of organizational tests today.

Part One
Background: The Fundamentals

Chapter One
Test Theory

• What Is Testing?
• What Does a Test Score Mean?
• Reliability and Validity: A Primer
• Concluding Comment

What Is Testing?

There are four related terms that can be somewhat confusing at first: evaluation, assessment, measurement, and testing. These terms are sometimes used interchangeably; however, we think it is useful to make the following distinctions among them:

• Testing is the collection of quantitative (numerical) information about the degree to which a competence or ability is present in the test-taker. There are right and wrong answers to the items on a test, whether it be a test comprised of written questions or a performance test requiring the demonstration of a skill. A typical test question might be: “List the six steps in the selling process.”
• Measurement is the collection of quantitative data to determine the degree of whatever is being measured. There may or may not be right and wrong answers. A measurement inventory such as the Decision-Making Style Inventory might be used to determine a preference for using a Systematic style versus a Spontaneous one in making a sale. One style is not “right” and the other “wrong”; the two styles are simply different.
• Assessment is systematic information gathering without necessarily making judgments of worth. It may involve the collection of quantitative or qualitative (narrative) information. For example, by using a series of personality inventories and through interviewing, one might build a profile of “the aggressive salesperson.” (Many companies use Assessment Centers as part of their management training and selection process. However, as the results from these centers are usually used to make judgments of worth, they are more properly classed as evaluation devices.)
• Evaluation is the process of making judgments regarding the appropriateness of some person, program, process, or product for a specific purpose. Evaluation may or may not involve testing, measurement, or assessment. Most informed judgments of worth, however, would likely require one or more of these data gathering processes. Evaluation decisions may be based on either quantitative or qualitative data; the type of data that is most useful depends entirely on the nature of the evaluation question. An example of an evaluation issue might be, “Does our training department serve the needs of the company?”

---

Practice

Here are some statements related to these four concepts. See whether you can classify them as issues related to Testing, Measurement, Assessment, or Evaluation:

1. “She was able to install the air conditioner without error during the allotted time.”
2. “Personality inventories indicate that our programmers tend to have higher extroversion scores than introversion.”
3. “Does the pilot test process we use really tell us anything about how well our instruction works?”
4. “What types of tasks characterize the typical day of a submarine officer?”

Feedback

1. Testing
2. Measurement
3. Evaluation
4. Assessment

---

What Does a Test Score Mean?

Suppose you had to take an important test. In fact, this test was so important that you had studied intensively for five weeks. Suppose then that, when you went to take the test, the temperature in the room was 45 degrees. After 20 minutes, all you could think of was getting out of the room, never mind taking the test. On the other hand, suppose you had to take a test for which you never studied. By chance a friend dropped by the morning of the test and showed you the answer key. In both situations, the score you receive on the test probably doesn’t accurately reflect what you actually know. In the first instance, you may have known more than the test score showed, but the environment was so uncomfortable that you couldn’t attend to the test. In the second instance, you probably knew less than the test score showed due now to another type of “environmental” influence.

In either instance, the score you received on the test (your observed score) was a combination of what you really knew (your true score) and those factors that modified your true score (error). The relationship of these score components is the basis for all test theory and is usually expressed by a simple equation:

where X_o is the observed score, X_t the true score and X_e the error component. It is very important to remember that in test theory “error” doesn’t mean a wrong answer. It means the factor that accounts for any mismatch between a test-taker’s actual level of knowledge (the true score) and the test score the person receives. Error can make a score higher (as we saw when your friend dropped by) or lower (when it got too cold to concentrate).

The primary purpose of a systematic approach to test design is to reduce the error component so that the observed score and the true score are as nearly identical as possible. All the procedures we will discuss and recommend in this book will be tied to a simple assumption: the primary purpose of test development is the reduction of error. We think of the results of test development like this:

where error has been reduced to the lowest possible level.

Realistically, there will always be some error in a test score, but careful attention to the principles of test development and administration will help reduce the error component.

---

Practice

See if you can list at least three situations that could inflate a test-taker’s score and three that could reduce the score:

Inflation Factors	Reduction Factors
1. Sees answer key	1. Room too cold
2. __________	2. __________
3. __________	3. __________
4. __________	4. __________

Feedback

...

Table of contents

• Cover
• Contents
• Title
• Copyright
• Dedication
• List of Figures, Tables, and Sidebars
• Introduction: A Little Knowledge Is Dangerous
• Part I: Background: The Fundamentals
• Part II: Overview: The CRTD Model and Process
• Part III: The CRTD Process: Planning and Creating the Test
• Part IV: Legal Issues in Criterion-Referenced Testing
• Epilogue: CRTD as Organizational Transformation
• References
• About the Authors
• Index
• Advertisements
• End User License Agreement