Improving Testing

Process Tools and Techniques to Assure Quality

Rohit Ramaswamy, Cheryl Wild, Rohit Ramaswamy, Cheryl Wild

• Publisher: Routledge
• Year: 2017
• eBook ISBN: 9781351563109
• Topic: Education
• Subtopic: Education General

The primary purpose of this book is to demonstrate how proven quality assurance tools and methods that have been applied successfully in the manufacturing and service industries for the past 20 years can be applied in the testing industry. It defines what is meant by the term "quality" in testing and reviews how three business process concepts – standards, process planning and design, and continuous improvement – can be used to improve the way in which tests are designed, administered, scored and reported so that errors can be eliminated.

Part I

The Quality Model

1

The Risks and Costs of Poor Quality in Testing

Cheryl L. Wild
Wild & Associates, Inc.

Introduction

The premise of this book is simple—the testing industry as a whole is in dire need of improving the quality of its products and services. How can such a transformation occur? By applying process-oriented quality management tools and techniques along with psychometric principles to formally manage product and service quality. Not surprisingly, there are best practice organizations in the testing industry that are using some of these tools and techniques. In the chapters that follow, this book demonstrates how tried and proven tools used in achieving process quality in other fields have been applied by best practice organizations within the testing industry.

The purpose of this chapter is to provide an overall orientation to quality and process terminology and methods as they apply to the testing industry. First, the chapter discusses the evidence for whether there is a quality issue in the testing industry. Next, the term quality is defined and psychometric quality is described as a part of a more general definition of quality. Third, the testing industry is described in process terms, and finally, the costs of poor quality in testing are discussed. Is there really a quality issue in the testing industry?

Poor Quality or Poor Press?

Recent headlines have been lambasting test providers—“Report Blasts Tests for EMTs” (Scanlon, 2004), “Scores on Math Regents Exam to Be Raised for Thousands” (Arenson, 2003b), “Health Fields Fight Cheating on Tests” (Smydo, 2003), “Errors on Tests in Nevada and Georgia Cost Publisher” (Hendrie & Hurst, 2002), “Scoring Error Clouds Hiring of Teachers” (Jacobson, 2004). From one year to the next the headlines are similar—the publisher, the sponsoring agency, and the test may change but the underlying issue remains the same: mistakes may have disastrous consequences for test takers.

For example, in the spring of 2000, a mistake in the key of the Minnesota Basic Standards Test resulted in 7,935 students being told they failed when they actually passed; 50 students did not receive their high school diplomas with their classmates (Rhoades & Madaus, 2003). In 2003, the New York Regents Math exam had such a high failure rate that the New York Commissioner of Education set aside the scores of juniors and seniors and established a panel to review the impact (Arenson, 2003a).

Certification and licensure tests have not been immune to problems. In 2004, approximately 4,000 examinees who had previously failed their teacher certification examination were informed that a mistake had been made and that they had passed rather than failed. Many of these examinees had lost opportunities to pursue teaching careers as a result of their initial failing scores (Jacobson, 2004).

Tests used for college admissions have also had their share of problems. A scanning problem resulted in incorrect scanning of some answer sheets for the October 2005 administration of the SAT, resulting in more than 4,400 incorrect scores (Caperton, 2006). Approximately 1,500 Advanced Placement Examinations were lost after the May 2006 test administration (Strauss, 2006).

Is this problem any more acute now than it was a few years ago? Absolutely. This is because the number of tests administered has increased dramatically, more tests have high-stake consequences, and the changing platform of test delivery (from paper-and-pencil tests to computer-based tests) has introduced new problems along with new delivery opportunities.

The No Child Left Behind (NCLB) law has resulted in a tremendous expansion of testing. NCLB will at least double the number of tests administered in elementary and secondary schools. This law requires that all states test math and language arts in grades 3 to 8 and in at least one year in high school. The law doesn’t just impact student testing; states are required to have qualified teachers in the classroom. As evidence that teachers are qualified, states are also increasing the demand for teacher certification and/or licensing.

The number of tests administered in the business sector appears to be increasing as well, although there is no single source that collects this information. In certification testing, the growth appears to have an international impetus. The International Organization for Standardization, a worldwide organization of standards bodies, developed a new standard for organizations conducting the certification of persons (ISO/IEC 17024, 2003), and envisioned that the use of this standard would result in an explosion of certification and licensure tests in the global market place. By developing this standard, the organization hopes to establish the environment for mutual recognition and global exchange of personnel. Recognition as an international standard requires that a standard receive approval by at least 75% of the member bodies—reflecting the international importance of certification in the marketplace today. The standard provides general requirements (including organizational structure, finance, security, psychometrics, management system, subcontracting, human resource, and certification process requirements) for bodies operating certification of persons, allowing for a third party to verify that certification bodies are operating in a manner consistent with the standard. (More details about this and other standards are provided in chapters 4 through 8 of this book.)

Increased testing has placed a huge demand on the industry to increase production capacity. In manufacturing, increasing capacity requires building a factory or a plant. The testing industry is a knowledge-based industry—one that is very dependent on subject matter experts (to write questions, edit questions, and develop tests) and on psychometricians, usually PhDs with expertise in measurement theory and statistics. As Toch (2006) discovered in his review of the education testing industry, the testing industry is having a great deal of difficulty finding qualified people to meet the increasing demand.

Not only have the numbers of test takers in both the education and certification/licensure arenas increased—more of the tests have high-stake consequences, especially for schools, students, and teachers. Test results trigger consequences for schools—if students don’t achieve annual yearly progress (AYP) goals, schools may be required to report the failure to parents, develop improvement plans, allow students to transfer to other schools or districts, or provide resources for tutoring. After four consecutive years of not meeting goals, the state could withdraw funding, require personnel changes, and/or require curriculum revisions. Students may not be allowed to graduate in some states if they don’t pass the high school graduation test. And teachers’ jobs may depend on passing a certification examination or on improving their students’ scores. In some states, including Alaska, Florida, and Texas, bonus pay for teachers is being linked to improvements in student’s test scores (McNeil, 2006).

The increasing importance of test results has implications for testing products and services. Specifically, the higher the stakes for any given test the greater the motivation for cheating. More attention may need to be paid to the security of test storage, to training test administrators in their responsibilities for maintaining security, to checking data for evidence of cheating, and to providing new test editions more often. Processes and procedures that seemed adequate even a few years ago may no longer meet current user needs.

Finally, not only are more tests being given with higher stakes, but there is also a changing platform for test delivery. The testing industry is in transition from paper-and-pencil testing to computer-based testing in both the educational and business settings. This transition requires the development of more complex systems and software, new procedures for registering for the examination, new methods for score reporting and equating, new methods for selecting questions for the test, and new testing sites. These changes are creating new problems while eliminating others. For example, one of the problems with reporting educational test scores in a timely manner is the time required to have paper-and-pencil tests returned from a test center. Computer-based testing eliminates the security and logistical issues connected to shipping test booklets and answer sheets. However, new problems can and do occur—a loss of power during a testing session or loss of an Internet connection would seriously disrupt the test session and might even mean a loss of examinee responses. The development of a new platform requires design of new processes and procedures to ensure quality and reliability of delivery.

These three changes in the environment—drastically increased volume, changing customer requirements in terms of quality, and a changing delivery platform—would strain and challenge any industry. The evidence of problems in the delivery of products and services in testing strongly suggests that the testing industry is indeed challenged and that there is a need for a quantum leap in quality for the testing industry in the next few years.

There is little evidence, however, that quality is on the rise. For instance, Kathleen Rhoades and George Madaus (2003), researchers at the National Board on Educational Testing and Public Policy at Boston College, identified over 100 errors that occurred in testing programs over the last 25 years. Each error was described in detail in appendices to the report, allowing for independent evaluation of the errors. The causes of the errors (primarily ambiguous questions, incorrect scoring, score equating errors [equating is an empirical procedure to produce scores on multiple forms of a test that can be used interchangeably], programming errors, and data management errors) have not changed across the 25 years of the study. And these are only a subset of the errors that can occur—problems in administration at the test center, delays in score reporting, or breaches of security are not reported in the study.

Toch (2006) has reviewed the current status of the educational testing industry and has identified problems in industry capacity, timeliness of delivery, accuracy, and uneducated consumers (consumers include state department of education staff who contract with testing companies).

At the same time as newspaper headlines decry the problems with testing, testing experts claim that there are no serious quality problems in the testing industry. For example, Cizek (2001) argues that tes...

Table of contents

• Cover Page
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• Preface
• Acknowledgments
• Part I The Quality Model
• Part II Leadership
• Part III Standards
• Part IV Design and Planning
• Part V Monitoring and Improvement
• Part VI The Future of Quality in the Testing Industry
• Contributors
• Author Index
• Subject Index