Defending Standardized Testing

Richard Phelps

• Publisher: Psychology Press
• Year: 2005
• Print ISBN: 9780805849110
• eBook ISBN: 9781135614263
• Topic: Education
• Subtopic: Education Teaching Methods

The education reform movement of the past two decades has focused on raising academic standards. Some standards advocates attach a testing mechanism to gauge the extent to which high standards are actually accomplished, whereas some critics accuse the push for standards and testing of impeding reform and perpetuating inequality. At the same time, the testing profession has produced advances in the format, accuracy, dependability, and utility of tests. Never before has obtaining such an abundance of accurate and useful information about student learning been possible. Meanwhile, the American public remains steadfast in support of testing to measure student performance and monitor the performance of educational systems.

Many educational testing experts who acknowledge the benefits of testing also believe that those benefits have been insufficiently articulated. Although much has been written on standardized testing policy, most of the material has been written by opponents. The contributing authors of this volume are both accomplished researchers and practitioners who are respected and admired worldwide. They bring to the project an abundance of experience working with standardized tests.

The goal of Defending Standardized Testing is to:
*describe current standardized testing policies and strategies;
*explain many of the common criticisms of standardized testing;
*document the public support for, and the realized benefits of, standardized testing;
*acknowledge the limitations of, and suggest improvements to, testing practices;
*provide guidance for structuring and administering large-scale testing programs in light of public preferences and the "No Child Left Behind Act" requirements; and
*present a defense of standardized testing and a vision for its future.

Defending Standardized Testing minimizes the use of technical jargon so as to appeal to all who have a stake in American educational reform.

1
Persistently Positive: Forty Years of Public Opinion on Standardized Testing

Richard P.Phelps
Third Education Group

Public-opinion polls make good news copy, but are much maligned as a source of data for research, and for rather compelling reasons. For example:

• Polling is episodic. Pollsters poll when a topic is topical, and not when it is not. This leaves researchers without baselines, time series, or benchmarks for making comparisons.

• Often, polls are written quickly as stories emerge in the news and before they sink, leaving little time to pretest and edit items. Too often, the result is poorly written questions and a too frequent use of terms with ambiguous or varied meanings, which produce ambiguous results.

• Pollsters are often unfamiliar with a topic and, as a result, may pose questions based on naïve or inaccurate assumptions.

• Pollsters often work for hire, and the preference of clients for certain results can be satisfied all too easily.

• Neither the government nor anyone else regulates the quality of polls or pollsters. Anyone can hang out a shingle and call themselves a polling expert.

Just as there exist a variety of pollsters, polls themselves vary widely in quality. On even the most fundamental aspect of polling—the sampling—one finds great disparity. The typical political poll taken during an election campaign consists of no more than single telephone call attempts to a representative random sample over a single weekend, garnering a response rate of 20% of that sample or less. It is common with such “quick-and-dirty” polls for its methodology report, if one is written, to wax eloquent on the high quality of the sample selection, and neglect completely to mention the pathetically low response rate, which degrades the representativeness of the sample. As the political pundit Ariana Huffington once quipped, the lonely or unemployed—those most dependably at home and with time on their hands to talk to pollsters—are over-represented in poll results and are, consequently, making public policy decisions for the rest of us (Huffington, 2003a, 2003b).

Moreover, response rates have been declining precipitously for more than a decade, most likely due to public revulsion toward telemarketing. As one of the nation’s leading pollsters remarked recently, “response rates are getting so low, they’re starting to get scary.” A half-century ago, when many polls were still conducted door-to-door and face-to-face, response rate percentages in the high nineties were ordinary. Today, pollsters feel lucky when they exceed 50.¹

This begs the question: If public-opinion poll data are so untrustworthy, where can one find a better gauge of the public’s preferences? Perhaps in elections and referenda? Unfortunately, at least in the United States, elections and referenda have serious response-rate problems, too. They attract only a small, usually unrepresentative, minority of the electorate to the polls, to choose among sharply constrained, limited choices, which were themselves made by small, exclusive elites within political parties or interest groups. Moreover, election outcomes are subject to multiple manipulations that can pervert the electorate’s true wishes, including gerrymandering, asymmetric financing, unfair candidate marketing (particularly just prior to election day), and bad weather.

Despite their flaws, opinion polls and surveys may represent the best available alternative for divining the public’s wishes in our democracy, arguably even better than the very expensive, highly controlled and monitored official elections.

Besides, not all polls are “quick and dirty.” High-quality polls take the time and make the effort to reach the more difficult-to-reach folk in order to make their samples truly representative in result as well as in design. More patient pollsters typically allocate 2 weeks of time for telephone calling and for repeated attempts to reach the busier, the seldom-at-home, and the recalcitrant. Any more, the better polls also supplement a standard sample questionnaire with small focus groups of key subgroups that delve more deeply and expansively into a topic.

Even the “quick-and-dirty” polls can provide useful insights, however, when reinforced by other polls and surveys conducted concurrently or by multiple efforts on the part of one pollster. Political polls in high-profile elections, for example, are seldom taken in isolation but, rather, form patterns of responses retrieved in series and batches. Particularly when conducted by experienced political pollsters with sufficient knowledge to appropriately adjust results to compensate for response group biases, the patterns can tell a trustworthy story.

Having said that, experience and insight with election polls do not necessarily translate all that well to education issue polls. Some pollsters have developed an expertise with the subtleties of public opinion on education issues, but more have not.

Still, despite their flaws, public-opinion polls on education issues in general and standardized testing in particular remain important as unique representations of the public’s preferences (Phelps, 1998). Moreover, they provide a counterweight to the numerous program evaluations and surveys conducted by education professors and education interest groups that, overwhelmingly, solicit the attitudes and opinions only of other educators—teachers and administrators.

Table 1.1 displays my count of the numbers of studies, both polls and surveys, which posed questions about standardized testing, classified by type of respondent group—education providers (i.e., teachers, administrators, education professors) and education consumers (i.e., the public, parents, students, policymakers). There is some double counting of studies, as some surveyed multiple groups of respondents.

TABLE 1.1 Respondent Groups in Public-Opinion Polls and Research Surveys

Number of Testing St...

Table of contents

• Contents
• Foreword: The Rest of the Story
• Introduction
• 1 Persistently Positive: Forty Years of Public Opinion on Standardized Testing
• 2 High-Stakes Testing: Contexts, Characteristics, Critiques, and Consequences
• 3 The Rich, Robust Research Literature on Testing’s Achievement Benefits
• 4 Some Misconceptions About Large-Scale Educational Assessments
• 5 The Most Frequently Unasked Questions About Testing
• 6 Must High Stakes Mean Low Quality? Some Testing Program Implementation Issues
• 7 Whose rules? The Relation Between the “Rules” and “Law” of Testing
• 8 Teaching For the Test: How and Why Test Preparation Is Appropriate
• 9 Doesn’t Everybody Know That 70% is Passing?
• 10 The Testing Industry, Ethnic Minorities, and Individuals With Disabilities
• 11 A School Accountability Case Study: California API Awards and the Orange County Register Margin of Error Folly
• 12 Leave No Standardized Test Behind
• APPENDIX A Polls and Surveys That Have Included Items About Standardized Testing, in Reverse Chronological Order: 1954 to Present
• Appendix B Some Studies Revealing Testing Achievement Benefits, by Methodology Type
• Author Index
• Subject Index