Measuring School Performance & Efficiency

Leanna Steifel

• Publisher: Routledge
• Year: 2013
• eBook ISBN: 9781317929895
• Edition: 1
• Topic: Education
• Subtopic: Education General

This yearbook focuses on alternative ways to measure how well schools are educating their students, delivering what parents want, and using resources efficiently.

1
Measuring School Efficiency: What Have We Learned?

Leanna Stiefel, Amy Ellen Schwartz, Ross Rubenstein, and Jeffrey Zabel

Introduction

The headline of an April 15, 2004, Palm Beach Post article about Florida’s “Return on Education Index” proclaimed, “Web Site to Show Parents How Efficient Schools Are” (Miller, 2004). Similarly, Virginia has implemented a statewide performance review of its school system designed to “analyze how effectively each school division uses resources to boost student achievement” (State of Virginia, 2004). In 2003, during a battle over school funding, Governor Mike Huckabee of Arkansas declared, “We must insist upon a [school] system that has a level of efficiency that we can live with and afford” (Jefferson, 2003). At the same time, efficiency, with its connotations of profit-driven, assembly line automation, is not a universally-desired goal for education. Indeed, an editorial in the Halifax Daily News argued, “Slow, inefficient schools teach our children well” (Lafolley, 2004).

As these articles suggest, school performance and efficiency measurement have taken center stage in much of the debate and research in education policy since at least the mid-1990s. Despite the clear theoretical and practical importance of understanding the ways in which school performance can be measured, only limited research exists on alternative ways to measure how well schools are educating their students, delivering what parents want, and using resources efficiently. In this volume, the authors of eight chapters address the measurement of school performance, an issue that lies in between the study of technical characteristics of student assessments, on the one hand, and the effectiveness of accountability systems that use those assessments, on the other. Although psychometricians focus on the reliability, validity, and fairness of individual student assessments, and social scientists address whether state and local accountability systems that use those student assessments are effective ways to influence school performance, the authors of this volume consider the pros and cons of alternative measurements of school performance and efficiency, per se.

The intermediate ground between psychometric criteria and social science modeling is important. Pyschometricians often warn that standardized tests are used for purposes other than the intended ones—for example, to decide on pupil promotion—even though the tests may have less-than-desirable levels of reliability. Student test results may be especially problematic when aggregated to the school or district level to assess the performance of school and district personnel, in part because of the variety of ways in which the aggregation can be undertaken. For example, comparisons of average scores may yield very different results than comparisons of pass rates or students scoring above a minimum threshold. Perhaps when combined with other measures or aggregated in certain ways, however, test scores can produce meaningful measures of performance and provide insight on effective ways to organize schools and deploy resources. The intermediate ground is also essential to the literature on the effectiveness of accountability systems because accurate performance measurement is essential when we want to understand whether or not the change has produced desirable results.

Although accountability systems typically seek to measure school performance to distribute rewards for high performance, to sanction schools and students for inadequate performance, or to identify schools and students in need of remediation, the aims of social science research on school performance measurement are more basic. Before effective accountability systems can be developed, it is critical to understand the variety of output measures, model specifications, data, and quantitative techniques available for this purpose. This task is particularly complex in the provision of social services such as education because no bottom line measure of performance exists. Moreover, as all the chapters in this volume explicitly recognize, each school faces very different constraints and obstacles to maximizing aggregate student performance. Thus, one of the most vexing and important questions that must be answered is, “should valid school performance measures account for differences in student characteristics and school resources?” And if the question is answered in the affirmative (as it is in each chapter in this volume), the more difficult issue is how the student characteristics and school resources should be included in the performance measures.

This question also points out an important distinction between efficiency measurement and performance measurement. Although performance measurement may range from simple comparisons of average test scores to complex multivariate models, efficiency measurement presumes a connection between inputs and outputs. At its most basic, measures of technical efficiency seek to identify those schools getting “the most bang for their buck,” where bang is typically a series of test score measures and bucks may include a variety of inputs, such as resources, organizational characteristics, and student attributes. Similarly, allocative efficiency examines whether the correct (cost minimizing) mix of inputs is deployed to produce outputs. The definition of efficiency that we use in this volume is broad, encompassing a variety of concepts, methods, and data, although most definitions stress technical rather than allocative efficiency All the approaches share the common feature of attempting to disentangle the effects of various inputs on school outputs. Thus, although the techniques that the authors in this volume employ use different theoretical underpinnings, all attempt to measure schools’ true contributions to student learning, accounting for the wide disparities in resources and student characteristics that these schools may face.

This focus on the relationship between inputs and outputs distinguishes the methods from those used in many state accountability systems and in the federal No Child Left Behind (NCLB) Act, which often focus exclusively on student performance, frequently disaggregated by subgroups. The latter systems typically do not explicitly account for disparities in the levels and qualities of resources, or of demographic differences in the composition of student bodies across schools.

Why Study School Efficiency Measurement?

As the title of this book suggests, the authors focus on school efficiency and performance measurement not simply as an academic exercise but as an issue with real implications for education policy. For example, state and federal education policy has largely focused on rewarding or sanctioning schools and school personnel based on measures of school performance. Similarly, many “natural experiments” in education seek to isolate the effects of policies and programs by comparing the performances of schools with and without specific policies.

The work presented here has several clear implications for policy and research. First, understanding the strengths and weaknesses of various measures can help policy makers to develop fairer methods of measuring school performance by taking into account the uneven distribution of resources, students, and other inputs across schools. Important incentives and disincentives are embedded in any method of performance measurement, and it is critical that we understand the potential behavioral effects of any methods that might be used for accountability purposes. It is well established that simple comparisons of test scores—and, often, of changes in test scores—will typically rank well-resourced schools serving students from more educated and high-income families at the top, whereas most schools serving more at-risk students will languish near the bottom. By controlling for these factors, school efficiency measurement provides an opportunity to identify schools that are uncommonly successful at educating their students, although their successes might not be as readily apparent.

Second, although simple test score comparisons tell us little about the causes of high performance, efficiency measures may help to shed light on the relationships between inputs, broadly defined, and test scores. The search for a definitive link between spending and performance has been, in some ways, the “holy grail” of school finance research for the past 30 years, but clearly no simple answers exist. The methods explored in this book seek to shed light on those relationships by exploring not only how to measure performance and efficiency, but also what factors appear to be systematically related to high performance and efficiency.

Third, resource constraints are a fact of life in education, and the constraints have been particularly severe in the early part of the 21st century. School efficiency measures explicitly take account of these constraints, asking how we can maximize performance given available resources. It is important to use measures that emphasize costs as well as outputs in education, lest the education system become wasteful of the public’s tax dollars.

Overview of the Volume

The chapters in this volume share a number of common features. First, rather than focusing on the underlying validity or construction of the tests and other output measures commonly used to assess school performance, the authors work with available data to examine the ways in which these data can be used to analyze performance and efficiency. Second, the authors all go beyond simple test score measures to ask how various desirable outputs of schools, measured or unmeasured, can and should be combined with other features of schools or districts, such as characteristics of students or lavishness of resources, to produce meaningful measures of school performance. Third, the focus of most chapters is on the school rather than on the students, teachers, or administrators who are part of the school. Thus it is the aggregate school’s performance that the authors are aiming to measure.

The book is divided into two parts. The first section reports results of studies examining four specific quantitative techniques to measure school efficiency and performance. These include adjusted performance measures, production functions, cost functions, and data envelopment analysis. A unique feature of these chapters is that they all grew out of a single project using similar data designed to assess and compare the usefulness of these techniques for school efficiency measurement.¹

As described below and in the individual chapters, the four chapters make use of administrative data from New York City and the state of Ohio, two locations that have particularly well-developed, school-level information systems. Each chapter demonstrates the possibilities and pitfalls inherent in measuring performance and efficiency with existing data, and details the specific strengths and weaknesses of each technique.

The second half of the book includes four chapters examining alternative ways to conceptualize and measure school performance and efficiency: the use of efficiency measures in state accountability systems, stochastic frontier analysis, property values as school efficiency measures, and teacher assessment systems. The chapters reveal insights about the contribution of each performance measure. After a discussion of the major contribution of each chapter, this introduction concludes with a discussion of general themes. It is intended that these themes will prove useful to district, state, and federal policymakers as they work to improve both the ways in which schools are assessed and the accountability systems that use those assessments.

Overview of the Chapters

Part One

Adjusted Measures of School Performance: A Cross-State Perspective

Leanna Stiefel, Amy Ellen Schwartz, Hella Bel Hadj Amor, and Dae Yeop Kim examine one of the simplest quantitative methods used to measure school efficiency: adjusted performance measures (APMs). An APM uses multiple regression analysis to adjust a measure or measures of school output based on the inputs facing the school, including resources, student characteristics, and organizational factors. By regressing the output on the inputs, the model holds constant school resources and estimates a predicted level of achievement. The measure of interest is the difference between each school’s actual and predicted performance, or its adjusted performance measure.

Although the method is relatively straightforward and intuitive for those with a basic grounding in regression analysis, it is not without complications, for several of which the authors offer guidance. The authors’ unique contribution, however, is to expand on their previous work by estimating both generic and tailored APMs for New York City and Ohio to explore the consistency of results across jurisdictions and model specifications (see Rubenstein, Schwartz, & Stiefel, 1998; Stiefel, Rubenstein, & Schwartz, 1999; Rubenstein, Schwartz, & Stiefel, 2003). The implications of these comparisons are important: If the appropriate parameters for making adjustments are relatively universal, and if complex models offer few advantages over simpler ones, the usefulness of such adjusted measures for policy making would greatly increase.

To investigate this issue, the authors first estimate generic APMs for New York City and for Ohio. The generic specification uses a standard set of output and input data that are available in most states or school districts. These data include the percentage of students passing state or local tests, student socioeconomic characteristics, school size, and some aggregate expenditure measures. Interestingly, the results are very similar for the New York City and Ohio schools, particularly when math scores are used as output measures. Thus, school rankings change little when APMs are constructed using the parameters estimated in the other location. In other words, rankings of New York City schools are relatively insensitive to the use of parameters estimated with either New York City data or Ohio data. The similarities imply that it may be possible to develop a standard set of adjustment parameters that could be shared across districts or states.

The authors go on to explore the stability of APM results using different test scores, grade levels, years of data, and model specifications. Although the results generally display fairly high levels of consistency, the authors find a negative correlation between APMs constructed using consecutive years of data and using test score gains rather than levels. This latter result points out a fundamental problem in accountability systems based on test score improvements: Gains are exceedingly difficult, and sometimes impossible, to maintain over multiple years.

In sum, the authors’ results point out not only the importance of adjusting for inputs that are outside the control of schools to construct more accurate measures of performance, but also the practicality of employing these adjustments even when data or analytic capacity is limited. The authors highlight several other policy areas where such adjustments are used, but it remains to be seen whether such methods will take hold in education.

The Good, the Bad and the Ugly: Measuring School Efficiency Using School Production Functions

Performance-based accountability, a term that includes a variety of ideas, can be operationalized in several different ways. Amy Ellen Schwartz and Jeffrey E. Zabel provide a useful summary, pointing out that even when performance is measured using test scores, there are many ways to measure school performance—raw test scores, changes in raw test scores, and test scores adjusted for student and school characteristics and efficiency. In particular, efficiency measures how well both financial and student resources are used to produce test scores. For example, high-performing schools, based on either the levels or changes in their scores, adjusted or not, might turn out to be inefficient. The authors focus on developing a conceptual measure of school efficiency that is derived from a school-level education production function, and then proceed to work their way through econometric issues involved in estimating such a measure based on the fixed effects from a panel of 602 New York City elementary schools.

Zabel and Schwartz distinguish between three possible dependent variables, all derived from an original conception of an education production function: level scores (with a lagged score for the previous year and grade as a right-hand side variable), gain scores (the change in a score between adjacent grades between years), and difference scores (the change in scores in the same grade between years). Each of these allows estimates of school- or grade-specific fixed effects (interpreted as efficiency measures), but each has econometric virtues and problems. The most econometrically straightforward equation has levels of test scores as the dependent variable and the lagged test score as an independent variable; but the fixed effects in this model may suffer from three sources of bias—endogeneity because of the lagged test score, measurement error, or endogeneity of resources. The most econometrically correct estimation involves difference scores, with corrections for endogeneity, but it uses more years of data and is h...

Table of contents

• Cover
• Title
• Copyright
• Acknowledgments
• Contributors and Editors
• Table of Contents
• 1 Measuring School Efficiency: What Have We Learned?
• 2 Adjusted Measures of School Performance: A Cross-State Perspective
• 3 The Good, the Bad, and the Ugly: Measuring School Efficiency Using School Production Functions
• 4 Measuring School Performance Using Cost Functions
• 5 The Reliability of School Efficiency Measures Using Data Envelopment Analysis
• 6 Measuring School Performance: Promises and Pitfalls
• 7 Teacher Accountability Measures and Links to Learning
• 8 Revealed-Preference Measures of School Quality
• 9 Rethinking Educational Productivity and its Measurement: A Discussion of Stochastic Frontier Analysis Within a Budget-Maximizing Framework
• Index