![]()
Chapter 1
The Framework
Kathryn A. Riley and Desmond L. Nuttall
Overview
Moves to develop indicators about school effectiveness and performance have been driven by national trends and by a broader debate about performance and accountability1. Nationally set indicators of performance — such as the standard assessment tasks, or the publication of performance on public examination — have increasingly become part of the new education currency: a medium for exercising choice and decision-making in the new education market place. As contributors to the book will suggest, such a framework is not unprobiematic.
This book offers some insights into the general debate about education, or performance indicators. It explores the background to the debate; the differing perspectives of policy-makers and practitioners; and the purposes, audiences and values of education indicators, both in the UK and international context. It builds on an international symposium on education indicators, held in 1991, which brought together a wide-range of participants to evaluate progress in the field of education indicators2.
Contributors to the book do not argue that education indicators offer a panacea to educational ills. But readers are offered an understanding of the issues involved; an appreciation of the role of indicators in evaluation and in sustaining school improvement; and a critique of their limitations. The contributors focus on both the context for measuring performance and the application of performance indicators at national and local systems levels, and at school level. The authors provide an overview of the current issues in performance measurement and illuminate the interrelated but different roles played by politicians, policymakers and practitioners in the development, interpretation and use of education indicators.
Contributions to the book focus on four major themes: why policymakers require information about performance; how such information relates to national contexts; the limitations of performance measurement; and the challenges in applying such information at the local system level.
Providing Information to Meet the Needs of Policymakers
Alan Ruby (chapter 2) suggests that education data collected in the past has been relatively simplistic and based on attempts to measure the size of the education system and its 'throughput'. From his perspective as a senior education adviser to the Australian Government, he suggests that politicians today — both in Australia and other OECD countries — increasingly require information about outcomes and in a timescale which is attuned to political realities. Such demands have implications for education officials who need to strengthen their skills in interpreting data; become more involved in specific policy-making problems; and give greater attention to data systems.
Desmond Nuttall (chapter 3) draws on work from a major OECD project on education indicators to examine the purposes and definitions of indicators and the lessons to be learned from past attempts to develop indicator systems. He explores the factors which have influenced the selection of particular indicators and suggests that research knowledge, technical, practical and policy considerations, as well as who the choosers are, influence the development of an indicator system.
Education Systems and National Contexts
Questions that are central to the concerns of politicians reflect different national contexts and purposes as Ruby suggests in his chapter. Ramsay Selden (chapter 4) describes the context for introducing a national indicators system in the USA, He traces the origins of the initiative's development and suggests that the initiative emerged from growing national concerns about the poor level of pupil performance and a belief that Federal Government should take a more defined role in setting national goals and in making state by state comparisons about performance. He draws on activities within the different states to illustrate how indicators can be used for very different sorts of purposes. In South Carolina, for example, performance indicators (as part of a wider model of accountability) have resulted in monetary rewards for good performance and sanctions, such as the removal of administrators, for poor performance.
The Limitations of Performance Measurement Systems
Selden also highlights the weaknesses in linking performance to pay and suggests that such systems can end up not necessarily rewarding the successful. He argues that whilst the introduction of performance indicators provides a useful focus on achievement, top-down approaches aimed at using testing to bring about change are limited, particularly when such developments are not linked to any concurrent efforts to provide support for school improvement. He points out, for example, that initial findings from the state by state analysis of pupil performance in mathematics suggest that student performance is closely associated with the levels of mathematics to which teachers were taught at colleges and their recent in-service professional development.
Kieron Walsh (chapter 5) also raises questions about linking performance to pay. He analyzes the broader issues around the rise of performance measurement within the public sector and argues that recent developments have ignored the complexities and difficulties of developing effective systems. Measuring performance does not automatically improve performance. Measurements need to be introduced for a system and not just for individuals, or individual organizations within that system. Otherwise hospitals, for example, might increase their 'efficiency' by discharging patients more rapidly — a move that would have a significant impact on housing and social services.
Walsh goes on to explore the problems associated with what he describes as a surveillance-based approach to performance measurement: an approach which he suggests can leads to rigidity, inflexibility and reduced learning. If organizations are to grow, he argues, they must be able to learn from their own mistakes. Problems are not necessarily resolved by tightening up inspection procedures, or introducing more indicators but by improving management systems and staff training and support.
Applying Indicators at the Local Systems Level
John Gray and Brian Wilcox (chapter 6) draw on a major survey of inspectors and advisers to establish what quantitative measures local education authorities have used to develop their framework for measuring performance. They examine the stumbling blocks in establishing indicators and suggest that if effective performance indicators are to be developed which will support the improvement of quality, then practitioners and policy-makers need to be clear about what counts as performance indicators; formulate a wide range of measures; and develop indicators which look at education processes, as well those which look at outcomes.
Kathryn Riley (chapter 7) charts the strategies and processes which local education authorities and schools have adopted in developing education indicators. She argues that indicators can be used to enhance decision-making and widen accountability — if LFAs are clear about their purposes. Education indicators also need to be integrated into a wider evaluative system which draws together inspection and selfevaluation.
Riley concludes that the task of linking the development of education indicators to improvements in the quality of the education service is likely to be made more difficult in the future by the requirements of the 1992 Education Schools Act (which has severed inspection from advice, support and development of schools) and by the 1993 Education Act (which has diminished the capacity of the local education authority to develop a quality framework and to support school improvement).
The chapter by Riley also illustrates the difficulties created by the increasing gap between the objectives and requirements of central government and the activities of local authorities. The Scottish context for change is, however, markedly different as John MacBeath illustrates in chapter 8.
MacBeath describes how the Scottish Office Education Department has worked to develop a a national indicators system which fosters ownership amongst Scottish regional education authorities and schools. He outlines how this partnership has taken place through a collaborative project with school administrators, classroom teachers, parents and pupils which compares perceptions about school performance. The major objective of the project has been to enhance school improvement: a theme that is explored in some detail in the concluding chapter by Riley and Nuttall.
![]()
Chapter 2
Education Indicators: Officials, Ministers and the Demand for Information
Alan Ruby
Politicians in all Western countries increasingly demand information that is simple, comparable, timely and which can be translated into the public arena. This chapter sketches out the implications and challenges of such demands for policy-makers.
Education officials in most Western countries have spent most of the last ten years being devolved, reformed, restructured, down sized and outplaced. Schools have become, or been exhorted to become, more self-managing, more client oriented, more outcome oriented and more accountable. As a consequence of these changes there has been and continues to be increasing demands on political decision-makers for information.
In the Anglo-Saxon countries the relationship between officials and 'ministers' is often summarized as 'advise, counsel and warn': to advise about the best ways to do things, to counsel the Minister about his/her responsibilities and to caution them against the inappropriate, the foolhardy and the counterproductive. We give facts, offer opinions and display our values. The essence of our business is information.
As information specialists our first response to these demands for information was literal compliance. We gave our Ministers more. We increased the quantity assuming the issue was volume not quality or timelines or 'relevance'. This failed — in most cases, or at least produced confusion.
The second response was didactic. Let me illustrate. In the early 1980s I was asked by my then Minister to provide the pupil teacher ratios (PTR) for ten specific countries. With the naivety of a junior official I replied: 'I'm not sure that you actually want that information. Its not really what you need to answer the question I assume you're grappling with.' Ignoring the silence I continued: 'PTR is not a good measure of relative resource allocation or usage. It does not necessarily translate into relative instructional time or offer valid proxies for inputs into the specific process of learning. PTR also ignores the fact that there are a lot of adults in schools who make substantial contributions to learning, both direct and indirect, who are not teachers.' I enumerated many categories and offered the observation that an adult to pupil ratio might be more meaningful if the data were available.
I went on to discuss technical difficulties in the comparable definition of 'teacher' across the ten countries, the differences between the ten countries and Australia which further limited the usefulness of the data and ended up with a survey of the data problems in this area both nationally and internationally. He with what I now see as inestimable patience thanked me and said 'Now, give me the data.' I did.
Reflecting on this exchange I was led to one conclusion — I could not deny the legitimacy of his request. I was unable to do so, and still am unable, not just because of notions of democracy and good government but also because we do not know how ministers actually make decisions. Personal and political decisions are so complex that we are unable to isolate and comprehend how all the different elements and factors are balanced and interconnected. In this context we cannot deny a decision-maker information sought because we believe it inappropriate to the task we assume is at hand. So the didactic response was also a 'failure' although it did strengthen our understandings about the weaknesses of our data. This has caused us to try to improve the quality of our information. To do that we need to understand then what has produced the demand for better information.
Education Reform and the Demand for Information
Apart from curiosity the four conventional motivations for wanting information are: the pragmatic, the moral, the conservative and the rational (Mitchell, 1989).
The pragmatic is, well, does it work? The moral: is it good? The conservative, which is not politically conservative in the sense of party political: is it necessary and the rational; can we make it better?
Those simple, conventional motivations do dominate public life but they do not help us understand why we have suddenly been faced, as officials, with demands for better informatio...
