New Directions In Educational Evaluation

Ernest R. House, Ernest R. House University of Colorado, Boulder, USA., Boulder Ernest R. House University of Colorado

• Publisher: Routledge
• Year: 2013
• eBook ISBN: 9781135850838
• Topic: Education
• Subtopic: Education General

First published in 1986. Measuring the outcomes of educational practices is a modern phenomenon. Valuing their worth is as old as philosophy itself. It is the singular value of this collection of papers set in context and introduced by Ernest House that it holds in dynamic equilibrium both the measurement and the valuing sides of educational evaluation.

1
Introduction:
Evaluation and Legitimacy

Evaluation and Legitimacy

Ernest R. House

Over the past two decades or so, education evaluation has evolved as a field of intellectual endeavor complete with its own theorists, controversies, journals, and conferences. For example, at the 1984 annual meeting of the Evaluation Conference in San Francisco, 600 people attended, and this was only a fraction of the total number engaged in evaluating educational programs. Most of this activity has centered on the United States, but there is considerable interest in Britain, Canada, Australia, some northern European countries, and a nascent interest in Latin America and Asia.

There was, of course, formal evaluation of education prior to the 1960s, embodied primarily in the regional accreditation organizations in the United States and the Inspectorate in Britain. In the late 1950s and early 1960s, however, the development of the so-called new curricula, such as the new maths and new science, generated an interest in social science evaluation of these curricula. The Elementary and Secondary Education Act passed by the US Congress in 1965 required that all Title I (economically disadvantaged student) programs be evaluated. This requirement engendered a flood of evaluation activities.

Eventually, an evaluation mandate was placed upon most federal social programs in the United States. In other countries evaluation has not been mandated across the board but has been initiated for particular programs. As a result a sizable evaluation establishment has come into being with its own organizations, publications, institutions, and ways of behaving. The purpose of this book is to bring the reader up to date with some of the most pressing issues and controversies in this rapidly developing field, although no single book could possibly do justice to all the evaluation developments now underway.

As I have indicated, the overwhelming number of evaluation activities are in the United States, and the Reagan administration has had a strong influence on evaluation, indirectly by cutting back the funding of social programs. Since evaluations have been tied directly to social programs, this has meant a significant decline in the number of evaluations undertaken. The field of evaluation is no longer permeated by the boom town atmosphere that it once was, and this decline has led to a more sober and pessimistic view of the future, perhaps the inevitable outcome of all recessions. One can say unequivocally that evaluation is intimately tied to the initiatives of governments, and that shifts in government policies can result in significant changes in evaluation practices in a particular country or region. To put it another way, if evaluation is the watchdog of the public welfare, as some would have it, then it is a securely leashed watchdog.

A second significant development is in the social role that evaluation is expected to play. In the United States, at least, public education is being subjected to yet another round of industrialization. The immediate pressure stems from the harshly critical reports of several national commissions and the actions of state governors and legislators. American education is being transformed by so-called scientific management techniques, also called Tylorism. This trend is most clearly manifested in a massive employment of standardized tests — tests to promote students from one grade to another, to certify graduation from high school, to allow older students to enter teacher training, and to certify teachers when they finish training. In addition, experienced teachers are being evaluated to determine their competency and to assign them merit pay. In this social transformation evaluators are often designing, implementing, and monitoring the necessary evaluation machinery. This monitoring role is akin to the role of efficiency engineers in the scientific management movement, and I believe is a further extension of that movement. Of course, this shift in the role of what evaluators do also changes the content of evaluation.

I do not perceive this trend towards strong accountability policies in other countries at this time. In fact, in some parts of Australia there is movement in the opposite direction, towards action research and teachers controlling their own evaluation. Whether the industrialization of American education spreads to other countries or remains another manifestation of the American mania for technology, I do not know. It does seem to me that we may witness in the United States, at least, the industrial transformation of vast sectors of society, such as education and health care, that have never been fully rationalized in the industrial sense, and evaluation is playing an instrumental role in that transformation and is being transformed itself in the process.

Several of the chapters in this book reflect this social transformation. For example, in the first chapter Nevo tries to order the issues now current in the evaluation literature and finds that there is a lack of consensus among evaluation theorists concerning the social-political role of evaluation. In the second chapter I analyze the conceptual structure of one of the most widely-used evaluation textbooks and find it to be based extensively on deep-seated metaphors of industrial production so that the criteria for judging social programs are efficiency and effectiveness. In the last chapter in the book, which deals with teacher evaluation, the authors contend that the evaluation of teachers differs depending upon whether one sees teaching as labor, craft, profession, or art. If one sees teaching as labor, as a set of standard operating procedures planned and programmed by administrators, then evaluation becomes direct monitoring of teacher performance according to set standards. This is the direction American evaluation is currently taking.

The reader interested in evaluation would do well to keep this developing drama in mind since evaluation will be affected differently in countries where this trend is and is not occurring. Perhaps I should add that the industrialization of education is not a trend of which I approve nor a use of evaluation which I endorse. But my resistance to these events seems to have no bearing upon their realization.

An issue of long standing in the study of evaluation is the connection between fact and value. The traditional position of positive social science, of course, is that fact and value are quite separate and that it is the evaluator’s job to ascertain the facts and the clients and the sponsors and the public will place value upon these facts. Hence, the evaluator is in a value-free or a value-neutral position. This value-neutral position is manifested in many different ways in the extant approaches to evaluation. The distinction between fact and value is one of the most fundamental to evaluation, and to social science, and it is now under serious attack.

At least two of the papers here directly reflect dissatisfaction with the traditional fact-value dichotomy and the consequent role for the evaluator. Scriven asserts that all social science is in fact value-based and that researchers had best adopt the evaluation paradigm for all educational research. According to Scriven, ascertaining the value of something is quite analogous to ascertaining a fact about something and can be accomplished just as objectively as the determination of fact. The determination of the worth of various approaches and teaching techniques is precisely what the educational researcher should be doing.

For her part Kirkup claims that she has tried various approaches to evaluation and has found them wanting, particularly when she deals with a controversial and value-laden area like women’s studies. Her radical solution is to abandon the burden of objectivity altogether and to join with the program developers in a collaborative effort to determine the worth of what they are doing. The evaluator becomes in part an advocate rather than a judge or a neutral broker of information.

These are only a few of the possible positions along the fact-value line, and I expect considerable intellectual activity and controversy in the future as traditional evaluators try to defend their own objectivity, and hence legitimacy, by claiming their approaches are value-neutral, while other evaluators disengage from this position and attempt more radical approaches. Few issues are as fundamental to evaluation.

Several new evaluation approaches have been tried over the past two decades and we now have enough experience to see how some have turned out in practice. One of the hottest controversies has been the use of qualitative as opposed to quantitative evaluation techniques. Quantitative techniques have long been considered the sine qua non of modern social science, largely because their advocates believe that quantitative techniques ensure a degree of objectivity that any science must possess. This position has been assailed by advocates of qualitative studies, who have been successful in establishing qualitative approaches as more or less legitimate ways of evaluating. The more or less qualification is advisable because qualitative studies are still not considered quite up to par with proper quantitative techniques, the reason being, of course, that qualitative approaches are considered too subjective by many. Nonetheless, qualitative studies have been established as a legitimate minority approach and the warfare between the camps is diminishing in intensity, although skirmishing continues.

Some qualitative evaluators work under the banner of naturalistic evaluation and the two primary centers for naturalistic evaluation have been the Centre for Applied Research in Education (CARE) at the University of East Anglia and the Center for Instructional Research and Curriculum Evaluation (CIRCE) at the University of Illinois, the former founded by Lawrence Stenhouse and the latter by Tom Hastings. The ties between the two centers have been close ones since the early 1970s, although the ideas and context of each center are different. Although the boundaries of naturalistic evaluation extend far beyond East Anglia and Illinois, it seems appropriate that the chapters here are from personnel at those centers. Stake presents the fullest rationale he has yet attempted of the nature of naturalistic generalization, which he sees as the basis for naturalistic evaluation, while Walker struggles with the ethical problem of actually trying to employ naturalistic techniques. As his article demonstrates, these approaches are not panaceas but have formidable difficulties of their own. The final chapter in this section is by Kemmis, who was educated at CIRCE and employed at CARE for several years before returning to Australia. Kemmis proposes several principles for naturalistic studies, although there is some question whether he has not extended the principles to the point where his approach should be labeled a new one together. In any case, having achieved a measure of legitimacy for their approaches, the naturalistic evaluators now face the problem of refining and explicating naturalistic studies. One would expect to see differences emerge among them now that the common goal of legitimizing naturalistic studies has been reached.

Another effort in evaluation has been to encourage non-evaluators to participate in evaluations, in other words to democratize evaluations somewhat. This has not been very successful. As Weiss notes in her chapter, there have been several attempts, one of the most ambitious being to apply the stakeholder notion to two large, highly politicized programs. What happened in one of them is documented here in detail. It is clear that evaluation is a highly political activity which can even affect politics at the national level. It is also clear that participatory approaches have a very long way to go and face an uphill struggle against evaluation conceived as a purely technical act conducted by experts. I expect renewed attempts at participatory evaluation and renewed controversy.

Finally, I would like to note a structural shift that has long-range implications for the practice of evaluation. This is the shift towards moving evaluators inside the organizational structures. That is, organizations such as state education agencies and local school authorities now have their own offices of evaluation. In earlier days evaluation studies were usually contracted out to other agencies, such as universities, and almost all the evaluation literature assumes that evaluation will be conducted by an outside agency on a contractual basis. Increasingly, however, evaluation has been moved inside the organization. This means that the evaluators are now subject to the internal administrative structure, and the authority relationships inside their own organizations. This has profound implications.

Evaluation has become so important a function that major government agencies cannot afford to be without their own experts. Hence, many organizations establish units inside to perform this task. In modern liberal society there are few deep-seated beliefs shared by everyone. Legitimating ones policies and decisions has become a major difficulty. One way of doing this is to appeal to formal evaluation studies, which presumably are premised upon scientific techniques. Evaluation seems to be a necessity for modern governments, and too important a function not to be brought under some government control.

2
New Analyses:
Issues and Metaphors

Introduction

The first two chapters in this book provide recent analyses as to the status and nature of the field of educational evaluation. Nevo’s chapter assesses the current status of the field by denning the key issues. My own chapter investigates the internal workings of how we actually develop ideas about a field like evaluation. One chapter attempts to be comprehensive, the other in-depth.

Nevo’s chapter orders the vast evaluation literature by focusing upon ten critical questions that he believes determine the shape of actual evaluations. These questions include who should do the evaluation, to whom the evaluation should be addressed, and how the evaluation should be done. On some of these questions he finds a considerable degree of consensus among evaluation theorists. On other questions there is little consensus.

For example, there is considerable consensus on the definition of evaluation, the objects of evaluation, who should do the evaluation, and on standards for judging an evaluation. On the other hand, there is not much consensus upon what the functions of evaluation are, who should be served by an evaluation, or what the criteria for judging an evaluation object should be. Nevo contends there is an emerging consensus upon the issues of what kinds of information should be collected and what methods of enquiry should be used, but no consensus as to the socio-political function of evaluation or the role that evaluation should play in society. Overall, Nevo’s brief chapter provides a quick entry into the voluminous evaluation literature and his ten key questions provide an update on the current status of the field. Some of the questions he has posed will receive rather different answers from other authors in this book.

My own chapter provides a rather different kind of analysis. Rather than surveying the entire field of evaluation and seeing how different theorists address key questions, my chapter takes one prominent evaluation work and examines the conceptual structure of this work in detail. The purpose is to discover what fundamental ideas lie beneath a particularly elaborate and systematic theory of evaluation. I contend in this chapter that the conceptual base of one of the most widely used evaluation textbooks is in fact composed of several highly elaborated, overlapping metaphors. In particular, these are metaphors of industrial production and sporting contests. The industrial production metaphors include concrete images of the machine, the assembly line, and the pipeline. In other words, social and educational programs are seen as machines, assembly lines, and pipelines.

These metaphors are not merely casual, adventitious uses of imagery to enliven the narrative of the textbook occasionally, but are rigorously and systematically developed analogues for educational programs: in fact they comprise the fundamental cognitive structure of the work. Furthermore, the evaluation of the program is based upon these systemic metaphors. That is, the criteria for the program emerge directly from the metaphorical transformation of the program into industrial production and the other metaphors.

If my analysis is correct, it raises a number of questions about evaluations of educational programs. Are all evaluation approaches based upon implicit metaphors? How does this change the nature of evaluation? Where do these metaphors come from? Does this make evaluation arbitrary? Unscientific? Some of these issues are addressed in the chapter.

The Conceptualization of Educational Evaluation: An Analytical Review of the Literature

David Nevo
Tel-Aviv University

Many attempts have been made in recent years to clarify the meaning of evaluation and expose the distinction between evaluation and other related concepts such as measurement or research. The literature contains many approaches regarding the conceptualization of evaluation and the determination of its countenance in education. Many of those approaches have been unduly referred to as ‘models’ (for example, the CIPP Model, the Discrepancy Model, the Responsive Model, or the Goal-Free Model) in spite of the fact that none of them includes a sufficient degree of complexity and completeness that might be suggested by the term ‘model’. Stake (1981) rightly suggested that they be referred to as persuasions rather than ...

Table of contents

• Cover
• Title
• Copyright
• Contents
• Dedication
• Acknowledgements
• General Editor's Preface
• 1 Introduction: Evaluation and Legitimacy
• 2 New Analyses: Issues and Metaphors
• 3 Radical Propositions: Fusing Fact and Value
• 4 Naturalistic Evaluation: Acting from Experience
• 5 Participatory Evaluation: The Stakeholder Approach
• 6 Teachers and Evaluation: Learning to Labor
• Contributors
• Index