Educational Evaluation, Assessment and Monitoring

A Systematic Approach

Cees Glas, Jaap Scheerens, Sally M. Thomas

• Publisher: Routledge
• Year: 2006
• Print ISBN: 9789026519598
• eBook ISBN: 9781135302061
• Topic: Bildung
• Subtopic: Bildung Allgemein

This book looks at the foundations of school self-evaluation from a scientific as from a practical perspective. Planning concepts, restructuring of education systems, organizational theory on schools, evaluation methodology and models of school effectiveness and school improvement are discussed as contributing to the overall conceptualization of school self-evaluation. A broad range of approaches is presented and methodological requirements are discussed. School self-evaluation contains controversial issues that reflect tension between the need for objectivity in a context that is permeated by values and potential conflicts of interests. Similar tensions may be seen to exist with respect to the static and "reductionist" aspects of available data collection procedures in a complex and dynamic situation and the appeal for external accountability on the one hand and improvement oriented self-refection on the other. The mission of the book is to clarify these tensions and offer ways to deal with them in practical applications. The school effectiveness knowledge base is offered as a substantive educational frame of references that serves an important function in selecting relevant factors for data collection and the use of the evaluation results.

PART 1

Basic Concepts

1

Monitoring and Evaluation
(M&E) in Education:
Concepts, Functions and
Context

1.1 Introduction

In this chapter basic concepts like evaluation, monitoring and assessment are defined. The chapter provides the outline of a framework to distinguish fifteen types of educational monitoring and evaluation. The framework depends on three basic dimensions: functions, data-strategy and (level of aggregation of) evaluation object.

All forms of evaluation consist of systematic information gathering and making some kind of judgment on the basis of this information. A further expectation is that this “valued information” is used for decisions on the day-to-day running of education systems or for more involving decisions on the revision and change of the system. The term “monitoring” is to be seen as a further qualification of evaluation, stressing the association with ongoing information gathering as a basis for management decisions, a reliance on administrative data and a stronger preoccupation with description than with “valuing”.

In the description the term “education system”, as the object of M&E, can be given different interpretations. It could be the national education system, a specific educational program, a school, or a classroom. The object of educational M&E can be defined at different levels of aggregation. Sometimes different terms are used when the object of evaluation differs. The term monitoring is often associated with the education system at macro level. Evaluation can be used for all objects but is most frequently associated with programs, as in program evaluation. When teachers are the object of evaluation the term “appraisal” is preferred in some national contexts (the UK in this case). And, finally, when the achievements of individual students are evaluated the term “assessment” is frequently used.

“Making empirically based checks on quality” can be seen as the overall purpose of educational M&E. Core functions are:

a. certification and accreditation; i.e. checking whether object characteristics conform to formally established norms and standards;

b. accountability; whereby object quality is made available for inspection to other units or the society at large;

c. (organizational) learning; whereby quality assessment is used a basis for improvement at the same object level.

M&E directed at these three core functions differs from a to c (in ascending order) in the degree of formality of criteria and standards, the external vs. internal nature of the procedures and a summative vs. a formative orientation.

Educational M&E makes use of different data sources. A pragmatic distinction is between data based on educational achievement measurement, data that is available from administrative records (including education statistics) and data that becomes available from expert review and educational research type of methods.

In subsequent sections specific types of educational M&E will be distinguished and elaborated by crossing object, function and data source as three basic dimensions.

1.2 Why do we Need Monitoring and Evaluation in Education?

The main motives for creating or improving provisions for monitoring and evaluation in education are three major concerns: to formally regulate desired levels of quality of educational outcomes and provisions; to hold educational service providers accountable and to support ongoing improvement in education. Decentralization policies in many countries are discussed as a stimulating contextual condition for systemic M&E.

• to formally regulate desired levels of quality of educational outcomes and provisions

Monitoring the quality of educational systems is not the first purpose of examinations that comes to mind. Examinations, for example at the end of lower secondary education, are there to certify individual students and to regulate what society can expect from those students (purposes of selection and stratification). Still examination can also be seen as a basis for determining the quality of educational systems and sub-systems, i.e. schools. Pass-rates on examinations are frequently used as performance indicators in judging the quality of educational programs and of schools.

When the unit of analysis to be formally evaluated is not the individual student but the school as an organization the term accreditation rather than certification is most commonly used. Quality control systems like the well-known ISO norms can be applied to schools to check whether central work and managerial processes are in place and the organization is customer oriented.

Finally, when explicit criteria and norms are used to compare educational achievement of national educational systems the term benchmarking is used. International assessment studies are needed to obtain basic and comparable data. In a global economy international benchmarking of educational quality is increasingly relevant for countries.

• to hold education systems accountable for their functioning and performance and support direct democracy in education

Accountability in education means that schools should provide information on their performance and functioning to outside parties. In this way schools and educational provisions are open to public review. Outside agencies, which have vested interest in the quality of education, may use this information for sanctioning (provide rewards or punishments). Such sanctions may be of an administrative nature, when originating from national, regional or local governing bodies, or take the shape of certain reactions from the consumers of education. Parents, for example, may try to persuade schools to alter their practices, or, in situations of free school choice, may take their children to another school.

Several global developments have stimulated demands for accountability in education, these are:

• the growing realization of the increasing importance of education, when economies develop into “knowledge societies”;

• the high costs of education, which in many countries are the highest post in government expenditure, paired with economic decline in the eighties this realization led to an increased concern with the efficiency of education provisions;

• an increased sense of openness and making public sector provisions in general accountable for the quality of their services (in the Netherlands for example the education inspectorate was forced by law to make public detailed reports on school reviews conducted by inspectors).

The substantive interest in accountability is usually in checking the quality or the general “well-functioning” of educational provisions. Quality is a rather general term. In actual practice concerns may relate to a good choice of educational objectives (relevance) or to the question whether the educational objectives are actually attained (effectiveness). There may also be an emphasis on the fair and equal distributions of educational resources (equity) or a specific concern with an economic use of these resources (efficiency).

Recognition that schools are to be accountable to other stakeholders than just administrators or governmental units also points at a basic requirement for democracy. Particularly when this concerns the immediate consumers and the clients of educational provisions, information from M&E can be seen as a basis for more direct democracy in education. In its turn more influence from the immediate clients and stakeholders is also seen as a stimulant of effectiveness and efficiency.

• as a mechanism to stimulate improvement in education

Next to formal regulation of performance norms and stimulating accountability and democracy a third major function of M&E. in education is stimulating learning and self-improvement of educational units. When evaluative information is fed back to the units concerned, this can be an important basis for corrective action and improvement. The evaluation-feedback-action sequence is a central mechanism for all kinds of learning processes, including so called “organizational learning”. The idea of learning from evaluation is central in the concept of formative evaluation, which is usually included in schemes for design and development in education.

• decentralization policies in education in many countries as a stimulating condition (either by decentralizing M&E as well, or centralizing M&E as a counterbalance of more autonomy at lower levels in other domains)

During the last two decades shifts in the patterns of centralization and decentralization have taken place in many countries, both in Western as in developing countries (OECD, 1998). Patterns of (de)centralization are best seen in terms of functional decentralization. This concept recognizes the fact that countries may decentralize educational systems in one domain, for example financial management, while simultaneously centralizing in other domains, like for example the curriculum.

This type of restructuring has stimulated the application of education M&E in two ways:

• more centralized control and stimulation of M&E as a counterbalance to providing more leeway and freedom with respect to school management and pedagogy (this pattern is most clearly visible in the UK);

• stimulation of school-based evaluation as part of decentralization “quality care” to the school level; to some extend this trend is discernable in Italy; in other cases despite decentralizing quality care to schools, M&E strategies are still mixed in the sense that more centralized forms are strengthened simultaneously (the Netherlands is a case in point).

What all three functions of M&E that were discussed in this section have in common is the purpose to stimulate quality. The first one, accreditation/certification depends on formally and officially established criteria and norms. The second one, accountability, may benefit from these formal criteria and norms, but is essentially relational in that lower level units in the system account for their performance to either official or unofficial (clients) stakeholders. The third (organizational learning) has a focus on within-unit improvement. Although accountability is ultimately related to improvement as well, the feedback-loop is shorter when M&E is applied internally.

In subsequent chapters specific applications of M&E will be discussed, which are differentially oriented towards one of these three basic functions (accreditation, accountability and self-improvement). The differences and correspondences between M&E types, primarily serving a particular function, will become clear when doing so. An important perspective, which will be given specific attention, is the option to exploit synergy between different basic forms and make efficient combinations.

1.3 A Conceptual Framework to Distinguish Technical Options in Educational M&E

Functions, data sources and objects are used as the basic dimensions to categorize M&E types in education. In this way 15 different types are distinguished.

Considering terminology assessment, appraisal, evaluation and monitoring are almost synonyms when one looks them up in the dictionary. They all have elements of valuing and judgments, being authorized to do so and of attributing numerical estimates. Monitoring stands out for its connotation of “detection” and association with controlling the running of a system over time and “keeping order”. (One definition the Concise English Dictionary gives of “monitor” is “a lizard supposed to give warning of approach of crocodiles”). In the usage of these terms in education, the most frequently chosen objects that are judged, appraised, evaluated and monitored seem to be most decisive in the choice of terms:

Assessment, when students are the object;

Appraisal, when teachers are the object;

Evaluation, when an educational program is the object;

Monitoring, when the day-to-day running of educational systems and organizations is at stake.

It should be noted, however, that the use of these terms differs between countries. The above definitions more or less confirm to the way they are used in the United Kingdom. In the USA, the term “testing”, or “educational testing” is more commonly used for the assessment of “traditional” subject matter mastery, whereas “assessment” has the connotation of “alternative assessment”, in the sense of measuring more general skills and attitudes.

The conceptual framework to categorize types of educational evaluation, assessment and monitoring consists of three basic data sources, three core functions and five different evaluation objects, each of which is defined at a particular level of aggregation.

The three basic data sources are:

• student achievement and assessment data

• administrative data and descriptive statistics

• data from expert reviews and systematic inquiry (surveys, observations and ratings)

The three functional areas are as described in an earlier section of this chapter:

• accreditation and certification

• accountability

• diagnosis/organizational learning

The five evaluation objects are:

• the education system at national level

• an educational program

• the school

• the teacher

• the individual student

By crossing these three dimensions (see Table 1.1) the main forms of educational M&E can be...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Contents
• Preface
• PART 1. Basic Concepts
• PART 2. Theoretical Foundations of Systemic M&E
• PART 3. Assessment of Student Achievement
• PART 4. Monitoring the Effectiveness of Educational Systems
• Part 1: Evidence from Industrialized Countries
• PART 5. Inspection and School Self-Evaluation
• References
• Index