Programme Evaluation and Quality

A Comprehensive Guide to Setting Up an Evaluation System

Judith Calder, Calder, Judith

• Publisher: Routledge
• Year: 2013
• eBook ISBN: 9781136351075
• Topic: Education
• Subtopic: Education General

Offers advice to those involved in the design of open and distance learning courses on creating an organized system of programme evaluation. The text addresses the need for such a programme, the organization of the evaluation, the determination of priorities and programme implementation.

Chapter 1

The nature of evaluation

Think about the last time that you considered the need to make some sort of change. Before choosing a particular course of action, you would have reviewed the available options, or at least the options that you knew about. You would have assessed how well each option might meet your needs, and at what cost. You would then have weighed up the advantages and disadvantages associated with each of the options before making your decision.

The change you selected might have been about some personal matter such as your family finances, or something to do with your children’s future. Or it may just as easily have been related to your professional life. You may have been thinking about introducing a new course, or modifying the student registration system, or increasing student retention. Whatever your area of concern, in order to carry out any change, you will have had to work through the process which we call evaluation.

The process of evaluation which we employ to reach a decision as to the way forward is the same regardless of the area of concern or its source or even of its importance. The care we take, the methods we use and the amount of attention we give to the process in those different situations is another matter. In this chapter we will be looking at formal evaluation, considering the purpose of formal evaluation activities in open and distance teaching organizations and examining the different types of approaches to evaluation which are available to us.

Formal evaluation

Evaluation then is an activity with which everyone is familiar. The question is, how you can best use evaluation with open and distance learning provision. At the informal level, individual members of any institution will be actively engaged in making their own personal evaluations of activities which come within their own areas of responsibility. The problem will be that, as with all other spheres of life, individuals’ perceptions will be coloured and distorted by the particular lenses through which they see the world. We can only make an evaluation on the basis of the information to which we have access. The conclusions that we reach will be limited by the quality of that information – its comprehensiveness, relevance, up-to-dateness, accuracy.

A more structured approach

One way of looking at the process of evaluation is to view it as a series of different stages. The stages which comprise this cycle are shown in Figure 1.1. It should be emphasized that reality is usually much more untidy and idiosyncratic. Some stages may be omitted, and the sequencing may not always operate as shown. The old joke about deciding what the conclusions will be before carrying out the evaluation does, as is often the case, carry a grain of truth. For example political pressures may result in stage 7 actions being agreed on political grounds before the evaluation findings in stage 6 are available (a frequent habit with government departments).

Figure 1.1 The basic stages of evaluation

Identify an area of concern

This stage can be triggered in a number of different ways. Formal monitoring procedures such as reviews of pass rates, or course registration figures often identify situations which should be giving cause for concern. Informal means such as letters of complaint, or anxieties expressed by staff can lead to the recognition of the existence of possible problem areas. Cost concerns may result in pressure within the organization for the evaluation of a specific project or innovation, such as the use of interactive video for example. Or again there may be an institutional commitment to provide certain data or certain types of evaluation for external auditing, review or grant awarding purposes. If you think of your own institution, you can probably think of just as many if not more instances where the evaluation process has been triggered by external requests for data or because of political pressure than through the process of objective review. The trigger for the evaluation cycle may therefore operate in a variety of ways.

Decide whether to proceed

Not all problems or potential problems which are identified will be seen as having a sufficiently high priority to warrant further investigation. A decision will therefore need to be taken about whether or not to investigate further, or whether to commit resources for a thorough evaluation.

Investigate identified issues

The ways in which issues are investigated should, wherever possible, be determined by the requirements of the problem. For example, the evaluation of an issue such as the quality of guidance to tutors may be usefully approached using a mixture of in-depth discussion to establish the criteria used by the tutors themselves, plus some quantitative feedback to establish the scale of any particular problem areas.

Analyse findings

Whatever the type of study devised and carried out for the evaluation, the data collected need to go through some form of analysis stage. The extent and depth of the analysis will depend in part on the technical competence and in part on the specific interests and institutional requirements of those carrying it out. I have known examples where the analysis of course feedback data was limited simply to a one-page summary of students’ written comments presented as a report from the teaching team to ‘higher authorities’. I have also seen examples where weeks of sophisticated computer analysis were carried out on complex quantitative data in order to help the course team pinpoint the precise sources of students’ problems with a course.

Interpret findings

The more sophisticated and complex the study, the more important is the interpretation phase. The same set of analyses may well be interpreted in very different ways depending on the particular perspective of the interpreter. A high difficulty rating for a course module may be interpreted as evidence that the teaching approach needs further investigation and possibly some revision, or it may be taken as evidence that the students are insufficiently prepared for the course.

Disseminate findings and recommendations

The dissemination phase can be key in determining whether or not the evaluation findings are used. The timing of the dissemination, the target group for the findings, and the perceived relevance of the findings to people’s concerns will all need to be taken into account. For example, the importance of variations in student retention rates may be different for those responsible for ensuring the viability of future courses than for administrators responsible for ensuring adequate provision of exam rooms. The same set of information can carry very different messages to different groups. Increased student retention rates may be good news to some staff in an organization, and a mixed blessing to others.

Review findings, agree and implement corrective actions

These final two stages do need to be seen as part of the evaluation process. Evaluation is not an abstract research exercise but an essential tool of good management. In general the methodologies for the design and implementation of evaluation studies are well developed, but the methodologies for enhancing the likelihood of organizational use of evaluation findings is still developing. Hence the importance of recognizing that these two stages must be included in the cycle.

The purpose of evaluation

The aim of evaluation in the case of any organization must be to support that organization in achieving its goals. In other words, to enable it to become a more effective organization within whatever constraints it has to operate. In educational organizations, the need for formal evaluation activities is usually clearly recognized. In their 1977 review of major evaluation studies, Guttentag and Saar drew attention to the fact that ‘education is one of the most highly researched evaluation fields’ (Guttentag and Saar 1977).

The learning organization

Evaluation is used, or should be used, to enable institutions to operate as learning organizations. The importance of the role of the detection and correction of error is the basis for the ideas on organizational learning put forward by Argyris and Schön (1978). An important feature of their argument is the view of the organization as a unit or a whole in respect of the reviews of performance and the implementation of subsequent modifications.

For example, individuals or small groups such as course teams may have learnt that the submission rates on assignments for a particular course drop sharply at a certain point. There are a number of possible explanations for this phenomenon which would have to be investigated. It may be to do with the difficulty of the assignment or the course workload at that point. If that is the case, then the person responsible for the course will probably attempt to deal with the problem by changing the assignment or by cutting out some of the student study tasks. However, there may be institutional-level implications for this state of affairs. For example, the number of assignments which students are expected to complete, the monitoring of standards, the course approval strategy and the course testing strategies are all aspects where the institutional procedures may have to be modified if the problem is found to be sufficiently widespread or severe.

Programme evaluation

Programme evaluation in the field of open and distance teaching is relatively underdeveloped. By programme evaluation I mean evaluation which focuses on programmes of study. It is at this level that the pedagogic, management and often the financial responsibilities lie in education and training. It is usually here that responsibility for the detailed issues of quality and accountability have to exercised.

I have chosen the term ‘programme of study’ to describe sets or groupings of courses. Usually, these would be sets of courses which share some sort of common aim. That aim may be the award of a qualification for students who successfully complete a requisite number or series of courses in an area of expertise; or it may be that a particular audience is targeted, or a particular teaching medium is used.

Within any institution it would be a simple if onerous task to list large numbers of possible issues to which evaluation could make some contribution. However ‘busyness’ is no substitute for purposeful intervention at key points. The question then is how to determine what the key points are – how are we to identify the purposes of evaluation in such a way as to achieve the best match with the goals of the institution?

Diverse institutional goals

The overarching aims of a provider of education will be related to the provision of learning opportunities and to such associated activities as the accreditation of learning. But such global aims can also contain a diverse range of subsidiary goals. In an earlier work I discussed the different types of goals that learning providers can hold (Calder 1993). Four distinct groupings can be identified:

• society/economy centred

• institution centred

• subject centred

• learner centred.

The society/economy centred goals refers to the skill centred education and training which both public and commercial providers are increasingly encouraged to offer. Institutional goals can include institutional survival; high status among clients, other providers or funders; or public recognition. Providers may also hold ‘subject centred’ goals, by which I mean claims to scholarship and the desire to provide courses of a high academic quality. The learner centred goals emphasize the personal development aspect of learning and the need for learners to achieve not only subject knowledge and skills but also more sophisticated learning strategies and such intangible outcomes as self confidence, recognition of self worth, and a commitment to the community.

You may have noted the absence of student performance from the list. In the UK, the assessment of student performance is referred to by the term ‘assessment’. The term ‘evaluation’ refers primarily to the evaluation of the teaching and organization activities which support student learning and includes the assessment of student performance as just one aspect or function. However in his book on the assessment of students, Rowntree highlights the feet that assessment and evaluation are often treated as ‘virtual synonyms’. As he points out, there are many countries, including the USA, where the term ‘evaluation’ is used to describe both the assessment of individual student performance in terms of what they have learnt or accomplished and the evaluation of the teaching and other organizational activities which support student learning (Rowntree 1977). In fact some institutions use the term ‘evaluation’ solely to describe the assessment of student performance.

Needless to say, such differences in the the way the term is used can on occasion lead to considerable confusion. Discussions about ‘evaluation’ between professionals from countries separated by different traditions of usage of the same term can be enlivened by the misunderstandings caused by failure to check on the definitions of apparently common terminology. In this book, I will stay with the UK meaning of evaluation.

Diverse interest groups

The particular interest group which sponsors the evaluation is of particular importance in determining the purpose of any particular evaluation activity. Kogan (1989) described well the complexity of the way in which the nature of the evaluation is determined when he commented that

The nature of the evaluation will vary according to whether an intervention is primarily directed to, for example, improvements in quality, reduction in cost, equalisation of access, or improvements in working conditions; and it will also vary according to its sponsors whether they be managers, political leaders, client groups, or the workers who are subject to the evaluation.

What Kogan was...

Table of contents

• Cover
• Title
• Copyright
• Contents
• Series editor’s foreword
• Preface
• Acknowledgements
• 1. The nature of evaluation
• 2. The wider environment and evaluation
• 3. Baseline student statistics
• 4. Programme and curriculum development
• 5. Course development and delivery
• 6. Student recruitment and support services
• 7. Developing your own programme evaluation system
• 8. Organizing your programme evaluation system
• 9. A self-improving system
• References
• Index