Planning and Conducting Formative Evaluations

Tessmer, Martin, Martin Tessmer

• Publisher: Routledge
• Year: 2013
• eBook ISBN: 9781136350511
• Topic: Education
• Subtopic: Education General

Formative evaluation is the process of reviewing of pilot stage courses in order to determine strengths and weaknesses before the programme of instruction is finalized. This text offers practical guidance on the main methods used to gather and analyze data on course effectiveness.

Chapter One

Central Questions and Issues in Formative Evaluation

What is formative evaluation?

Perhaps the best way to begin is to define each word in the term. ‘Formative’ is used in a developmental sense, as children are in their ‘formative’ or developing years and are susceptible to growth and change. The evaluation target is instruction in its formative stages, instruction that is developing and not yet finished or ‘grown up’ and is thus amenable to revision.

‘Evaluation’ is a data gathering process to determine the worth or value of the instruction, of its strengths and weaknesses. The identified strengths and weaknesses are used to revise the instruction to improve its effectiveness and appeal. Thus, ‘formative evaluation’ is a judgement of the strengths and weaknesses of instruction in its developing stages, for purposes of revising the instruction to improve its effectiveness and appeal. The evaluation is conducted by collecting data about the instruction from a variety of sources, using a variety of data gathering methods and tools.

When did formative evaluation originate?

It is always difficult to track the origin of an idea or method, but we do know that instructional materials were tried out and tested for improvement during the audiovisual movement of the 1920s when educational film was evaluated (Williams, 1983; Cambre, 1981). The use of formative evaluation (albeit under different names) was used through the 1930s to the ’50s in various projects, many of them upon educational film. Be that as it may, formative evaluation did not become a systematic design method until the late 1960s, when formal models of the process were drawn up, and tryout-and-revision became an integral part of the programmed instruction movement. Scriven (1967) then attached the name ‘formative evaluation’ to a revisionary process that had been referred to as ‘tryout’, ‘developmental testing’, and other terms. Today, there are still a number of terms used in place of formative evaluation, terms such as ‘pilot test’, ‘formative assessment’, ‘dry run’ ‘alpha/beta testing’, ‘quality control’ (my favourite) and the impressive ‘learner verification and revision!’

Until the 1960s evaluation of instruction was usually summative. Summative evaluation was of materials in their ‘adult’ (not formative) stage, meaning that they were already completed and in final form. The purpose of the evaluation was often comparative, to determine if the instruction was better than some other form of instruction. In terms of the design and improvement of instruction, this method was similar to locking the stable door after the horse had bolted, since the materials were already completed and disseminated, and could only be revised if a new version was designed. Markle (1989) refers to this as the difference between using evaluation to prove (summative) versus improve (formative), and Baker and Alkin (1973) as the difference between evaluation for validation (summative) versus evaluation for revision (formative).

With the advent of large scale curriculum projects in the 1960s, designers began to more strongly heed the words of Ralph Tyler (1942) and to see curriculum development as a continuous process of tryout and development for improvement. Thus, designers including Ken Komoski, Eva Baker, George Geis and Walter Dick began to set forth principles and procedures for formatively evaluating instruction. Formative evaluation then grew (with many growing pains) to become a recognized and respected part of systematic instructional development. Braden (1987) believes that formative evaluation distinguishes instructional design from all other instructional improvement methods.

Formative evaluation is not an activity to prove or validate the effectiveness of your instructional design, it is part of the instructional design itself. In other words, this is not an add-on process to find out if your instruction is effective, it is a problem-finding part of a design and product development process. The importance of this distinction is that many organizations still do not perceive formative evaluation as necessary to instructional development. They say, ‘If you have good designers and good producers, why have formative evaluation?’ They see formative evaluation as a waste of resources, or as a sign of insecurity about quality of the instructional development effort. However, many training and education organizations that utilize formative evaluation do so because they understand it is part of the design of effective instruction, and because they see it as cost-savings measure to economically ‘debug’ instruction and increase client satisfaction. This perspective is one that designers often must foster within the organizations in which they work.

Why do formative evaluation?

Over the last 30 years, a number of empirical studies have shown that formatively evaluating instructional materials has resulted in revised instruction that produces statistically significant increases in student performance over the original, unevaluated versions of the instruction (Nathenson and Henderson, 1980). These improvements have been reported on all types of instruction: computer-based instruction, simulations and games, text, and multimedia. Beneficial results have also been reported upon every evaluation stage: expert review, one-to-one evaluation, small group and field test. Even the use of a single learner for an evaluation has resulted in improved materials (Lowe, Thurston and Brown, 1983). Thus, there is evidence that using formative evaluation can improve the learning effectiveness of instructional materials.

Can formative evaluation make instruction more motivating or interesting as well as more effective? There is not enough evidence to say. Historically, most formative evaluation studies have measured student performance gains and ignored measures of student attitude or acceptance (or instructor attitude/acceptance!). However, evaluators have used both experts and students to evaluate the interest and acceptability of instruction to its users, and have obtained suggestions on how to revise instruction to meet these goals (Flagg, 1990; Nathenson and Henderson, 1980). Certainly, formative evaluation can be used to obtain criticisms and suggestions on the interest/motivation of the instruction to its users.

Do instructional designers actually utilize formative evaluation in their design projects? Zemke (1985) surveyed readers of Training magazine and found that more than 60 per cent of the respondents used some form of formative evaluation in their projects, more than most instructional design activities listed in the survey. Tessmer and Wedman (1992) in a later survey of corporate training professionals, found that almost half their respondents used some type of formative evaluation in their projects. Formative evaluation then appears to be part of the ‘real world’ of instructional design.

Even though formative evaluation is frequently used by practitioners, this does not mean that it is readily accepted by all organizations. Many managers or clients do not understand the purpose or utility of formative evaluation. They may think that:

• evaluation is something that you do after the product is finished,

• all these evaluations are unnecessary if you have competent and experienced designers,

• their personal evaluations are sufficient.

This means that instructional designers must often argue for including formative evaluation in a project, and must demonstrate its benefits. Designers argue for the evaluation as a quality control measure of instructional products. They demonstrate its benefits by citing the errors identified and revisions made in other formative evaluation projects, and explaining the cost savings that can result from revising instruction before it is finalized.

What instruction is evaluated?

The most popular target for formative evaluation is instructional materials. Surveying formative research for the last 30 years, it becomes obvious that text is the most evaluated type of instructional material, with numerous evaluations also undertaken with film, video, multimedia, and computer-based instruction. In fact, any form of material can be competently and profitably evaluated, including CD-ROM, interactive video, hypertext/media, and broadcast television or radio. The evaluation questions may change depending on the media attributes of the format evaluated, but the overall formative evaluation process remains the same.

Many designers somehow think that instructional materials are the sole target of evaluation. They ignore live instruction as evaluation candidates. In fact, any form of instruction that can still be revised before being used in the ‘real world’ can be formatively evaluated (Baker and Alkin, 1973). Workshops, seminars and lectures are all viable formative evaluation candidates. Person-led instruction is really just another form of media, since the person acts as an instructional delivery system with a unique set of media attributes. As a ‘media’, person-led instruction will also have its own type of evaluation questions that are pertinent to that type of media. As with other media, the overall process of formative evaluation is still applicable.

Formative evaluation is usually applied to materials being developed by the design team, but finished materials can also be evaluated for their adaptability to the instructional environments in which they will be used. As long as the purpose of the evaluation is to ‘revise’ the instruction by reorganizing or supplementing it, the evaluation can be a type of formative evaluation. For example, instructors can evaluate commercially prepared materials to see if they fit in with their learners, classroom schedules, curriculum objectives, and physical facilities. The results of the evaluation may mean that only parts of the instruction are used, or that new materials are developed to supplement the inadequacies of the commercial ones. Similarly, learners can evaluate the instruction’s learnability and appeal to determine if adaptations must be made to it.

It is important to remember that any size of instruction can be formatively evaluated. The target for a formative evaluation can be a unit, lesson, course or curriculum. Frequently, the target for evaluation is a unit or lesson simply because of the considerable time and money required to evaluate whole courses or curricula. As we shall see in the next chapter, designers must often choose the amount and type of formative evaluation they will do based on the constraints of the project. Where there is ‘too much’ instruction to evaluate within project constraints, such as a course or curriculum, they may select smaller segments to more thoroughly evaluate, segments that may allow them to generalize their findings and revisions to unevaluated segments.

The focus of this book is on the formative evaluation of instructional products such as texts, lectures, and multimedia instruction. The formative evaluation of instructional programmes is also an important part of education and training, but is conducted less frequently by instructional designers than formative product evaluation. Programmes could be an instructional service such as a writing programme, or a curriculum-wide project such as a maths literacy programme. Programme evaluation is frequently conducted by educational researchers, and can be a complex and time-consuming process. Readers who are interested in formative programme evaluation are referred to Bloom et al (1971).

What types of formative evaluation are there?

There are four classically recognized types of formative evaluation, which have these general characteristics:

• expert review – experts review the instruction with or without the evaluator present. The experts can be content experts, technical experts, designers or instructors.

• one-to-one evaluation – one learner at a time reviews the instruction with the evaluator and comments upon it.

• small group – the evaluator tries out the instruction with a group of learners and records their performance and comments.

• field test – the evaluator observes the instruction being tried out in a realistic situation with a group of learners.

Ideally, the evaluator would conduct expert and one-to-one evaluations first, revise the instruction, and then conduct a small group evaluation and ‘polish’ the instruction once again (see Figure 1.1). Revisions may be made between the one-to-one and expert reviews, or the evaluations may be carried out simultaneously. The final step is then to field test the instruction in the learning environments for which it was intended.

Figure 1.1 General sequence of formative evaluation types

Within these four types there are many variations. The number and variety of experts in an expert evaluation can vary, just as the number and variety of learners in a one-to-one can vary. The reality of the situation also varies, so that a small group evaluation is done in learning situations that are more realistic than one-to-ones, with field tests most closely emulating real-world situations and procedures. In a one-to-one evaluation the evaluator works closely with the learners, while in a field test the evaluator may be present only as an observer.

The methods also vary in the time and effort to conduct them, and the degree of sophistication of the evaluated materials. A single one-to-one or expert evaluation may require little time or money to conduct. Often these two evaluation stages are used with instruction in its rough stages: scripts of videos, photocopied texts with hand-drawn illustrations, outlines of workshop content and activities. Small group and field test instruction may be more resource-intensive and be used with more polished instruction: the video is produced, the text is in publishable form, and the workshop is conducted as a dress rehearsal. Because of these differences, expert reviews and one-to-ones are often completed first, to eliminate mistakes and inaccuracies before materials are produced in a more sophisticated format that costs more to revise.

In addition to these four formative evaluation stages, there are variations that can be used during an evaluation, such as:

• self-evaluation – designers or the design team evaluate their own instruction.

• expert panels – teams of experts discuss the instruction together with the evaluator.

• two-on-one evaluation – two learners review the instruction with the evaluator.

• rapid prototyping – an abbreviated version of the instruction is produced for immediate field evaluation

Self-evaluation i...

Table of contents

• Cover
• Half Title
• Full Title
• Copyright
• Contents
• Glossary
• Introduction: A Note to my Readers about this Book
• Chapter One Central Questions and Issues in Formative Evaluation
• Chapter Two Planning the Evaluation
• Chapter Three Expert Review
• Chapter Four One-to-One Evaluation
• Chapter Five Small Group Evaluation
• Chapter Six Field Test Evaluation
• References
• Index