Program Evaluation

Methods and Case Studies

Kenneth J. Linfield, Emil J. Posavac

• Publisher: Routledge
• Year: 2018
• ISBN:
  9781351592017
• Edition: 9
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

This text provides a solid foundation in program evaluation, covering the main components of evaluating agencies and their programs, how best to address those components, and the procedures to follow when conducting evaluations. Different models and approaches are paired with practical techniques, such as how to plan an interview to collect qualitative data and how to use statistical analyses to report results. In every chapter, case studies provide real world examples of evaluations broken down into the main elements of program evaluation: the needs that led to the program, the implementation of program plans, the people connected to the program, unexpected side effects, the role of evaluators in improving programs, the results, and the factors behind the results. In addition, the story of one of the evaluators involved in each case study is presented to show the human side of evaluation. This new edition also offers enhanced and expanded case studies, making them a central organizing theme, and adds more international examples.

New online resources for this edition include a table of evaluation models, examples of program evaluation reports, sample handouts for presentations to stakeholders, links to YouTube videos and additional annotated resources. All resources are available for download under the tab eResources at www.routledge.com/9781138103962.

1 Program Evaluation

An Overview

Suppose you are working for the Student Counseling Center at Western Tech in a year or two. The center plans to offer a Sexual Assault Prevention Program to women and men in the following month, and you have been asked to develop a plan that will permit the center’s staff to decide whether (a) to offer the program again because it is successful, (b) to alter the program in order to make it more useful to participants, or (c) to drop the program because it fails to meet a need. You might stop reading for a moment and write down the steps you think are necessary to measure the success of such a program. Among other points, consider what evidence would indicate the program is successful, what would mean that it should be improved, and what would show that it should be ended.

Because the center wishes to maintain and promote useful services to students, evaluating a program that might reduce sexual assaults on campus is an important assignment. The staff of the center would not want to devote resources for a program that does not meet the needs of students on campus; in such a case, something more helpful should be offered instead. If the program is good, you would want your plan to detect its strengths; where it needs improvement, you would want to detect its limitations. When you are finished with this text, you will be ready to tackle this project. To begin, what is program evaluation?

The practice of evaluating one’s own efforts is as natural as breathing. Cooks taste their gravy and sauce and basketball players watch to see whether their shots go in. Indeed, it would be most unwise after turning on the hot water to neglect to check the water temperature before stepping into a shower. At a basic level, program evaluation is nothing more than applying this commonsense practice to settings in which organized efforts, called “programs,” are devoted to helping people who need education, medical treatment, job training, safe streets, welfare assistance, safety while traveling in airplanes, recreational services, or any of the thousands of services provided in a modern society. Program evaluation is a methodology to learn the depth and extent of need for a human service and whether the service is likely to be used, whether the service is sufficient to meet the unmet needs identified, and the degree to which the service is offered as planned and actually does help people in need at a reasonable cost without unacceptable side effects. Utilizing research methods and concepts from psychology, sociology, administration and policy sciences, economics, and education, program evaluators seek to contribute to the improvement of programs.

There are several crucial differences between the natural, almost automatic, evaluations we carry on as we work and play and the practice of program evaluation in organizational settings. These differences make program evaluation harder than the self-evaluations that we all do. First, organizational efforts are nearly always carried out by a team—whether that be a school team of teachers, coaches, and administrators; a hospital team of physicians, nurses, and technicians; or a welfare department team of social workers, counselors, and file clerks. This specialization means that the responsibility for program effectiveness is shared among many people. Furthermore, the results of such programs—an educated student, a well patient, or an effective young mother—are not the sole responsibility of any individual.

Second, most programs attempt to achieve objectives that can only be observed sometime in the future rather than in a matter of minutes, as is the case with a cook adding a spice to a simmering pot or a painter repairing a damaged wall. In other words, as the time between an activity and the desired outcome of that activity lengthens, it becomes less clear what we are to observe in order to decide that the activity is being carried out appropriately or not and what could be done to improve the result of the activity. In fact, the choice of criteria to use in deciding whether a program is working well creates debates among evaluators, program personnel, clients, and funding agencies. How to respond when a problem is detected is often just as controversial.

Third, in contrast to evaluating our own efforts, there are many different parties involved in the evaluations of programs. For example, the effectiveness of teaching in a school may be assessed by evaluators who work for the central administration or the school board. It would not be surprising that individual staff members might feel threatened if the evaluators do not have their trust because, in an extreme case, the livelihood of the teachers might be harmed by an incompetently done program evaluation.

Last, programs are usually paid for by parties other than the clients of a program. Funds for the salaries of nurses are obtained from payments to hospitals by insurance companies and government agencies, and schoolteachers are paid through taxes collected by school districts. Although nearly all nurses and teachers want to do effective work, their job security is at least in the short term more dependent on keeping the program funders satisfied than serving their patients and students well.

Such complications as these make program evaluation considerably more difficult to do than the evaluations of our own activities that we perform so easily and naturally. This text focuses its discussion on evaluations of programs in human services, such as education, medical care, welfare, rehabilitation, or job training. Program evaluation is one form of the general discipline of evaluation that includes employee assessment, quality control in manufacturing, and policy analysis, among other activities (Scriven, 2003). The various kinds of evaluation are sometimes confused with each other. All forms of evaluation involve discovering the value and worth of something; however, the focuses, purposes, and methodologies are quite different.

Evaluation areas that need to be considered

A common assumption by people who are not familiar with program evaluation is that all evaluators do is measure the outcome of a program—perhaps counting how many of the clients in a substance abuse treatment program no longer use the substance, either at the end of the program or at a later time, or determining the typical change in knowledge (hopefully an increase) for those who have attended a series of workshops. As will be seen below, measuring outcomes is a very important part of program evaluation. But it is only one part. We will make the case briefly here and then repeatedly throughout this text that program evaluation is a much broader set of activities than just measuring the outcome of a program and that good evaluators consistently think about this wider range of elements. The following paragraphs introduce the seven general areas that program evaluators need to consider. It should be noted that some evaluations may not explicitly include all these elements, and at times a single one of the components may be a complete evaluation (for example, a needs assessment). But choosing not to include one or more elements should be a deliberate decision based on a clear rationale, not a matter of focusing so intently on a limited set of areas that one forgets other considerations.

Meeting Needs

Most often, program evaluations are conducted on existing programs where there is a reasonable likelihood that the programs will continue in somewhat similar fashion for the foreseeable future. As a result, the original need that led to the program may not be raised as an important point of the evaluation; nearly everyone involved may just assume that doing something to address that need is a worthwhile activity. In fact, many people may be unable to imagine not having certain programs. Most of us assume that education is necessary for children and that we must have police and fire departments in our communities without needing to document precise details of what those children do not yet know or of the potential for crime and fires. As you will see in Chapter 6, “The Assessment of Need,” there are a number of important aspects of this component of program evaluation beyond just documenting that there is a legitimate problem that the program addresses.

One overarching point, however, is a focus on unmet needs. Although we all need to breathe air, we do not have large numbers of programs providing air nor evaluations to discuss them. Rather obviously, programs are developed when one or more needs are not already addressed. So evaluators consider unmet needs—whether in an area that has never been addressed or the extent to which current needs are not fully met by existing programs. Various other aspects can also be considered, such as how the need might be expected to change in the future or how the need is perceived differently by different people.

Implementation

A common problem with programs is that some are either never implemented as planned or are implemented in such a way that people in need receive no or minimal benefit. Consequently, it is essential that evaluators confirm points ranging from the fact that program personnel have been hired and offices have been opened to documenting how many clients have found the program and the nature of the services provided. When the primary focus addresses such issues, the project may be called an implementation evaluation, in contrast to an outcome evaluation that also considers the results of the interventions.

Although the monitoring function of program evaluation may at first strike the reader as unnecessary, there are many examples of programs that, although funded, were never put into practice. Even more common, descriptions of programs in agency materials may be accurate only with regard to what was intended, but not what is currently happening. In fact, a recent estimate notes “that failure rates for organizational change implementation typically range from 28% to as high as 93%” (Durand, Decker, & Kirkman, 2014, p. 404). To be fair, the actual practices could be better than originally planned, rather than worse. In any case, an essential area for evaluators is monitoring the details of how the program has been implemented.

Stakeholders

In the process of conducting an evaluation, evaluators might generally expect to have some contact with the clients who are served by the program, especially indirect contact, by considering data from the clients such as information on their changed condition or their responses to a questionnaire. But it is essential that evaluators take into account the many people who are involved with a program in one way or another, those who provide the service as well as those who receive it, and those who are supportive from a distance as well as those whose connection is either mixed or even opposed to some degree. The idea that all of these people have a stake of some form in the program has led to the term “stakeholders” as a general way to talk about them.

Staff providing the services are obviously highly involved both because they deal day-to-day with the clients and because they generally are paid to provide the services. Paying attention to staff as stakeholders involves many points such as recognizing that therapists often know important details about clients beyond what is contained in agency records, that staff may either resist change or be eager for it, and that most employees have some level of apprehension about the results of an evaluation in terms of their own future. Of course, there are a range of staff positions in most agencies—therapists, administrators, support staff, and others. In addition, most agencies have a board that oversees general operations. Often there are others in the community who care about the program in some way—typically including family and perhaps friends of clients, people in the community who are passionate about the specific need being addressed, and occasionally those who are affected by financial aspects of the program. At times, those who live near a treatment facility may have reactions to details of the operation such as traffic or clients’ behavior or still other points.

On the one hand, it is impossible for evaluators to know everything about all stakeholders and completely unrealistic to try to make all stakeholders happy. But on the other hand, it is essential that evaluators make reasonable efforts to learn about the major stakeholders and any important current issues. A common element is that the staff providing services, administrative staff, and board members have different expectations for the evaluation itself. If administrators want to improve efficiency, board members want to lower costs, and therapists want more freedom to extend treatment, there are important implications for the progress of an evaluation and what evaluators should expect regarding cooperation from the stakeholders.

Recently, evaluation researchers have explored the principles that appear to support successful collaboration with stakeholders, such as clarifying the motivation for collaboration (Shulha, Whitmore, Cousins, Gilbert, & al Hudib, 2016) as well as what can be learned about the involvement of stakeholders in evaluation more generally, such as noting that studies have found substantial evidence both for benefits and for liabilities (Brandon & Fukunaga, 2014).

Side Effects

Efforts to solve problems usually create some undesirable side effects (Sieber, 1981). For example, medications that relieve one health problem may create problems in other systems of the body; sometimes these side effects are minor, sometimes they require medical attention or a change in medication, and sometimes, though rarely, they are fatal. Patients and physicians must therefore weigh the dangers of side effects, neither minimizing their importance nor avoiding helpful treatments. Even when the benefits appear clearly to outweigh the costs to medical professionals, others may perceive these effects differently. An important current challenge is that some parents come to different conclusions about the comparative value of vaccinations than physicians (Harmsen, Mollema, Ruiter, Paulussen, de Melker, & Kok, 2013). A parallel situation occurs in the area of educational and social service programs. Welfare policies can help people during crises to reestablish their productive roles in society; on the other hand, welfare policies can create long-term dependency (McKnight, 1995). Programs to provide social support to isolated people can have negative effects that have typically been ignored (Lincoln, 2000). Special education classes permit children with unique needs to learn at a comfortable pace, yet being in such a class carries a stigma. Consequently, program planners seek to develop services that provide benefits but minimize negative side effects. On the other hand, unforeseen side effects are occasionally positive: When people learn new skills, self-esteem often increases (Dawes, 1994; Felson, 1984).

Improvement Focus

Most of these seven areas cover elements of programs that good evaluators consider, measure, and even analyze with forma...