The ABCs of Evaluation

Timeless Techniques for Program and Project Managers

John Boulmetis, Phyllis Dutwin

• Publisher: Jossey-Bass
• Year: 2014
• ISBN:
  9781118103661
• Edition: 3
• Topic: Education
• Subtopic: Evaluation & Assessment in Education

Thoroughly revised and updated, the third edition of bestselling The ABCs of Evaluation offers an introduction to program evaluation. This comprehensive textbook covers topics such as stakeholder relationships, program design, data collection and analysis, reporting results, and other important steps in the evaluation process. The ABCs of Evaluation shows how to select participants for the evaluation and how to deal with multiple goals and objectives—including those of the organization, the staff, and the client. The authors illustrate the circumstances under which each evaluation model can be used, and offer tips on identifying data sources and collecting the data. This revision includes substantially increased coverage of theory, methodological approaches, the business of evaluation, the evaluator's role and responsibilities, RFPs and the grants process, logic models, data analysis, and writing the evaluation report. Also included are new cases and scenarios from various evaluation realms in social sciences, education, health, and human services. Throughout the book, charts, graphs, models, and lists help organize, extend, and facilitate the understanding of each evaluation concept.

Praise for the Previous Edition of The ABCs of Evaluation

"A useful general overview of the evaluation process. I would recommend it to program or project managers wanting to know more about the process of evaluation." —American Journal of Evaluation

"All students (and practitioners) should have this in their library; they will use it frequently." —Patricia McGee, PhD, associate professor, University of Texas, San Antonio

Companion Web site: www.josseybass.com/go/Boulmetis

1
What is Evalution?

---

Scenario One

The administration at Grandview Retirement and Nursing Facility strongly promotes ongoing education and training for all staff. In fact, when new personnel are hired, they agree to take two courses each year, selected from a long list of possible courses decided upon by administration and staff. Topics are many and diverse, from the law and patient rights to medication in geriatric care to the mind-body connection. Two of the recent training programs were devoted to answering these questions: (1) Is there a connection between mental and physical activity and sustained good health and longevity? (2) How do you encourage residents to stay both mentally and physically active? Program and nursing staff alike found the answer to the first question compelling: activity is indeed linked to well-being and longevity (Wells, 1997). They decided to institute additional programs for residents that would involve mental and physical activity.

Coincidentally, a volunteer had been coming to the retirement and nursing facility for about six months to work with interested residents in an informal gardening program. The volunteer, Ruth, discovered that one of the things the residents missed most when they left their homes was their gardening. The activity helped them regain what they had enjoyed and gave them a real opportunity for mental and physical engagement. The facility had a limited budget, so Ruth made her own in-kind donations.

From the beginning, however, the activity was extremely popular and she knew that available resources would not be sufficient to serve all the people who wanted to take part in the program. Ruth volunteered to look for funds, a grant that would allow Grandview to continue and enlarge the program. The administrators said they would be delighted if she would do the legwork, but of course they would need to know exactly what she was doing along the way so that they could keep their board of directors apprised.

A short while later Ruth discovered that the Beed Foundation funded this kind of program and she got their grant application. One of the first items she saw on the application asked for an evaluation of the program. The items in the evaluation question included listing program objectives (for example, to increase residents’ activity and mobility level) and listing measurements (both quantitative and qualitative) that would be used as indicators of achievement of those objectives. Ruth later learned that a quantitative measure might be the number of times residents took part in the gardening activities before the onset of the funded program compared to how many at the end of the program. A qualitative measure might be perceptions of garden activity staff regarding the focus and mood of residents.

The grant also called for explaining data collection methods, a discussion of any sampling that would be done, a description of the evaluation design, the data analysis that would occur, the staffing, and the final report.

Although she was puzzled about most of the items, Ruth hoped that subsequent meetings and discussions with the staff would answer her questions. She needed some basic questions answered first: Why do programs need to be evaluated? What is evaluation? What are you looking for when you evaluate a program? How is this different from research?

---

Unlike Ruth, readers of this book will not have to wait to have these questions answered. You just need to read Chapter One. After reading this chapter, you should be able to answer the following questions as they relate to the preceding scenario as well as to the chapter material:

1. What do all evaluations have in common?
2. How would you characterize the differences in the efficiency, effectiveness, and impact of a program?
3. Why evaluate in the first place?

Common Denominators

Before we embark on a definition of evaluation, we need to answer the volunteer's question. How does program evaluation differ from research? Unlike evaluation, research takes place in a precisely controlled environment. The Grandview gardening program could not comply with this definition. Among a number of other important reasons that we will discuss in Chapters Five and Eight, Grandview's project staff could not precisely control—nor would they want to—the number and choice of participants. In addition, research collects data to stretch the envelope of what is known in order to prove or disprove a hypothesis or presupposition. Evaluation looks at program and project objectives and asks whether they have been achieved, judges the worth of ongoing programs, decides upon the usefulness of new programs or projects, and so forth (Rossi and Freeman, 1993).

In both for-profit and nonprofit organizations, managers possess data (information) that could help to evaluate a program or project. These data are the one thing that all evaluations have in common regardless of the particular definition of evaluation one embraces: evaluation is the systematic process of collecting data that help identify the strengths and weaknesses of a program or project. The data may be as simple as records of attendance at training sessions or as complex as test scores showing the impact of a new educational program on increasing students’ knowledge across an entire school system.

Whatever definition you apply to evaluation, when your task is to perform a program evaluation you will almost certainly include a number of common evaluation steps. You will find these listed as program cycles in the Putting It All Together section later in this chapter.

Two Definitions of Evaluation

People do not always agree on one definition of evaluation. Following are two different definitions:

• Evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.
• Evaluation is the systematic process of collecting and analyzing data in order to make a decision.

Notice that the first ten words in each definition are the same. However, the reasons—the why—for collecting and analyzing the data reflect a notable difference in the philosophies behind each definition. The first reflects a philosophy that as an evaluator, you are interested in knowing only if something worked, that is, whether it was effective in doing what it was supposed to do. The second statement reflects the philosophy that evaluation makes claims on the value of something in relation to the overall operation of a program, project, or event. Indeed, many experts agree that an evaluation should not only assess program results but also identify ways to improve the program evaluated (Wholey, Hatry, and Newcomer, 1994). A program may be effective but of limited value to the client or sponsor. One can imagine, however, using an evaluation to make a decision (the second definition) even if a program has reached its objectives (the first definition). Federal grants are based on the first statement, that is, whether the program has achieved its objectives, but the harder decision to downsize or change may be a consequence of the second definition of evaluation.

Evaluating Efficiency, Effectiveness, and Impact

We can define evaluation even more precisely as a process that is guided by the reason for doing the evaluation in the first place. An evaluation might be a process of examining a training program in light of values or standards for the purpose of making certain decisions about the efficiency, effectiveness, or impact of the program. To carry out this task, you need to understand the concepts of efficiency, effectiveness, and impact. These three terms will be referred to from this point on as the levels of program evaluation. (See Table 1.1 later in this chapter.)

Table 1.1 Questions of Efficiency, Effectivene...