The ABCs of Evaluation

Timeless Techniques for Program and Project Managers

John Boulmetis, Phyllis Dutwin, Phyllius Dutwin

• Publisher: Jossey-Bass
• Year: 2014
• Print ISBN: 9780470873540
• eBook ISBN: 9781118103661
• Edition: 3
• Topic: Education
• Subtopic: Evaluation & Assessment in Education

The ABCs of Evaluation: Timeless Techniques for Program and Project Managers, 3rd Edition

Thoroughly revised and updated, this third edition of the bestselling book offers an introduction to program evaluation including information on stakeholder relationships, program design, data collection and analysis, reporting results, and other important steps. This edition expands coverage of evaluation models and logic models. Written for any program type or setting, The ABCs of Evaluation shows how to select participants for the evaluation and how to deal with multiple goals and objectives—including those of the organization, the staff, and the client. All references and pedagogical features including learning objectives, key terms, and questions for discussion have been updated.

1
What is Evalution?

---

Scenario One

The administration at Grandview Retirement and Nursing Facility strongly promotes ongoing education and training for all staff. In fact, when new personnel are hired, they agree to take two courses each year, selected from a long list of possible courses decided upon by administration and staff. Topics are many and diverse, from the law and patient rights to medication in geriatric care to the mind-body connection. Two of the recent training programs were devoted to answering these questions: (1) Is there a connection between mental and physical activity and sustained good health and longevity? (2) How do you encourage residents to stay both mentally and physically active? Program and nursing staff alike found the answer to the first question compelling: activity is indeed linked to well-being and longevity (Wells, 1997). They decided to institute additional programs for residents that would involve mental and physical activity.

Coincidentally, a volunteer had been coming to the retirement and nursing facility for about six months to work with interested residents in an informal gardening program. The volunteer, Ruth, discovered that one of the things the residents missed most when they left their homes was their gardening. The activity helped them regain what they had enjoyed and gave them a real opportunity for mental and physical engagement. The facility had a limited budget, so Ruth made her own in-kind donations.

From the beginning, however, the activity was extremely popular and she knew that available resources would not be sufficient to serve all the people who wanted to take part in the program. Ruth volunteered to look for funds, a grant that would allow Grandview to continue and enlarge the program. The administrators said they would be delighted if she would do the legwork, but of course they would need to know exactly what she was doing along the way so that they could keep their board of directors apprised.

A short while later Ruth discovered that the Beed Foundation funded this kind of program and she got their grant application. One of the first items she saw on the application asked for an evaluation of the program. The items in the evaluation question included listing program objectives (for example, to increase residents’ activity and mobility level) and listing measurements (both quantitative and qualitative) that would be used as indicators of achievement of those objectives. Ruth later learned that a quantitative measure might be the number of times residents took part in the gardening activities before the onset of the funded program compared to how many at the end of the program. A qualitative measure might be perceptions of garden activity staff regarding the focus and mood of residents.

The grant also called for explaining data collection methods, a discussion of any sampling that would be done, a description of the evaluation design, the data analysis that would occur, the staffing, and the final report.

Although she was puzzled about most of the items, Ruth hoped that subsequent meetings and discussions with the staff would answer her questions. She needed some basic questions answered first: Why do programs need to be evaluated? What is evaluation? What are you looking for when you evaluate a program? How is this different from research?

---

Unlike Ruth, readers of this book will not have to wait to have these questions answered. You just need to read Chapter One. After reading this chapter, you should be able to answer the following questions as they relate to the preceding scenario as well as to the chapter material:

1. What do all evaluations have in common?
2. How would you characterize the differences in the efficiency, effectiveness, and impact of a program?
3. Why evaluate in the first place?

Common Denominators

Before we embark on a definition of evaluation, we need to answer the volunteer's question. How does program evaluation differ from research? Unlike evaluation, research takes place in a precisely controlled environment. The Grandview gardening program could not comply with this definition. Among a number of other important reasons that we will discuss in Chapters Five and Eight, Grandview's project staff could not precisely control—nor would they want to—the number and choice of participants. In addition, research collects data to stretch the envelope of what is known in order to prove or disprove a hypothesis or presupposition. Evaluation looks at program and project objectives and asks whether they have been achieved, judges the worth of ongoing programs, decides upon the usefulness of new programs or projects, and so forth (Rossi and Freeman, 1993).

In both for-profit and nonprofit organizations, managers possess data (information) that could help to evaluate a program or project. These data are the one thing that all evaluations have in common regardless of the particular definition of evaluation one embraces: evaluation is the systematic process of collecting data that help identify the strengths and weaknesses of a program or project. The data may be as simple as records of attendance at training sessions or as complex as test scores showing the impact of a new educational program on increasing students’ knowledge across an entire school system.

Whatever definition you apply to evaluation, when your task is to perform a program evaluation you will almost certainly include a number of common evaluation steps. You will find these listed as program cycles in the Putting It All Together section later in this chapter.

Two Definitions of Evaluation

People do not always agree on one definition of evaluation. Following are two different definitions:

• Evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.
• Evaluation is the systematic process of collecting and analyzing data in order to make a decision.

Notice that the first ten words in each definition are the same. However, the reasons—the why—for collecting and analyzing the data reflect a notable difference in the philosophies behind each definition. The first reflects a philosophy that as an evaluator, you are interested in knowing only if something worked, that is, whether it was effective in doing what it was supposed to do. The second statement reflects the philosophy that evaluation makes claims on the value of something in relation to the overall operation of a program, project, or event. Indeed, many experts agree that an evaluation should not only assess program results but also identify ways to improve the program evaluated (Wholey, Hatry, and Newcomer, 1994). A program may be effective but of limited value to the client or sponsor. One can imagine, however, using an evaluation to make a decision (the second definition) even if a program has reached its objectives (the first definition). Federal grants are based on the first statement, that is, whether the program has achieved its objectives, but the harder decision to downsize or change may be a consequence of the second definition of evaluation.

Evaluating Efficiency, Effectiveness, and Impact

We can define evaluation even more precisely as a process that is guided by the reason for doing the evaluation in the first place. An evaluation might be a process of examining a training program in light of values or standards for the purpose of making certain decisions about the efficiency, effectiveness, or impact of the program. To carry out this task, you need to understand the concepts of efficiency, effectiveness, and impact. These three terms will be referred to from this point on as the levels of program evaluation. (See Table 1.1 later in this chapter.)

Table 1.1 Questions of Efficiency, Effectivene...

Table of contents

• Cover
• Title Page
• Copyright
• Preface
• The Authors
• Chapter 1: What is Evalution?
• Chapter 2: Why Evaluate?
• Chapter 3: Decision Making: Whom to Involve, How, and Why?
• Chapter 4: Starting Point: The Evaluator's Program Description
• Chapter 5: Choosing an Evaluation Model
• Chapter 6: Data Sources
• Chapter 7: Data Analysis
• Chapter 8: Is It Evaluation or Is It Research?
• Chapter 9: Writing the Evaluation Report
• Chapter 10: Evaluation as a Business
• A: More On Data Analysis
• B: Sample Evaluation Report
• C: Sample Evaluation Contract
• References
• Index
• End User License Agreement