Evaluation

A Cultural Systems Approach

Mary Odell Butler

• Publisher: Routledge
• Year: 2016
• Print ISBN: 9781629580821
• eBook ISBN: 9781315428871
• Topic: Social Sciences
• Subtopic: Anthropology
• Index: Social Sciences

In an era of budgetary belt-tightening, policymakers must prove that their programs work or face drastic cuts in spending. This book, informed by the author's many years of practice in program evaluation and expertise as an anthropologist, discusses in plain prose the theory and methods of culturally-competent evaluation across a number of disciplines, such as health and education, for graduate and advanced undergraduate students and professionals. The book-guides readers through the process of evaluation in complex contexts created by cultural change, the movement of populations, economic forces and constantly emerging crises;-introduces rich ethnographic theory and methods developed by anthropologists to evaluators in other fields;-teaches anthropologists and other social scientists research techniques developed in such fields as business or public-policy evaluation;-provides a strategy for building evidence from both qualitative and quantitative sources to form conclusions that have scientific credibility.

CHAPTER ONE INTRODUCTION

This book is about how to link evaluation and anthropology into a dynamic and flexible approach that brings out the vitally human nature of the ways people organize themselves to accomplish their goals. I present an approach to evaluation that merges anthropology and evaluation to make something new, an approach to valuing from the perspective of both the insiders who generate the things being evaluated and the outsiders who will use the evaluation. Evaluation anthropology, alone or in combination with other approaches, can generate explanations of human activities in ways that are both rigorous in their ability to explain and adaptable to changing circumstances. As we will see in the chapters that follow, evaluation is a large field and one requiring many different kinds of abilities.

Evaluation teams go far beyond anthropology. But anthropology has something important to contribute. By using anthropology, evaluators can tease out the multiple perceptions of the thing being evaluated—the evaluand—from the various kinds of people who are influenced by it, untangle the complex dynamics of communities in which programs are embedded, qualify the contextual influences coming from outside of programs yet affecting them often profoundly, and integrate large amounts of information to build descriptions of both individual implementations and widespread programs.

So what are we talking about here? What is it that can help us map diversity on diversity, relationship on relationship, in such a way that we get a three-dimensional view of what kinds of processes generate modern life? Capturing and translating the “inside” of human communities to the “outsiders” who observe them is the time-honored task of the ethnographer (Harris 1968; Stake 2006).¹ Ethnography too must be re-interpreted and disciplined to meet the scientific demands of evaluation. This volume will explore how the two disciplines—anthropology and evaluation—can move together to help us see how people organize to get the jobs of living done in the complex systems in which all of us must function.

I will argue here that anthropology, as refined and used in this century, enables us to layer reality upon reality in human activity in such a way that the evaluator can assemble what can be known and map program activities onto it. In this way the value of a program in the natural community can be inferred. Evaluation adds the perspectives, the methods, and the science needed to do this in a way that is credible to those who rely on evaluation to make decisions.

WHY EVALUATION?

I chose to write about evaluation because evaluation is what I have done with myself for the past twenty-five years. Evaluation is a scientific endeavor conducted for the purpose of describing the worth, value, and/or effectiveness of some activity directed to serving a human need or solving a human problem. Evaluations may be focused on programs, projects, products, media campaigns, curricula—almost anything that is delivered to communities of humans. I will focus on program evaluation in this volume because it is what I know best. However, the same approach can and has been used to evaluate projects, policies, and products.

One of the most important things about evaluation, however it is done, is the need for clear thinking and methodological rigor to protect the credibility of results. Decisions about policies, careers, and funding often rest on evaluation results. Evaluations can protect the public good while alerting authorities to things that aren’t working as they should. The scientific demands of evaluation require that anthropologists incorporate into their methods the safeguards needed to protect their results’ validity, reliability, and credibility.

I did not find evaluation; evaluation came to me because some people knew I could do ethnography and believed that ethnography would be helpful to their efforts to understand programs. In the early years I don’t think my clients cared whether I was an anthropologist—although they all said it must be interesting! Instead, they wanted program evaluation and knew that they couldn’t understand how a program worked unless they could ask the people involved with it. They were pretty sure I could do that.

I knew nothing about evaluation when I started twenty-five years ago. Fortunately my colleagues were better informed. I quickly learned that evaluation has a body of method and theory of its own, not deducible from any other discipline. To become an evaluator, I had to take courses, read the literature, and attend and participate in professional meetings. But once I discovered the field, evaluation became the arena in which I do anthropology. I was impressed with the rigor of evaluation and its flexibility as a way of understanding why the things that people try to do—programs, projects, product development—succeed or fail. I discovered that people doing programs can and will explain to me in great detail what they are trying to do if I ask the right questions and listen carefully to what they are telling me. It was a familiar set of skills, one I had learned as I became an anthropologist.

I used to say to anyone who would listen that “I landed on my feet as an anthropologist when I began doing program evaluation.” Evaluation is a wonderful career choice for anthropologists. In this era, when the traditional ways of doing ethnography become less accessible to us, evaluation provides a strategy to investigate human life that is connected to our own time, deepens our understanding of what people are about, and produces something useful. I have tried to teach anthropologists to be evaluators for the past fifteen years. Recently I have reversed the arrow and begun to teach ethnography to evaluators. As teaching so often does, this has helped me see evaluation from many different sides. Evaluation—all evaluation—is very applied research, although some people distinguish between evaluation (doing it) and evaluation research (developing and testing ways to do it.)² Evaluation is defined by usefulness to a client. It is oriented to action and generates knowledge for decision making.

Thus, evaluation is a scientific study of something—an actual or planned program, process, or product—to determine whether it can or does achieve what it sets out to do. The goal of evaluation is to assess something. This can be contrasted with studies in which the purpose of the research is, for example, to test a theory. People normally do not pay for evaluations because they are curious about a theory; they commission evaluations because they need to know how to do something for some purpose or reason. They need to plan a program, discover how people relate to a problem, and determine whether what they are doing is working.

The research problem is developed by working with users and/or stake-holders to understand and articulate what they need to know. Stakeholders are people with some kind of interest or “stake” in the outcome of the evaluation. Stakeholders can have very diverse interests in the outcome of the evaluation. For example, I once evaluated a program to support families in which the project officer (in this case, the “client”) hoped the program would fail because staff disagreed with the political agenda implicit in the program’s design. Of course, the client didn’t tell us that at the first meeting. It took time, skilled interviewing, and several meetings before this information emerged. In evaluation, as in ethnography, good research depends on building trust with stakeholders.

The product of evaluation may be knowledge or recommendations for action or both. Sometimes what clients want is the summary and conclusions only. Sometimes they want the evaluation to go beyond that to build recommendations on how they should proceed given the results of the evaluation.

Who are evaluators? Right now an evaluator is someone who evaluates. Evaluators are defined by their use of evaluation methods and theories and by their participation in an evaluation community. Obviously there are better and worse evaluators based on training and experience. The American Evaluation Association debates the issue of building credentials for those who can call themselves an evaluator, but there is no formal credentialing at the present time. Regardless of the issue of credentials, evaluation has its own body of method and theory. You will not succeed as an evaluator unless you learn evaluation. You will need to take courses, read up, go to evaluation meetings, and get smart. I will try to help you do that in this book. However, this will be the merest introduction. Throughout your career you will continue to build your knowledge and understanding of evaluation.

Evaluation is multidisciplinary. Evaluation of programs and projects originated in educational psychology in the 1930s, but it is now done by people with many kinds of scientific preparation. It is very often done in teams with people from various backgrounds. Anthropologists fit well into these teams because of our holistic perspective and tendency to systems thinking. To function effectively on teams, however, we must commit to lifelong learning. It is wise to acquire enough understanding of the various skills that may be called on in evaluation designs. I would strongly recommend, for example, a working knowledge of statistics and scientific design from a general social sciences perspective. Learn the meaning of things like experimental and quasi-experimental designs, validity, and reliability that come up constantly in evaluation. It is very difficult to work with your teams if you can’t think in these terms.

Who buys evaluations? People who make decisions and administer efforts to meet human needs and solve human problems make up most of the clients for program evaluations. For all evaluations within the scope of this book, the viewpoints of human beings are central to collecting and using information from evaluations. Clients are the funders and supervisors of evaluations. They define the scope of the evaluation and the information the evaluation should provide. Much success in evaluation comes from a cumulative understanding of your clients and their needs. A nonexhaustive list includes private business, government agencies, international development organizations, nonprofit agencies, nongovernmental organizations (NGOs), health care organizations, and educational organizations. You will probably discover other examples. Private businesses often do product evaluation. These seek to find out how people interact with products like computer systems and drugs. For example, Susan Squires conducted several projects assessing the needs of consumers for communications technology in the global market (Squires 2005).

Some evaluators look at programs—that is, organized sets of activities that are trying to achieve some specific benefits in a defined group of people.³ Government is probably the biggest consumer of program evaluations. Government agencies are often required to be accountable for the programs they are mandated to deliver. Legislative bodies assess programs for compliance with goals of legislation. Government evaluations can be domestic or international. Sometimes they are completed by agencies themselves; other times government agencies develop contracts with outside companies to complete evaluations. Charity Goodman and her colleagues at the US Government Accountability Agency provide an example of an evaluation done by a government agency. In this case they incorporated ethnography into GAO’s investigative audit techniques to understand clinicians’ adherence to patient safety measures in Veterans Administration hospitals (Goodman, Trainor, and Divorksi 2005).

If you have your heart set on a career in an international arena, you can do this easily as an evaluator. Many of us work for agencies, NGOs, foundations, and universities that are building and evaluating programs and projects to alleviate social, health, and economic problems all over the world. International development organizations of various kinds commission or perform evaluations. For example, USAID does evaluation using their own specialized set of methods. If you plan to pursue a career in international development organizations, you are well advised to look for a job—at least your first job—with an organization or a company that has an established track record in doing this so that you can become familiar with this evaluation paradigm. It is extremely difficult for companies to break into the USAID arena for the first time. Be skeptical if a potential employer tells you in a job interview that their company plans to get into international development work—you may not live long enough for this to happen!

Health care organizations—hospitals, managed care organizations, and other health care providers—conduct evaluations to see whether people are using services the way they should or to ascertain the needs in the community for services. With changes due to the Affordable Care Act, the demands for these kinds of evaluations can be expected to grow in the immediate future. Large health organizations often have internal evaluators on staff. This is a good job for an anthropologist because health care is a culturally sensitive domain requiring community ethnography.

Educational organizations—school systems and universities—evaluate curricula, educational programs, and special programs for designated population subgroups. Like health care evaluations, these are often directed to community-based programs. Also, large educational organizations and school districts often have their own evaluators. As we’ll see, evaluation originated in educational settings and is still very important in education.⁴

WHY ANTHROPOLOGY?

I have chosen to bring in anthropology because I am an anthropologist by both training and inclination. Anthropology is, of course, many different things. There are three anthropological ideas that I have built this volume around: the use of culture to orient ourselves to human life, the method-theory nexus that is ethnography, and the concept of the community as the arena in which people do things. Many of the tools we have built as anthropologists can make important contributions to the field of evaluation. The methods and skills evaluators use are important to anthropology as well. It is important to note that many of the concepts used here are in no way unique to anthropology. Ethnographic methods, for example, are part of the tool kit of other disciplines. The fact that they are not unique to anthropologists in no way diminishes the value of an anthropological perspective for evaluators. They are important to the way we present ourselves as evaluation anthropologists. If we present ourselves to potential employers and clients as anthropologists who evaluate, we enhance our value in the workplace for ourselves and those who come after us.

Some of the perspectives that anthropologists can bring to evaluation are:

• The ethnographic method—critical to understanding human activity in the world—is the core of what cultural anthropologists do. Ethnography supports us as we solicit the diverse perspectives of stakeholders and bring them to our thinking about evaluation design. And it is often a method for structuring data collection and data analysis.

• Systemic/holistic thinking—We are trained to see wholes and the fit of parts into them. Whole human experiences are our subject matter. It always surprises me that this is not intuitive to some people from other disciplines who are trained to understand things in terms of economics or epidemiology or genetics. My career as an evaluator has brought home to me over and over again the variety and legitimacy of different perspectives on the world. I learned this, of course, as an anthropologist. But I understand it from my experience as an evaluator working on interdisciplinary teams.

• Participatory philosophy—For traditional anthropological fieldwork the first step is building rapport with people in the field site and bringing them into the research as soon as is feasible. This trust-building process is equally important in evaluation. It also suits us to conduct participatory evaluations that we often find intuitive. We are comfortable with participation. From evaluation I learned the critical need not to over-identify with participants because to do so risks bias.

• Relativism and the ability to withhold judgment—Because we seek to work with culture, we must detach ourselves from judgments about what may be very strange or even practices we consider immoral. Female infanticide springs to mind. The very idea of killing infants because they are a less preferred gender is appalling to most people coming from a European and American background. But it has its own logic to those who practice it. As anthropologists, we learn to listen with suspended judgment to what people tell us. This allows us to see unexpected things rather than screening them out.

• Understanding of language and the workings of symbol systems—Language is the major way in which culture is expressed; however, people don’t usually say what they mean and mean what they say. An understanding of the arbitrary linkage of symbols to the symbolized helps us distinguish the meanings of utterances from speech habits and differences in word usage.

I have observed that when people come looking for an anthropologist to help with an evaluation, they are looking for someone to help them figure out what questions are important to one or more constituencies they serve. This usually involves, in whole or in part, the use of ethnography, the core methodological and theoretical concept in which much of anthropology is embedded. The ethnography that anthropologists do is part of a worldview that is not unique to anthropology but remains the philosophical stance in which anthropologists are grounded.

Anthropology—at least wearing its sociocultural hat—is the study of human life as seen through the concept of ...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Acknowledgments
• CHAPTER ONE Introduction
• CHAPTER TWO Evaluation: An Orientation for Anthropologists
• CHAPTER THREE Anthropology: An Orientation for Evaluators
• CHAPTER FOUR Putting Them Together: Evaluation Anthropology
• CHAPTER FIVE Ethics, Clearances, and Ethical Decisions
• CHAPTER SIX Method in Evaluation Anthropology
• CHAPTER SEVEN Jobs and Career Planning
• CHAPTER EIGHT Future Pathways
• Appendix A: Glossary of Terms
• Appendix B: The Business Side
• Notes
• References
• Index
• About the Author