Introduction
Interpretive and ethnographic approaches have long been used in research to monitor and evaluate programmes in social development and health. The successful use of these approaches has been documented in relation to schooling and community (Dorr-Breme 1985, Erickson 1977), drug and substance use (Singer 1993, Singer et al. 1995, Hong et al. 2005, Needle et al. 2008, McNeil et al. 2015), rural livelihoods (Niehof and Price 2008, Siyoum et al. 2012, Harrison 2015), peacebuilding and justice (Miller and Rudnick 2010, 2012, Mieth 2013), adolescent marriage (White 2015), and the delivery of a range of health care services (Diamond 1986, Bentley et al. 1988, Shawyer et al. 1996, Price and Pokharel 2006, Savage 2006, Jain and Jadhav 2009, Brewer and Sparkes 2011, Millard and Ladia 2015), among other topics. In recent times, the fields of HIV and sexual health have perhaps benefitted the most from the application of interpretive and ethnographic approaches to programme monitoring and evaluation across a diverse range of social and cultural global settings (e.g. Clatts 1989, Schensul and Weeks 1989, Ingstad 1990, Schensul and Schensul 1990, Carrier and Magana 1991, Farmer 1992, Green 1992a, 1992b, Singer 1994, Obot et al. 1997, Waterston 1997, Parker 2001, Price and Hawkins 2002, Green et al. 2009, Leclerc-Madlala 2009, Mignone et al. 2009, Turkon et al. 2009, Simmons 2011, Bell and Aggleton 2012).
Despite this activity, interpretive and ethnographic approaches are side-lined in much contemporary evaluation work, and current monitoring and evaluation practice remains heavily influenced by more positivist approaches (Eyben 2010, 2013, Natsios 2010, Bell and Aggleton 2012, Harrison 2015, White 2015, Winther 2015). These not infrequently seek to specify in advance what will be measured and how predetermined outcomes will be brought about. This style of work, while popular, is problematic in several respects, not only because of cost (the analytic frameworks advocated for by development agencies often suggest that expensive experimental and quasi-experimental attribution analyses are required to identify whether a programme has had its intended effect), but also because it often occurs at the expense of sensitive, in-depth qualitative research documenting community perspectives on impact and change (Bell and Aggleton 2012).
The emphasis given to results-based programme reporting also means that monitoring and evaluation are usually undertaken for upward accountability purposes and to demonstrate success (Batliwala and Pittman 2010), rather than to learn how change happens, what went wrong, what works within a particular context, and how best to refine programmatic approaches. The tendency to monitor and evaluate health programmes using a âlogical frameworkâ (or log frame) or a âtheory of changeâ can flatten complex processes of growth, development and change into overly simple linear models of cause and effect (Batliwala and Pittman 2010, Grove and Zwi 2008). Within these models, programme impact and success are often measured against external programme goals, rather than against the more local changes that have occurred in peopleâs lives.
This book emerges from a growing sense of concern about the challenges faced in knowing how best to intervene to improve health and social development. The current preference for outsider-led, top-down quantitative studies tends to be driven by concerns that are external to the affected communities. All too often, local people are involved in programmes as âparticipantsâ or âbeneficiariesâ, rather than as informed, local experts capable of ensuring that programmes are relevant and achieve the desired results. Responding to these concerns, this book seeks to counter the marginalisation of qualitative perspectives resulting from the prevailing âmethodological triumphalismâ (Barrett and Carter 2010: 516), which assumes that quantitative approaches to evaluation are superior, more robust and more scientifically valid. Chapters seek to illustrate the potential of interpretive and ethnographic methods to improve understanding and make a difference to communities on the ground. Through a focus on individual and community perspectives and locally grounded explanation, we aim to offer a richer way to assess the relationship between intent, action and change in health and social development.
Current trends in monitoring and evaluation
Monitoring and evaluation serves a range of purposes including learning what works, refining and redesigning programme approaches, and attributing cause and effect. When implemented effectively, it involves reporting downwards to local communities and upwards to development agencies and funders for the purpose of accountability and credibility (Batliwala and Pittman 2010, OâFlynn 2010, Bell and Aggleton 2012).
Most often the focus is on measuring the âimpactâ of specific programmes and interventions. However, as is explained elsewhere (Bell and Aggleton 2012: 796â798), confusion abounds concerning two rather different understandings of what impact evaluation might involve (White 2009). The first approach prioritises the use of studies that aim to attribute changes in selected outcomes to a specific intervention using âscientifically validâ designs. These studies should more accurately be called âattribution analysesâ (White 2009). Proponents of this approach argue that carefully designed experimental and quasi-experimental research can determine whether (and to some degree why) an intervention had its intended effect.
A number of concerns can be raised about this way of working (Bell and Aggleton 2012: 797). First, while attribution analyses have their place, attribution is not the only purpose of impact evaluation, and there are important ethical concerns relating to the use of âcontrol groupsâ as counterfactuals who do not benefit from programmes or interventions. Second, these approaches are costly and often beyond the budgets of national and local programmes in developing countries. Third, the methods used tend to rely on a level of expertise that excludes important local and professional programme stakeholders, including community members, from being involved in the evaluation. Finally, efforts to understand context and engage with community members in order to develop a theory of change to guide programme design, implementation and quantitative evaluation tend to be weak, instead relying on the expertise of programme managers and/or external evaluators.
A second approach to impact evaluation advocates that monitoring and evaluation research should centre on the programme implementation cycle. This type of research attempts to understand whether expected programme outcomes have been achieved, as well as their resulting long-term effects. It aims to evaluate success as an integral dimension of programme design and implementation, answering questions such as âwhat should we do?â, âare we on track?â and âdid it work, and how?â (Collumbien et al. 2006: 155). Typically, information is gathered in line with a monitoring and evaluation framework â an integrated system blending monitoring research with evaluation research that collects experiences and information during and after programme delivery â to assess programme delivery, progress and impact. This tends to involve four rather different things: (i) needs assessment and insight research to inform programme planning, (ii) the monitoring of programme delivery, (iii) impact assessment and evaluation, and (iv) the analysis of the information collected to document best practice.
Monitoring, or process evaluation, is an ongoing programme management activity that assesses the implementation of programme activities and progress towards meeting target programme outcomes. Ideally, it should take place periodically and systematically throughout the programme cycle to determine whether work is proceeding as planned. In doing this, it generates information that enables managers to make improvements or put programmes back on track (Batliwala and Pittman 2010, Collumbien et al. 2006). Evaluation research, on the other hand, is designed to assess the overall impact of a programme, typically against an explicit set of predetermined goals and objectives. It involves the systematic collection and analysis of data to help discover if, how and why a particular intervention or set of interventions worked (Batliwala and Pittman 2010). It usually takes place less often than monitoring, is more comprehensive in character, and tries to capture impact at particular moments in time. Evaluation research can take several different forms. For example, summative evaluation may occur at the end of a given programme cycle and focuses in on programme outcomes, strengths and weaknesses (Batliwala and Pittman 2010). In contrast, impact evaluations offer an analysis of lasting or significant change â positive or negative, intended or not â in peopleâs lives brought about by an action or a series of actions (OâFlynn, 2010).
A recent review (Batliwala and Pittman 2010) highlights a number of weaknesses in current evaluation practice. For example, the log frame approach heavily promoted by some funding agencies frequently flattens complex change processes into overly simple causes and effects. It prioritises attention to programme goals or a theory of change, rather than the changes taking place in ordinary peopleâs lives. Likewise, too keen an interest in individual behaviour change (again, a much loved focus of health and development work) can blind us to the influence of social, cultural, economic and political factors in bringing about change. Finally, too close a focus on what it is that a programme seeks to achieve can cause us to ignore or misrecognise negative change, reversals and unexpected outcomes, and the longer-term changes that may take place beyond a programme itself.
Lack of familiarity with qualitative approaches to evaluation by programme staff and donor agencies also influences the preference for quantitative methods in monitoring and evaluation work (Bell and Aggleton 2012). At donor level, this tendency is compounded by a lack of space to talk about alternatives to conventional monitoring and evaluation approaches, and the benefits of using evaluation as a learning device rather than as an accountability tool. Chapters in this book provide the opportunity for practitioners, donors and policy makers to learn about the benefits of using these kinds of strategies.
Interpretive and ethnographic approaches
Some forty years ago, the social sciences witnessed a major epistemological shift â from hitherto predominantly positivist ways of understanding social life to more interpretive approaches. Often referred to as âthe interpretive turn in social scienceâ (e.g. Geertz 1973, Rabinow and Sullivan 1979), researchers sought to modify the methodological logic underpinning social enquiry. Methodologically, positivist research tends to be hypothetico-deductive in character. Hypotheses are created from existing theory in advance of data collection, and research is conducted to test whether or not these âhunchesâ are supported by the data. This type of enquiry tends to employ quantitative methods, using either comparati...
