Evaluation Methodologies for Aid in Conflict

Ole Winckler Andersen, Beate Bull, Megan Kennedy-Chouane

• Publisher: Taylor & Francis
• Year: 2013
• Print ISBN: 9780415870801
• eBook ISBN: 9781136027284
• Edition: 1
• Topic: Economics
• Subtopic: Development Economics
• Index: Economics

Knowledge and rigorous evidence around the role of external development partners in situations of conflict and fragility is still lacking. There is little accountability for the billions in aid being spent in places like Afghanistan, Iraq and the Democratic Republic of Congo.

This book analyses evaluation theory and practice in order to help fill this knowledge gap and advocates a realistic and rigorous approach to evaluating international engagement. Through a series of case studies, this book highlights both the promise, and potential pitfalls, of taking a more evaluative approach to understanding aid in conflict regions. These illustrate the methodological and analytical approach taken by researchers working to understand the results and effectiveness of conflict prevention and peacebuilding support. While well-grounded in current theoretical and methodological debates, the book provides valuable practical information by examining how and why different choices were made in the context of each evaluation. The book shows what future steps may be envisaged to further strengthen evaluations of support for conflict prevention and peacebuilding. The analysis draws on a wealth of perspectives and voices to provide researchers and students in development studies and conflict and peace studies as well as development evaluators with a deep and broad understanding of evaluation methods and approaches.

1 Introduction

Ole Winckler Andersen

Danish Ministry of Foreign Affairs ¹

Megan Kennedy-Chouane

Development Co-operation Directorate, OECD ²

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know.

United States Secretary of Defense, Donald Rumsfeld, 12 February 2002

1 A primer

International interventions in foreign conflicts, both military and humanitarian, are hotly debated. Hundreds of millions of dollars are being spent each year to try to support recovery from conflict, peace, stability and economic development in places such as Afghanistan, the Democratic Republic of Congo and South Sudan. Yet many of the strategies, policies and programmes currently pursued are based on weak – or no – evidence that they actually help. In most cases, international partners have only a vague understanding of how their aid for conflict prevention and peacebuilding is actually affecting local populations and the long-term prospects of peace. And not knowing matters a lot in these settings because the consequences of making the wrong decisions, based on weak or unreliable evidence, are potentially so dramatic. That is why this book focuses on dissecting the evaluation process itself to explore how we know, and whether we actually know, what we think we know.

The book examines the methodological and analytical approaches taken by evaluators working to understand the results of international conflict prevent and peacebuilding support. These analyses highlight the real-life trade-offs but also the importance of maintaining methodological rigour while adapting evaluation approaches to conflict settings. In so doing, the authors point the way towards potential avenues to help strengthen the knowledge base through better evaluations.

2 Need for learning

The aid to conflict prevention and peacebuilding has increased significantly in the past decade. US$46.7 billion in net official development assistance (ODA) went to 45 fragile states in 2011,³ or roughly 37 per cent of all ODA.⁴ In spite of an acknowledged general professionalisation of these interventions, lack of progress in fragile states is widespread and a number of studies have pointed to the need for more learning and systematic evaluation of aid interventions in fragile states.

Across the globe, some 1.5 billion people live in countries affected by persistent political and criminal violence and weak or dysfunctional state institutions where life is not improving (World Bank 2011).⁵ While progress is being made to reduce poverty and achieve common development goals, like the Millennium Development Goals (MDGs), people living in fragile states cannot expect their countries to achieve the MDGs by 2015. As has been documented in several analyses, the fragile states score significantly lower than other developing countries on most social indicators (OECD 2012b). The slow progress on social and economic indicators (e.g., poverty reduction, infant mortality, hunger) has led to a greater emphasis on creating an enabling environment for development by giving more attention to peacebuilding, Statebuilding and armed violence prevention (OECD 2012b). In addition to the lack of progress, in spite of the high levels of aid, these countries often slip back into conflict.

The learning and accountability gaps all too common in development cooperation are multiplied and deepened in these settings.⁶ High risk, fluidity and complexity make the political and operational realities of these settings particularly difficult for external development partners. Simultaneously there is strong political demand for analyses of ‘what works’ and ‘what doesn't’ in these contexts. Some lessons learned do exist, and experience has shown that aid can support progress towards positive change. For this role to be realised, however, external assistance needs to be targeted towards peacebuilding and Statebuilding goals, its allocation optimised across countries, and its quality improved.⁷

It is widely accepted that approaches to development work and evaluations have to be adapted to settings of violent conflict and state fragility (Anderson et al. 2007; OECD 2012a). However, donors have only partly adjusted their aid to reflect this.⁸ In addition, development agencies often find themselves operating in ‘emergency mode’, with high staff turnover and unrealistic spending targets. Relations with national partners can be extremely challenging, due to low capacities, lack of legitimate, accountable institutions or political instability. As a result, systematic learning, analysis and accountability for results have been weaker in these contexts, compared to other areas of development cooperation. This reflects the challenging context, but also symptomatic of development agencies’ relative weaknesses in effectively tackling problems that require long-term, multifaceted and flexible solutions.

The increasing interest in providing support to these countries has created broader interest in understanding the mechanics of peace. The perception of failure – countries sliding back into conflict after a period of stability or stagnating in long-term crisis – has also motivated a deeper investigation of the role of external actors and a nuancing of the conflict continuum, moving away from rigid definitions of conflict/post-conflict periods. The pressure inside development agencies and international organisations has resulted in a number of analyses, which try to understand how development in these contexts can be understood and influenced and, in particular, in recent years there has been a demand for more evaluations of interventions in these contexts.

3 Fragile settings and types of interventions: implications for evaluation

Evaluation is broadly understood as the systematic and objective assessment of a programme or policy, its design, implementation and results (OECD 2002). Definitions of what should be understood with ‘evaluation’ in a conflict prevention and peacebuilding context and how this differs from evaluation in other fields, has been debated in the literature.⁹ In particular, how core evaluation criteria (relevance, sustainability, effectiveness, impact and efficiency) may be understood in the context of conflict and fragility has been discussed.¹⁰

Two main issues complicate the development of robust evaluations of interventions in the field of conflict prevention and peacebuilding: the context of fragility itself and the types of interventions being undertaken by international actors in these settings. These issues make it difficult for those involved in an evaluation to reach agreement on what is to be analysed and how. The diversity across so-called fragile contexts also has major implications for both the external and internal validity of evaluation findings, as will be discussed further in Chapter 2.

First, as indicated earlier, the fragile states and contexts vary significantly and there is an ongoing discussion on how to define fragility.¹¹ The definitional challenge is compounded by the fact that fragile states terminology has been much maligned as stigmatising and analytically imprecise. The term ‘fragile’ is usually considered pejorative and some view it as an ‘inherently political label reflecting Weberian ideals of how a "successful" state should function… [which] arguably does not adequately differentiate between the unique economic and sociopolitical dimensions of states’ (GSDRC 2012). Fragility may therefore be viewed as a continuum of state performance, rather than an either or category. Many instead refer to ‘situations of fragility’ to account for this nuance and highlight issues of social and political fragility and conflict that may extend beyond the state apparatus. The group of 47 countries in situations of crisis and acute fragility includes a wide range of countries from essentially ‘post-conflict’ situations in Liberia, Nepal and Sri Lanka, to war and protracted violent conflict in Somalia, Iraq and Afghanistan. Other countries are mainly characterised by low capacities of the state and its inability or unwillingness to protect basic human rights and provide core services to its people. Still others, including South Sudan and Palestine, are characterised by a focus on state formation in the context of ongoing violent conflict. Regional and cross-border conflicts characterise other conflicts, particularly central Africa and the Great Lakes region.

The notion of peacebuilding is also debated.¹² Peacebuilding is used to refer both to the process of supporting stabilisation and the reduction of violence, as well as the longer-term transformation towards resilient cultures capable of resolving disputes without violence, meeting preconditions for sustainable development. Two groups of definitions of peacebuüding therefore exist. A broader definition, which seems to be close to Statebuilding, and a narrower one, which focuses on stopping violence. This second definition is close to a definition of conflict prevention. The first definition will have a main focus on society/institutional (macro or impact) level while the second will have its focus on the more immediate outcome (micro) level.¹³ In addition, conflict prevention and peacebuüding cover a wide range of different types of interventions, such as election support, support to parliament, good governance, demobilisation of ex-combatants, etc. Many publications (and analyses) no longer make a distinction between the three areas (conflict prevention, peacebuüding and Statebuilding).

The focus of this book being on evaluation methods, we will not go further into these definitional debates. Most chapters will use the definition that was provided in the evaluation's Terms of Reference, or the author's own definition. For readers that are new to this field, the Organisation for Economic Co-operation and Development (OECD) working definition provides a useful place to start (OECD 2011a, 2012a). This definition covers activities designed to prevent conflict through addressing structural and proximate causes of violence...

Table of contents

• Cover
• Half Title
• Routledge Explorations in Development Studies
• Title Page
• Copyright Page
• Table of Contents
• List of Illustrations
• List of Contributors
• Foreword
• 1 Introduction
• 2 Evaluative knowledge: considerations on methodology
• 3 Critical reflections on the South Sudan evaluation of conflict prevention and peacebuilding activities
• 4 Battlefields of method: evaluating Norwegian peace efforts in Sri Lanka
• 5 The case of Congo: an evaluation approach focusing on context
• 6 Assessing development cooperation in northeast Afghanistan with repeated mixed-method surveys
• 7 Impact evaluation for peacebuilding: challenging preconceptions
• 8 Evaluating Statebuilding support: learning from experience or judging from assumptions?
• 9 Systems thinking in peacebuilding evaluations: applications in Ghana, Guinea-Bissau and Kosovo
• Index