Speaking Justice to Power

Ethical and Methodological Challenges for Evaluators

Kim Forss, Mita Marra

• Publisher: Routledge
• Year: 2017
• Print ISBN: 9781138514959
• eBook ISBN: 9781351488495
• Edition: 1
• Topic: Politics & International Relations
• Subtopic: Politics
• Index: Politics & International Relations

Efficiency, economy, and equity are policy goals pursued by governments around the world, but analysts and evaluators have devoted more effort to measuring and evaluating the first two. In Speaking Justice to Power, contributors examine the concept of equity, the role it plays, and its application in policy evaluation. Here some of the most valuable thinkers in the area of policy studies address key questions: How should evaluators develop criteria for measuring equity as they analyze both program and policy implementation as well as their impacts? What distinctions among people should be taken into account when measuring and valuing impacts? What sorts of data should be used to analyze processes and impacts in different settings? How might such data be validated? The contributors employ grounded-theory thinking as they translate key ethical principles into their work and draw important lessons from their experiences. The work discusses equity in interventions addressing a variety of social and environmental problems. This volume continues the fine tradition of Transaction's Comparative Policy Evaluation series.

1

---

Introduction

Kim Forss and Mita Marra

Equity and Evaluation

Most of us refer to equity when we talk about public policy on tax structure, recruitment to schools, participation in elections, and access to health care, and about the way we are treated under the policy, to name but a few subjects. We often refer to equity when we talk about the private sector as well, and equity is a central concern in many civil society organizations. Equity is everywhere. But does it play an important role in the evaluation discourse? In theory, it should because one of the main criteria in the assessment of worth and merit would be equity.¹ Equity is often suggested as one of the key qualifications for a just social order. Equity is concerned with grand themes in society such as the distribution of power and influence, the rights and duties of people, and the qualities of life. These are themes that have attracted the most influential thinkers since ancient times; Plato and Aristotle, Thomas Aquinas, Machiavelli, Rousseau, Smith, Marx, and in recent times, John Rawls (1971), Robert Nozick, Martha Nussbaum (2001), and Amartya Sen (1999), to name but a few. It should attract—and challenge—evaluators too.

Fundamental changes occur in the pursuit of equality: this was evident in the great liberal remaking of society during “the long” nineteenth century, the French Revolution, the Russian Revolution, the Chinese Revolutions, the anticolonial movements, the collapse of the Soviet Union and its client states, and the Arab Spring. Throughout, the notion of equity has been a formidable driving force. At the beginning of the twenty-first century, the global community rallied around the Millennium Development Goals. These are supreme international efforts in the pursuit of equity.

By the outset of the international financial crisis in 2008, equity considerations in public policy analysis and evaluation had acquired center stage. With neoliberal thinking being under profound reassessment, social justice has received increasing attention both within policy and academic circles. Accounting for social costs in relation to income, class, gender, age, disability, ethnicity, environmental quality, context, and culture has become both a political quest and a complex methodological undertaking.

A Plea for Equity: Why Equity-Focused Evaluations Are Needed

Without claiming to be in any way exhaustive or comprehensive, we would like to point to ten reasons why it is even more urgent today than ever before that evaluation addresses equity—and what, in particular, the challenges of equity-focused evaluation might be.

1.The current economic crisis has exacerbated inequities within society. Past gains in reducing poverty levels have to some extent been lost, and there are increasing gaps between the rich and the poor in most countries. Evaluation is called to explore distributional issues on different groups in society as well as redistributional effects of public policies within the current critical circumstances.

2.Along with the economic and financial international crisis, globalization is changing the landscape of competition. In the global economy, both large and small firms face environmental, labor, and social dumping. Equity-focused evaluation is needed to exercise socially and environmentally responsible practices to assure fair competition.

3.Intergenerational imbalances are on the rise because of growing youth unemployment and precarious work prospects. A largely shared conviction that previous generations were wealthier and happier than the ones now facing current economic crisis is spreading. This undermines social cohesion. Thus, equity-focused evaluation should look at how and to what extent education achieves development results for children and enables youth to acquire valuable skills to pursue their professional aspirations. Equity-focused evaluation should assess how schools and societies at large invest in youth capacities and participation, bridging the gaps between schools and communities.

4.Gender disparities are persistent in decision making both within households and economic and political institutions. Available data highlight that, because of educational achievements and the feminization of traditionally male-dominated sectors, women’s condition has improved over time. Yet, the traditional gendered division of labor remains. Although in most advanced Western societies, maternalism seems to have lost its appeal in public discourse, care continues to be assured within the household, predominately by women. Work and family imbalances originate in both the weak labor market and welfare system. Equity-focused evaluations are needed not only to highlight existing gender inequalities in freedom, income, work conditions, access to resources, and physical integrity but also to assess the effects of policies, which aim to promote women’s participation in public life. Furthermore, equity-focused evaluation should gauge the adverse effects of supposedly neutral public policies on women, disabled, and ethnic and social minorities.

5.Migrations are increasing throughout the world. People relocate for many reasons, but often face marginalization and discrimination in the country of settlement, barriers to accessing social services, challenges to the rights to citizenship and identity, economic insecurity, and social and cultural dislocation. Equity-focused evaluation is needed to understand human-rights issues and cross-cultural differences, in addition to the costs migrants and their children bear to combine work with schooling or training while managing to save.

6.Welfare policies are progressively limited in scope. Social assistance frequently relies on the fl awed assumption that poverty originates exclusively in the dysfunctional relations occurring mostly within the family and not in the uneven and exploitative interactions existing in the labor market. For lower-income households, poverty is often framed in terms of the psychological damage associated with the faulty childrearing practices of dysfunctional families. Equity-focused evaluation is needed to underscore how this approach reinforces the social stigma against the poor. Although the political debate surrounding welfare policies seems to align with a more egalitarian vision of society, and with the notion of welfare provision as social investment for individual and collective well-being, active social inclusion measures may end up reinforcing class and gender stratification, casting a negative light on welfare recipients. Furthermore, this policy approach may result in a supply-side intervention with limited potential for redistribution if feedback from targeted groups is not the basis to detect latent social needs related to breadwinning and caregiving.

7.Social philanthropy is increasingly delivered by charity organizations, which are often unable to alter existing power dynamics between rich and poor and support causes that do not challenge the status quo, emphasizing the temporary alleviation of poverty. Furthermore, the charitable motivations of donors may be self-interested or patronizing. In this sense, equity-focused evaluation should support individuals or groups working toward issues related to democracy, citizens’ rights, justice, and quality of life as opposed to directly assessing performance of service agencies, hospitals, schools, museums, and non-political organizations engaged in providing public goods.

8.Administrative agencies are increasingly delegating design and delivery of public services to private for profit and not-for-profit organizations. Beyond privatization, public-private partnerships represent the second generation of efforts to bring competitive market discipline to bear on the government provision of goods and services. Partnership then becomes a process to design, organize, and implement action for development in a participatory way. In this regard, equity-focused evaluation needs to take into account the institutional, financial, and symbolic aspects related to public-private partnerships. This calls for identifying the underlying causal mechanisms that link inputs, implementation processes, and outcomes: that is, taking into consideration the dimensions of cost (monetary and non-monetary), risk sharing, accountability, power, prestige, commitment, and trust.

9.Contextual conditions vary. Differences include multiple dimensions of economic, social, political, and institutional dynamics, which lead to different experiences, expectations, outcomes, and outlooks for both individuals and communities. Much of the ability of programs/ projects to take off and develop depends upon the “local chemistry” and “embeddedness” of implementation. That is, the historical and sociopolitical—perhaps cultural—conditions leading actors to behave in one way or another. Equity-focused evaluation is called on to involve local stakeholders to unveil local needs and resources. A good understanding of these contextual conditions is just the first step to formulate some kind of flexible “working hypothesis” about future behavior. The implementation process is, therefore, a major concern for equity-focused evaluation to promptly correct deficiencies underway and promote organizational learning.

10.Public policies have grown as complex interventions with numerous actors involved in implementation and emergent results, which cannot be anticipated and measured ex ante. As policies and programs grapple with increasingly innovative policy instruments as well as increasingly sophisticated citizen demands for participation and accountability, equity-focused evaluations are likely to provide some of the vital responses relying on connections among actors and the understanding of the links between micro and macro variables. Equity-focused evaluation should promote integrating local and transnational work, linking micro analyses of specific and delimited contexts to synthesis of experiences. Equity-focused evaluation should establish formal and informal information channels, increasing the space for communication, enhancing rewards for interaction. The information and the knowledge thus generated may constitute dynamic assets that form the basis for improved future interactions, stronger empowerment for marginalized groups and affirmative actions throughout the world.

The Purpose of the Book

The list could be made longer and more nuanced, and the challenges to the practice of evaluation more elaborate. The challenges and what it takes to engage with equity is the subject of this book. What is missing in current debates and in evaluation assignments is assessed. Broadly speaking, we suggest that the issue of equity needs to be addressed at three levels:

• There is a moral and philosophical level at which there is a need to be precise and clear about terms being used. Social justice, equity, and equality are concepts that need to be distinguished and related to professional practice. Equity relates to other goals and objectives, often pointing at conflicts. It is of course self-evident that a concern for equity would raise issues related to other policy objectives as almost all public (and private) activities have equity effects of some form.

• There is a systems-wide professional level at which evaluation needs to be sure of its mandate and at which evaluators need to think clearly about why they are in this profession and no other, and at the same time, where all stakeholders need to be clear about their expectations around evaluation. In practice, many evaluations are supposed to provide feedback on whether policy objectives are reached or not, and on the efficiency and effectiveness of implementation. Does that include the equitable distribution of impacts? Evaluators need to make such choices and to set boundaries around their inquiry, and on what side of that fence should equity questions be—and who makes those decisions.

• There is a practical and methodological level at which equity needs to be discussed. Once an evaluation has addressed issues of impacts, the next step would be to analyze the distributive aspects of impact, that is, equity. The impact of a microcredit program may be positive in terms of business growth and development, but it may also have equity outcomes that need to be understood. Although policies that reduce poverty generally improve equity, this relationship does not hold without exception. A poverty eradication scheme could improve the welfare of the poor while worsening the lot of the poorest and most vulnerable. A focus on equity calls for a nuanced understanding of existing social conditions, and thereby adds conceptual sharpness to assessments of poverty alleviation and brings to the fore considerations of justice at a social scale (McDermott, Mahanty, and Schreckenberg 2011). This suggests two separate issues, which also summarize the topics covered in this book:

1.What value basis can evaluators refer to when they analyze results, particularly the impact, how do evaluation teams arrive at conclusions on fairness, justice, and equity, and how can those processes be made transparent and gain credibility and legitimacy?

2.What are the challenges in terms of data collection and analysis when pursuing the equity questions? Are there similarities when different aspects of equity are treated? What are the particular difficulties of addressing stakeholders from an equity perspective? Are the groups whose concerns need to be voiced often “outside” the project/program setting, and if so, how do evaluators come to engage with them? The classical case for this question would be the intergenerational aspect of equity, where the interests of those not yet born should be considered—the right to sustainable livelihoods of future generations.

Much like the previous twenty volumes in the Transaction series on Policy Evaluation, this book responds to these challenges through an international comparative perspective, and is based on practical cases and experience from different sectors and in different policy settings.

The Outline of the Book

This volume is organized along three themes, which unfold along separate tracks while highlighting their evident connections. Some of the sixteen chapters are short and focus on a particular aspect of equity in evaluation, while others are longer and present empirical material as well as theoretical arguments. The book is thus different from other volumes in the Transaction series on Policy Evaluation. We have encouraged this mixture of short and long presentations as we would like to present a multifaceted debate on equity in evaluation. The first theme, Equity in Theory: Implications for Evaluation , presents theoretical perspectives that different disciplinary bodies of knowledge have thus far developed. We open up the first part of this book, by touching upon the philosophical roots of equity, its meanings from the tradition of public administration, the new-institutional as well as social theory strands. For those who long to see definitions of terms such as equity, equality, justice, and fairness, this will be where to go. Our discussion in that chapter obviously refers to Rawls’s (1971) definition of equality in “The Teory of Justice.” In the next chapter, Befani explores that concept by adopting an economic analysis of equity to reconcile utilitarian and Rawlsian notions of what is fair and equitable, and points to the responsibility of the evaluator to make the appropriate assumptions. The author argues that the source of conflict lies, at least in part, in the way utility is defined; thus, she proposes a practical definition of utility, based on the quantity of a s...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• 1. Introduction
• Part I: Equity in Th eory: Implications for Evaluation
• Part II: Equity in Evaluation Approaches: Challenges for Evaluators
• Part III: Equity in Program Evaluation: Lessons Learnt
• Contributors
• Index