Evaluation in Small Development Non-Profits

Deadends, Victories, and Alternative Routes

Leanne M. Kelly

• Publisher: Palgrave Macmillan
• Year: 2020
• Print ISBN:
  9783030589783
• eBook ISBN:
  9783030589790
• Topic: Social Sciences
• Subtopic: Nonprofit Organizations & Charities
• Index: Social Sciences

Research on evaluation shows that low-use and non-use of evaluation is common, yet evaluation is hailed as beneficial and worthwhile. The worth of evaluation is tied to its utilisation, presenting a paradox if evaluation is both revered and underutilised. This book investigates this paradox in the under-researched context of small development non-profit organisations, which have specific resource constraints and 'bottom up' community development values that complicate their ability to do and use evaluation in line with top down directives. The book examines the utility, meaningfulness, and purpose of evaluation from small non-profit perspectives, and explores whether evaluation has value for these organisations. For development practice, it presents evaluative alternatives that reconceptualise evaluation as part of the active process of development rather than as an interval-based add-on. For evaluation theory, it highlights a historical preoccupation with improving evaluation without assessing its inherent worth, and considers alternative ways to enhance the value of evaluation for small non-profits.

© The Author(s) 2021

L. M. KellyEvaluation in Small Development Non-Profitshttps://doi.org/10.1007/978-3-030-58979-0_1

Begin Abstract

1. The Nature of the Problem

Leanne M. Kelly¹

(1)

Faculty of Arts and Education, Deakin University, Geelong, VIC, Australia

Leanne M. Kelly

Email: [email protected]

End Abstract

Program evaluation is curiously paradoxical. Fifty years of research into evaluation utilisation suggests that evaluation has suffered from non-use, misuse, and low-use. Some of these underutilised evaluations are poorly conducted without sufficient thought to context or stakeholder buy-in. Others are formulated with meticulous rigour. Regardless, many of these end up relegated to the dusty shelf or filed away in a drawer. Maybe evaluation is not all it is cracked up to be.

Despite the high prevalence of evaluation underutilisation, almost unanimous voices, across the evaluation and development sectors and beyond, hail evaluation as vital for accountability, demonstration of effectiveness, and program improvement. It is considered fundamentally good, like ‘motherhood and apple pie’ (Garbutt, 2013, p. 1). The assumption of evaluation’s inherent worth and the orthodoxy that surrounds it is rarely explored.

This book explores this paradox in the niche context of small-sized international and community development non-profit organisations, defining ‘orthodoxy’ as practice and ideology that relate or conform to accepted conventions within a discipline, which may operate subconsciously under the influence and direction of those with power. As evaluation is a nascent trans-discipline, it has no clear overarching meta-theory. However, I have observed non-profits and their donors accepting standard evaluation approaches as being best practice, gold standard, and evidence-based. While alternative evaluation theories exist that are arguably more relevant to the evaluation of development programs, these are overpowered by the evaluation orthodoxy. The understanding of evaluation orthodoxy adopted in this book encompasses dominant elements of evaluation most polarised from the values espoused in community development theory. This is not to suggest that all evaluators operate at this far end of the evaluation spectrum as, in reality, they spread along its length, but continue to be influenced by the prevailing discourse.

The evaluation discipline’s occupation with utilisation belies a key assumption adopted in this book: that the existential worth of evaluation is inextricably linked to its utilisation. However, rather than using the research on utilisation to question evaluation’s appropriateness in certain contexts or to rethink its right to exist, the evaluation literature has been ‘almost silent on the normative matters of why such measurement is occurring’ (Barraket & Yousefpour, 2013, p. 455). It focuses instead on improving the skills of (particularly external) evaluators to enhance rigour and utilisation of approaches deemed suitable by the orthodoxy. Academic literature on evaluation seeks to improve evaluation, without seeking to excavate the root causes of low- and non-use. In this, the evaluation orthodoxy has colonised evaluation discourse.

Unchecked by existential critique of evaluation’s right to exist, the evaluation discipline is experiencing a thriving era of intense popularity around a narrow set of normative ideas. The focus on outcomes, impact, and evidence-based approaches in the development sector and beyond has been accelerating worldwide, with demand for demonstrable effectiveness catalysing an unprecedented push for evaluation.

While the push for evaluation has intended to strengthen accountability, effectiveness tracking, and program improvement, the growing orthodoxy is privileging certain types of evaluation as providing the most rigorous and credible ‘gold standard’ evidence, regardless of methodological appropriateness to the context of the subject of evaluation. This notion opposes Patton’s (2012) argument that the highest standard in evaluation is methodological appropriateness. Although commonly accepted as a good idea in rhetoric, in practice, methodological appropriateness is often overruled by adherence to normative hierarchies of evidence.

The importance of evaluation utilisation is acknowledged by evaluation stakeholders across a range of sectors; however, evaluation utilisation may be particularly significant for small non-profits due to profound resource-scarcity and limited capacity for evaluation. Details of this potential problem are unknown as small non-profits’ marginalisation from academic discourse has resulted in a dearth of available information on evaluation in these settings, despite them facing a number of unique context-sensitive evaluation challenges.

Upholding the expectations of the evaluation orthodoxy is particularly problematic for small non-profits operating within international and community development frameworks, a problem elaborated throughout this book. This is due to both the size of their overall program in relation to evaluation expectations, and the fact that human-centred principles underpin their approach and contradict elements of the dominant evaluation discourse. Numerous authors writing from within the development discipline have identified this disconnect between development principles and the prevailing orthodoxy of evaluation at a theoretical level (e.g. Eyben, 2015; Ife, 2013; Lane, 2013; Natsios, 2010; Roche, 2000, 2009, 2015). This disconnect has been largely ignored in the evaluation literature and unexplored using empirical data, a gap compelling the investigation and revision undertaken by this book. Additionally, this book helps fill the gap on small non-profits by providing previously unrecorded information regarding how these organisations undertake, utilise, and conceptualise evaluation.

The Focus

The nature of small non-profits means these organisations operate in significantly different conditions from their larger counterparts and, yet, are faced with increasing pressure to evaluate in similar ways due to belief in evaluation’s axiomatic beneficence. The problem addressed in this book builds on my suspicions that evaluations, including those conducted by small-sized development non-profits, may have low levels of utilisation, and can suffer from an apparent irrelevance to improvement of either operations or programming.

The overarching question explored in this book asks whether evaluation in small development non-profits has a point. This question raises existential and appropriateness issues as ‘the point’ refers to evaluation’s ability to be purposeful, meaningful, and utilised. This book approaches the research problem through a pragmatist epistemology that seeks to evaluate knowledge, action, and experience according to the success of its practical application (Kelly & Cordeiro, 2020). Using this theory of knowing, the question investigates both a critique of the idea that evaluation is inherently good and worthwhile for small non-profits, as well as a critique of various evaluation practices, with evaluation worth determined by the outcome of evaluation’s application in these organisations. This pragmatist epistemology seeks to search out ‘what works’ in specific contexts, defined through an interpretivist lens that recognises knowledge as constructed reality around useful and pragmatic ideas. Construction of new ways of knowing, acting, and experiencing is evidenced, in this case, through ethnographic participatory observation, document review, and semi-structured interviews with non-profit practitioners.

Three key assumptions regarding evaluation in small development non-profits gird the research question of what is the point? In this book, fulfilment of these assumptions constitutes evaluative ‘success’: (1) Development non-profits aspire to enact their stated values and commitments, and evaluative processes should support and enhance these aspirations. (2) Small non-profits are under increasing pressure, particularly from donors and development sector expectations, to conduct rigorous evaluations. (3) The existential worth of evaluation is bound to its utilisation. Therefore, without utilisation of some description, evaluation is pointless.

The question and assumptions probe the intersectionality between evaluation expectations, practical considerations, and development praxis. They inquire into evaluative processes by examining their consequences, to discover whether evaluation has a point. Armed with this information, this book uncovers and distils non-profit led ideas for meaningful and methodologically appropriate evaluation for small development non-profits derived from what is successful or unsuccessful in practice, as determined by the three key assumptions outlined above.

The Significance

Small non-profits are characterised in this book as organisations with a revenue under one million dollars Australian per year. This is about the equivalent of 560,000 British pounds or 677,000 United States dollars. In Australia, for example, eighty-one per cent of the Australian Charities and Not-for-profit Commission’s members have annual revenues under this amount. In the United Kingdom, ninety-three per cent of non-profits listed with the Charity Commission of England and Wales have annual revenues under half a million pounds. Reid and Kerlin (2006) corroborate this prevalence in the United States, calculating that seventy-five per cent of American international non-profits have revenues of less than half a million American dollars per annum. Furthermore, the actual number of small non-profits is likely to be significantly higher than recorded in official figures due to thousands of unregistered small organisations.

Nevertheless, while small non-profits outnumber large ones, large non-profits dominate academic literature and practice norms, including in regards to evaluation. Perhaps this is because ninety-six per cent of the funding to the non-profit sector in Australia, in a trend repeated worldwide, is channelled through large organisations (Cortis, Lee, Powell, Simnett, & Reeve, 2015). In this, small non-profits might be conceived as minorities in an organisational world. They can be viewed as marginalised and disadvantaged entities, fringe dwellers, unable to access the same privileges or move in the same circles as larger organisations, and therefore, invisible.

While small non-profits are subject to the same increasing pressure to demonstrate effectiveness as their larger counterparts, their contexts are fundamentally different to those of larger non-profits in a multitude of ways. Resource poverty is the most obvious disadvantage, with small non-profits often lacking in funds, time, equipment, workers, evaluation skills, and access to professional development. In terms of evaluation, this lack of resources limits the extent to which small non-profits can conduct and use evaluations.

International and community development non-profits have the added difficulty of aligning donors’ evaluation expectations with their organisational values, a challenging task as community development values often clash with the tools and methods employed for evaluating outcomes and impact. The special operating considerations of these often under-resourced non-profits, and their specific paradigm of praxis, confounds attempts to navigate an orthodoxy of evidence that is not always appropriate or feasible within the development context (Swanepoel & De Beer, 2006).

As the vast majority of the estimated ten million non-profit organisations operating worldwide are small, improved program effectiveness could deliver significant benefits, both economically and for some of the world’s most vulnerable people. For this reason, meta-evaluation that assesses evaluations is vital, including the questioning, analysis, and contextualisation of their axiomatic worth and normative practices. This is essential for small non-profits, given their uniquely resource constrained and altruistic setting. This distinctive setting requires bottom-up, context-sensitive, and adaptive solutions that may differ from those appropriate in other settings.

While the value of small non-profits is examined in detail in Chapter 4, the sheer volume of small non-profits worldwide necessitates deeper investigation of their ability to undertake and utilise evaluation. Despite less fiscal influence, there is a progressive observation that small non-profits offer greater value for money, in part due to high rates of volunteerism, and that they are valuable contributors of innovative practices that are regularly overlooked or unseen as they are crowded out by larger organisations. Improvements in evaluation utilisation, or appropriate context-sensitive alternatives, could increase dissemination of these innovations and ways of operating, assist program development, and secure sustainable funding sources. This could provide small non-profits with a platform to inform the sector and be active participants in the development discourse, from which they have been largely marginalised.

The Boundaries

Non-profit, non-government organisations, not-for-profits, charities, and community-based organisations are overlapping terms, sometimes used synonymously, for civil society institutions of the ‘third sector’—being neither for-profit, nor government, that exists to improve quality of life for people, animals, or the environment. They survive primarily on funding from donor sources including government grants, philanthropic trusts, and citizen donations, and are, in most cases, guided by altruistic principles. They differentiate from for-profit and government entities as they are value-oriented organisations th...

Table of contents

• Cover
• Front Matter
• 1. The Nature of the Problem
• 2. The Story of Evaluation
• 3. A Framework for Community Development Evaluation
• 4. Small Is Beautiful
• 5. Overt and Hidden Practices of Small Non-Profit Evaluation
• 6. Forms of Evaluation Use in Small Non-Profits
• 7. Practitioner Perceptions of Evaluation in Small Non-Profits
• 8. Alternative Routes: So What and Now What?
• Back Matter