Internal Evaluation in Non-Profit Organisations

Practitioner Perspectives on Theory, Research, and Practice

Leanne M. Kelly, Alison Rogers

• Publisher: Routledge
• Year: 2022
• Print ISBN: 9781032023472
• eBook ISBN: 9781000592795
• Edition: 1
• Topic: Business
• Subtopic: Business General
• Index: Business

Focused on the interpersonal aspects of internal evaluation in non-profit organisations, this book presents practice-based discussions centred on six key topics identified through the authors' experience as evaluation practitioners.

Internal Evaluation in Non-Profit Organisations: Practitioner Perspectives on Theory, Research, and Practice is not a step-by-step how-to guide; instead, each chapter unpacks an aspect of internal evaluation in non-profits that is paid insufficient heed in the existing literature. Written by and for internal evaluation practitioners, the book contains a plethora of practical strategies and critical analysis of thought-provoking topics that are of particular interest and importance to internal evaluators in non-profit settings. The authors understand the pressures facing practitioners and non-profit organisations and share their insights around improving evaluation's ability to be efficient, embedded, useful, and meaningful.

This book will be of interest to researchers, scholars, and students focusing on non-profit management and will hold specific value for internal evaluators who want to harness their unique and influential position to help organisations achieve their goals. Further, this book is ideal for individuals wanting to think critically about evaluation and improve evaluation utilisation by developing their professional capability, building teamwork skills, using informal everyday data, incorporating theory, and developing fruitful relationships with external evaluators.

1 Internal Evaluators: An Influential Position

DOI: 10.4324/9781003183006-1

Introduction

The purpose of this chapter is to highlight our positionality as internal evaluators in human-focused non-profits and provide a definition and discussion of evaluation and evaluators. As part of this discussion, we unpack the reasons why internal evaluation is a worthy activity, highlight what makes conducting evaluation within non-profit organisations particularly challenging, and note the tensions between internal and external evaluation. We conclude by introducing the reader to the topics covered in this book.

This book is written by internal evaluation practitioners in non-profits who have an experiential understanding of what it is like to try and muddle through alone. The evaluation literature provides limited guidance for internal evaluators and the discipline largely focuses on evaluation as an action conducted by external consultants. This information is still useful; however, internal evaluation is much more than a point in time epoch of engagement with evaluation users. The internal evaluation role is comprised of numerous tasks that are not expected of external evaluators, particularly around continuous evaluation capacity building, promoting ongoing utilisation of findings, developing an evidence base for work undertaken, and cultivating organisationally embedded evaluative processes and thinking (Volkov, 2011a). Importantly, internal evaluators’ immersion in the organisational setting gives them an interesting advantage in terms of creating sustainable change. This level of influence is unusual for people in roles that tend to be mid-range in the organisational hierarchy and highlights the potential for internal evaluators to create positive and lasting change. This requires patience, as it can take years of (sometimes exasperating) labour, but internal evaluators have the power to transform individual and organisational thinking.

In our experience, the career of an internal evaluator can be frustrating at times. On those occasions when colleagues are reluctant to engage with evaluation (see Chapter 3), when managers avoid prioritising community recipient perspectives (see Chapter 5), or when evaluative recommendations languish unaddressed (see Chapter 7), we might feel like it is time for a career change. The realisation of this frustration is often gradual, layering up resentment over time. For others, the realisation hits suddenly; the multitude of incremental disappointments that they had swept under the carpet reveal themselves simultaneously when the carpet is pulled back in a moment of clarity. While not blaming the evaluator (or their colleagues) for these frustrations, we hold that it is the evaluators’ obligation to attempt to address and reframe them. We offer ideas from theory, research, and practice throughout the book as suggested remedies to help internal evaluators set realistic timelines and expectations to achieve goals and highlight positive practices to help internal evaluators celebrate small wins along the way.

We raise this problem here to let internal evaluators know they are not alone and to help them reframe their frustrations. Instead of being disappointed when our efforts seemingly go to waste, we should see this as an opportunity for deep self-reflection (see Chapter 2) and a chance to pivot our positionality to walk alongside frontline workers and community recipients, starting where they are at and building on their small-scale embedded everyday evaluative activities (see Chapter 4). We can channel our frustrations towards understanding more about the different standpoints between us, our colleagues, community recipients, evaluation in the non-profit sector, and the evaluation discipline in general. Understanding more about how dominant narratives embed assumptions around the best types of evidence and the most worthy methodologies can help internal evaluators surface these assumptions and assess their consistency with the needs and values of colleagues and community recipients (Eyben et al., 2015; Ife, 2013; Kelly, 2021a).

This book acknowledges the joys of supporting colleagues and community recipients to capture their progress, broadcast their innovations, and improve their programmes. Additionally, we note the great skill and artistry that evaluators require to juggle scientific expectations of rigour and credible evidence against the practice realities and values inherent within non-profits (see Chapter 5). Exposing and being keenly aware of the potential paradigmatic disunity between the evaluation discipline and the non-profit sector can help us start to navigate a path through this quagmire in a manner that remains firmly dedicated to and consistent with organisational values. Throughout the book, we note the centrality of interpersonal skills and relationships in internal evaluation (see Chapter 3). Foregrounding the interpersonal and relational, evaluation in this book is repositioned away from methodological rigour, evidence hierarchies, and the latest fad in measurement (while still noting the importance of these) and instead focuses on people and organisational values such as social justice, community-led development, and redistribution of power. This positioning is core to the relevance and meaningfulness of internal evaluation in non-profits as these organisations were often established to fulfil purposes surrounding social betterment. As such, it stands to reason that non-profit’s evaluation should follow suit and align with the organisational values espoused (Kelly, 2021c).

The Point of Difference: The Authors are Internal Evaluators Working in Non-Profits

We begin by recognising the inspiring and seminal work of Preskill and Torres (1998), who made the link between evaluation and organisational learning and suggested that evaluative inquiry was an approach to learning that fully integrated evaluation into the way an organisation operates. However, since then books on internal evaluation are often based on the profession within the public and private sectors, in large organisations, with reference to evaluation units (Laubli Loud & Mayne, 2014; Love, 1991; Sonnichsen, 2000). These books are valuable as they outline how internal actors can develop systematic and efficient internal evaluation, particularly to help managers with decision-making. However, we saw a definite gaping hole in these and other evaluation tomes surrounding the interpersonal aspects of internal evaluation. Information is lacking on conducting rigorous evaluation in ways that align with non-profit organ-isational values, which are often divergent from the values of profit- or power-seeking organisations (Eyben et al., 2015; Ife, 2013). We noted the need for a book that explicitly unpacked the components of internal evaluation that are often more prominent in non-profit organisations than other settings. These elements include limited resources (staff, funds, and time), high levels of diversity (around demographics but also educational levels, skills, and technological aptitude), intricate and multiple competing compliance and accountability needs, low coordination across and between organisations, largely passionate and altruistically driven personnel, and a focus on resolving or alleviating complex and intractable social problems.

Clarifying that the non-profit context is different from other settings where internal evaluators may work highlights the need for and focus of this book. There are many how-to evaluation books in the market, so we do not intend to replicate that work here. Instead, we offer guidance and prompts to undertake practice improvement beyond the step-by-step stages of procedural evaluation work. As such, we have moved away from generalist and methodological discussions that are well-covered in the existing literature. We focus specifically on providing critical analysis of thought-provoking topics that are of special interest and importance to internal evaluators in non-profits. Further, we sought to contribute a new publication to the general internal evaluation literature from the practitioner’s perspective to update Sonnichsen’s 20-year-old and Love’s 30-year-old contributions. This addresses the identification that “the voice of those working inside the [organisations is] often silent” (Laubli Loud, 2014, p. vii). Lastly, in contrast to Sonnichsen and Love’s male perspectives, we consider it fitting that two female authors wrote this book since the majority of employees and volunteers in the non-profit sector are women (Productivity Commission, 2010).

Between us, we have worked for a wide variety of non-profits across four continents, including public health, peacebuilding, international and community development, family violence, child protection, humanitarian action, emergency services, disability, housing, and homelessness, among others. We have both undertaken doctoral-level research on internal evaluation in non-profit organisations (Kelly, 2019; Rogers, 2021). Our focus is on non-profits that provide direct support to people, rather than those supporting the environment and non-human animals, or providing research or advocacy. As our research and experiences cover a large portion of the various human-centred social issues addressed by non-profits, we are confident that the information and guidance provided throughout this book is applicable across the suite of non-profits providing social programmes.

Defining Evaluation

Evaluation is a systematic judgement of the quality, worth, significance, and merit of an evaluand, which may be a programme, project, or other entity under scrutiny (Mertens, 2009; Scriven, 1991; Stufflebeam & Coryn, 2014). We distinguish evaluation from research; research focuses on describing what is occurring while evaluation has an explicit and intentional focus on making value judgements (Mathison, 2008). Throughout this book we usually refer to the evaluand as a programme for simplicity but note that this is reductionist. Rigorous evaluative processes collect relevant data from appropriate sources to base judgements on sound evidence and reasoning. These judgements should consider stakeholder values, perceptions, and context and provide credible and trustworthy information that can be used for improvement, learning, accountability, knowledge generation, and advocacy (Mikkelsen, 2005). Evaluation can uncover whether the right things are being done, whether they are being done well, and whether they are having the desired impact.

Evaluation is usually considered a formal process conducted at specific intervals in the programme cycle such as baseline, interim, and final at programme end. Interim evaluations are typically formative and provide recommendations to improve the programme. Final evaluations are typically summative and draw evaluative conclusions about the programme to feed into the design of future programmes and provide accountability to stakeholders (Alkin, 2011; Patton, 2011; Scriven, 1967, 1991; Stake, 2013). Developmental evaluation offers a third category of evaluation, which occurs iteratively throughout the programme cycle and provides real-time learning-action feedback loops to facilitate ongoing improvement (Patton, 1994, 2011). Developmental evaluation is particularly applicable to work undertaken in non-profits providing social programmes as it is highly adaptive and suitable for measuring non-linear, complex, emergent, and dynamic contexts (Togni et al., 2015). Unlike the formative-summative binary, which sees programmes as reaching an end point and measures them against rigid, pre-determined outcomes and indicators, developmental evaluation sees programmes as undergoing continual change and supports them to adapt to context and innovate to meet emerging needs (Patton, 2011, 2015). Summative and formative, with the increasing inclusion of developmental, are regarded as the standard overarching types of evaluation across most disciplines (Davidson, 2005; Patton, 2011; Scriven, 1991).

Evaluation is frequently paired with monitoring, and they work symbiotically. Monitoring is captured in the many acronyms used to describe evaluation focused work such as M&E (monitoring and evaluation), MEL (monitoring, evaluation, and learning), MERL (monitoring, evaluation, research, and learning), and MEAL (monitoring, evaluation, accountability, and learning). While not judging quality, worth, significance, and merit, monitoring can help make real-time evidence-driven programme improvements by highlighting concerns and demonstrating trends (Kelly & Reid, 2020; Owen, 2006). Monitoring data is usually utilised as a foundation for evaluation, which incorporates this data in addition to data from other sources (Markiewicz & Patrick, 2016; Mikkelsen, 2005). At its most basic, monitoring refers to regular collection of data and the act of checking that data at certain short intervals to access how the programme is tracking against expectations (Owen, 2006; Rossi et al., 2004; Scriven, 1991). Markiewicz and Patrick (2016) argue that monitoring is “the planned, continuous and systematic collection and analysis of program information able to provide management and key stakeholders with an indication of the extent of progress in implementation, and in relation to program performance against stated objectives and expectations” (p. 12). Their identification of monitoring as an analytical activity moves it away ...

Table of contents

• Cover
• Endorsements
• Half Title
• Series
• Title
• Copyright
• Contents
• List of Figures
• List of Tables
• Foreword
• Acknowledgements
• About the Authors
• 1 Internal Evaluators: An Influential Position
• 2 Professional Development for Internal Evaluators
• 3 Practical Strategies for Building Interpersonal Skills
• 4 Informal Everyday Evaluation for Organisational Change and Improvement
• 5 Using Theory, Research, and Practice to Evaluate Effectiveness
• 6 Internal and External Evaluators Working Collaboratively
• 7 Increasing Evaluation Use
• 8 Internal Evaluation Enhances Organisational Effectiveness
• Index