Realist Evaluation in Practice

Health and Social Work

Mansoor A F Kazi

• Publisher: SAGE Publications Ltd
• Year: 2003
• Print ISBN: 9780761969969, 9780761969952
• eBook ISBN: 9781446230671
• Edition: 1
• Topic: Ciencias sociales
• Subtopic: Trabajo social

?The strengths of this book are the first two chapters outlining the issues around realist evaluation and the final chapter, as well as the use throughout the book of real-life examples… in conclusion this is a timely introductory text on the methods and methodology of realist evaluation? - Critical Public Health

Realist evaluation is an exciting new way to investigate what works, for whom and in what contexts and here for the first time, Mansoor A F Kazi shows how it applies to today?s practice in social work, health and other human services.

To show how realist evaluation has the potential to change practice, the author provides in-depth examples of evaluation in adult rehabilitation, drug-using communities, users of family centres and an NSPCC project that provides services for children who sexually harm other children. Within these instances, realist evaluation procedures were integrated into the practice and decisions regarding the content and the targeting of services were developed effectively.

Written in an engaging style, this book will be of value to social workers and all those engaged in service delivery or research across the spectrum of health and social care.

1

Introduction

This publication aims to outline and develop a framework for the application of the new, emerging realist paradigm in evaluation research in practice, and applies it to social work practice in particular and to the practice of human service programmes in general. This paradigm is reported to have the potential for an evaluation strategy that not only systematically tracks outcomes, but also the mechanisms that produce the outcomes, the contexts in which these mechanisms are triggered, and the content of the interventions (Kazi, 2000a; Pawson and Tilley, 1997b). According to realism, human service programmes introduce interventions that interact with the other causal mechanisms in the contexts of service users to produce outcomes. The purpose of realist evaluation is reportedly to investigate ‘what works best, for whom, and under what circumstances’ (Robson, 2002: 39). However, to date the realist paradigm largely remains at the level of a philosophy of science (Bhaskar, 1998; Sayer, 2000), and as a manifesto for evaluation research (Pawson and Tilley, 1997b; Robson, 2002). As the title suggests, this publication attempts to develop an approach for the integration of realist evaluation into the practice of human services, and the methodologies that can be used for realist evaluation. Practice examples are provided from a number of studies to substantiate the potential contribution of realist evaluation for practice.

To date, there is a dearth of published examples of realist evaluations in any area of human service practice which can demonstrate the utility of the paradigm or the processes and methodologies that can be used to actually achieve an investigation of ‘what works best, for whom, and under what circumstances’. The paucity of examples is exemplified by the fact that the advocates of realism tend to use the same few examples as illustrations (e.g., Pawson and Tilley’s, 1997b example of crime prevention measures which also appears in Robson, 2002 and Sayer, 2000). This publication attempts to make a contribution by providing real examples and helping to develop ways in which this philosophy of science could be translated into an evaluation paradigm for practice. In particular, this publication develops realism as a paradigm for practitioner-evaluators, with the development of a realist effectiveness cycle which can be integrated into the practice of human services in a partnership between internal evaluation (that is, what the agencies themselves do) and external evaluation (that is, evaluations carried out by academics and external consultants). In this way, the findings from evaluation can be directly utilised to influence the future development of programmes.

Evaluation for Practice

The pressures on social work practice to demonstrate effectiveness have continued to grow in the last two decades. The pressures from changes in the legal and societal context mean that social work is no longer taken for granted and that its worth has to be demonstrated (Parton, 1994). In Britain, the Children Act 1989 and the Community Care Act 1990 both included requirements for planning in response to need, and reviewing progress. The purchaser–provider split, the growth of the voluntary and private sectors alongside the public sector, and the introduction of competition for contracts also made monitoring and evaluation more central in social work practice. The current British government has continued this trend with an emphasis on ‘league tables’ which rank health and social services according to performance. There is a growing emphasis on evidence-based practice and clinical effectiveness (Powell, 2002). The resources are finite, and yet the social needs are complex and in a state of flux. Evaluation research is one way to make social programmes accountable and to enable politicians, agencies and practitioners to make hard choices in the allocation of scarce resources. The analysis thus far has concentrated on the question of accountability and demonstrating the merit and worth of social work. There is another dimension to this – the need to develop and improve the content of social work practice itself, so that it is better able to meet the needs of its clients and the wider society.

According to Mark, Henry and Julnes:

Evaluation assists sensemaking about policies and programmes through the conduct of systematic inquiry that describes and explains the policies’ and programmes’ operations, effects, justifications, and social implications. The ultimate goal of evaluation is social betterment, to which evaluation can contribute by assisting democratic institutions to better select, oversee, improve, and make sense of social programmes and policies. (2000: 9)

Robson adds: ‘Evaluation is often concerned not only with assessing worth or value but also with seeking to assist in the improvement of whatever is being evaluated’ (1993: 175). Therefore, there are two main purposes of evaluation research – providing evidence of the merit and worth of social work practice, and striving to improve practice itself to respond to the changing needs and contexts, for the betterment of society. Whether emphasis is placed on one or the other of these purposes may depend on the paradigmatic influences that are inherent in the effectiveness inquirer’s activities. For example, from a critical theorist perspective, Shaw adds another purpose in evaluation for practice: ‘Evaluating in practice is not limited to determining whether social work is effective, but must be a means of empowerment and social change’ (1996: 189).

The epistemological debates in philosophy and in other sciences are also reflected in social work research, with perspectives ranging from the ‘empiricist’ view that effectiveness can be demonstrated through empirical evidence of effects, to the ‘humanist’ or interpretivist position that effectiveness can be demonstrated through the subjective perspectives and the meanings attached to such perspectives (as described in Shaw, 1996: 21).

In response to these developments in the philosophies of science, there has been a growth in research methods’ textbooks and other publications addressing the need for social work to demonstrate its effectiveness (Newburn, 2001). Most of the authors have tended to be university-based, but these publications also reflect a developing partnership between academics and social work practitioners. For example, Macdonald (1996) is one of a number of publications on effectiveness from Barnardos – a children’s charity and a voluntary social work agency; and Everitt and Hardiker (1996) is a British Association of Social Workers’ publication. Kazi (1998a) and Fuller and Petch (1995) directly address practitioner research, and Shaw (1996) has a number of examples from practice. These and other publications are contributing to the development of effectiveness strategies that can be applied to human service practice, by both practitioners and researchers.

This book also seeks to make a modest contribution to the development of evaluation research in contemporary practice in social work, health and other human services. The book begins properly in Chapter 2 with a critical analysis of contemporary paradigmatic approaches to the evaluation of practice, including the extent to which each approach is able to ‘capture’ the breadth and depth of the effectiveness of practice. In the main (but not exclusively), the recent publications referred to above reflect the influences of some of the main paradigms from the philosophies of science. Each of these approaches can be credited with the contribution it has made, and continues to make, to various aspects of practice effectiveness – this can be substantiated through an analysis of the ontology, epistemology and methodologies (Guba, 1990) associated with each paradigm. At the same time, each of these paradigms also has its limitations, and this critical analysis will attempt to show the extent to which any one of these paradigms can address the complexities of practice effectiveness.

Paradigms and Influences

The term paradigm is used a great deal in this book, and therefore it is important to clarify what this means from the start:

Close historical investigation of a given speciality at a given time discloses a set of recurrent and quasi-standard illustrations of various theories in their conceptual, observational, and instrumental applications. These are the community’s paradigms, revealed in its textbooks, lectures, and laboratory exercises. By studying them and by practising with them, the members of the corresponding community learn their trade. (Kuhn, 1970: 43)

The recent publications in social work effectiveness research (reviewed in more depth in Chapter 2) encompass the main contemporary paradigms, as Kuhn suggests. However, like most of the terms used in this book, precision in definitions is not possible, as concepts tend to be used in different ways, and the definitions shift according to the way a term is used. Kuhn is credited with the introduction of the notion of the paradigm, but Masterman (1970) notes that he used the term in 21 different ways. In this book, the term paradigm is used in the sense of a ‘set of beliefs about the nature of the world and the individual’s place in it’ (Mark, 1996: 400). In an operational sense, paradigms are characterised by the inquirer’s world view, or his/her outlook with respect to the existence of reality, the theory of knowledge, and the way one conducts an inquiry. In the practice of evaluation, discrete boundaries cannot be drawn for each paradigm (Kazi, 2000a; Shaw, 1996; Trinder, 1996), as the evaluator is likely to cross many a boundary. Nevertheless, paradigmatic influences can determine the selection of evaluation questions and the selection of research methods to deal with those evaluation questions.

The Realist Evaluation Paradigm

Realism is … a common-sense ontology in the sense that it takes seriously the existence of the things, structures and mechanisms revealed by the sciences at different levels of reality … the task of science is precisely to explain ‘facts’ in terms of more fundamental structures, and in the process it may reveal some of these ‘facts’ … to be, in part, illusions … we may not yet, and never finally, know whether it is true or false. (Outhwaite, 1987: 19–20)

The term fallibilistic realism was first suggested by Donald Campbell in a personal communication (Manicas and Secord, 1983); and it is also used by Anastas and MacDonald (1994) (and more recently by Anastas, 1999) who were the first to introduce this perspective in social work effectiveness research. However, if we include texts in the Finnish language (see Rostila and Kazi, 2001), then Professor Mikko Mäntysaari (University of Jyväskylä, Finland) wrote about realism in relation to social work a few years before Anastas and MacDonald. This perspective is also known by other terms, such as scientific realism, transcendental realism, referential realism or generally as a realist view of science or even as post-positivism (Fraser et al., 1991; Phillips, 1990). Archer (1998), Bhaskar (1998) and Robson (2002) prefer the term critical realism, emphasising realism’s critical role in social research. In the application to evaluation for practice, this author prefers the term realist evaluation, which is similar to realistic evaluation (Pawson and Tilley, 1997b). However, realistic evaluation implies a tendency whereas realist evaluation is a more emphatic description of this new paradigm in evaluation research.

Realist evaluation is based on the work of the philosophers Roy Bhaskar (1997, 1998) and Rom Harre (1984). Mark, Henry and Julnes (2000) and Pawson and Tilley (1997a, 1997b) have developed the realist paradigm as a legitimate evaluation research perspective in its own right. Anastas (1999), Kazi (1998a, 1999, 2000a, and 2000b) and Rostila (2000, 2001) attempt to apply this perspective in the evaluation of social work practice; and in other human services, Porter and Ryan (1996), Tolson (1999) and Wainwright (1997) apply realism in health services.

Realist evaluation seeks to evaluate practice within the realities of society. Practice takes place in an open system that consists of a constellation of interconnected structures, mechanisms and contexts. Realism aims to address all the significant variables involved in social work practice, through a realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place.

Realist evaluation research is about improving the construction of models and, therefore, about improving the content of the practice itself. Evidence from data gathering is used to target and adjust the content of the programme in such a way that it can have a generative impact on preexisting mechanisms and contexts, and help to bring about the desired changes. Objectivity lies not just in the use of outcome measures, but in the extent to which the model is analogous with reality. At each cycle, a better approximation of reality is obtained, as compared with the previous cycle. In this way, realism addresses all the dimensions and questions of effectiveness of practice, including contexts, the perceptions of all involved, ethics and values, and the content of practice. The multi-method data gathering addresses the questions of what actually works, for whom and in what contexts. (Kazi, 2000c: 317)

Underdevelopment of Realist Evaluation Methodology

The above definition provides a summary of this author’s development of the realist paradigm in the evaluation of practice in human service programmes. However, a contention of this book is that whilst realism is developed as a philosophy of science, at the level of methodology this paradigm is relatively underdeveloped at this stage. For example, at the sixth annual conference of the International Association for Critical Realism (IACR) in August 2002, one of the three main themes was ‘research using realism’:

What constitutes critical realist empirical research? In what ways does a critical realist perspective influence or facilitate substantive research? We are particularly interested in papers providing answers to these and other questions in the best possible way – by reporting the results of substantive research undertaken from a realist perspective. (p. 1, http://www.criticalrealism.demon.co.uk/iacr/conference_2002.html)

The fact that this is still a major theme suggests that these issues are far from being resolved at the level of realist research designs generally, and the same is true for evaluation research. There are no complete published realist evaluations of human services; and to date, no account of dedicated methodologies that can be applied to investigate what works, for whom and in what circumstances. For example, Pawson and Tilley (1997b) outline the rules and framework for a realist evaluation, but they fall short of specifying the methodologies for investigating what they call mechanism–context–outcome configurations. Robson (2000) has a helpful section on mechanisms and how they influence programme design and evaluations, but again does not explicitly state the methodologies that may be appropriate for a realist evaluation. Robson (2002) does suggest that inferential statistics may be used, as well as qualitative methods as specified by Miles and Huberman (1994); however, no complete examples of realist evaluation are provided.

As Tolson explains, realist evaluation

is an applied form of research which lends itself to the process of practice innovation through its contextual sensitivity. Accordingly it is complex and its methodological rules are still emerging. The apparent complexity will undoubtedly ease as this type of evaluation research matures and its practice is documented. (1999: 389)

Realism is methodological-pluralist, but the methods it can draw upon were developed either within the empirical or interpretivist paradigms that may not have the same ontological depth as realism. These issues are considered in Chapter 3, and then the rest of this publication provides examples of the application of realism in the evaluation of human services – with the aim of easing the complexity of realist evaluation, as indicated by Tolson.

Issues of Implementation and this Book’s Contribution

Realism transcends the qualitative and quantitative divide, or the epistemological divide between empirical and interpretivist approaches (Mark, Henry and Julnes, 2000; Pawson and Tilley, 1997b). Nevertheless, realists continue to be influenced by these debates and express a preference for either of these approaches, at the methodological level, even within the stated methodological-pluralist approach in realism. For example, Sayer (2000) advocates intensive research (which he defines in qualitative terms), as the only way of achieving an explanatory critical realist inquiry. On the other hand, Lawson (1998) emphasises the need to investigate demi-regularities, and thereby advocates extensive research, with an emphasis on empirical methods. These preferences are within the paradigm of realism, as it is expected that the selection of evaluation questions, and how the research is conducted, depend upon the theoretical and methodological preferences of the inquirer. However, the preferences expressed by these two authors are of crucial importance at this stage when realism is still an emerging evaluation research perspective, and when there is a need to develop this paradigm further in its actual application to the evaluation of human services. Realism is relatively underdeveloped at the level of methodology at this stage; and therefore, the question remains – can you go further from ‘what works’ to ‘what works, for whom and ...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• List of Figures
• List of Tables
• Dedication
• Acknowledgements
• Preface
• 1. Introduction
• 2. Contemporary Perspectives in Practice Evaluation
• 3. The Realist Evaluation Paradigm for Practice
• 4. An Example of Qualitative Research Methods in Realist Evaluation
• 5. Quantitative Methods: Evaluation of Family Centres
• 6. A Realist Evaluation of Practice: the NSPCC’s Shield Project and Intensive Analysis
• 7. The Shield Project: Extensive Analysis in Realist Evaluation
• 8. Patterns of What Works, For Whom and in What Contexts
• 9. The Contribution of Realist Evaluation for Practice
• References
• Index