The Future of Evaluation

Global Trends, New Challenges, Shared Perspectives

Wolfgang Meyer, Reinhard Stockmann, Wolfgang Meyer, Reinhard Stockmann

• Publisher: Palgrave Macmillan
• Year: 2016
• Print ISBN: 9781137376367
• eBook ISBN: 9781137376374
• Topic: Politics & International Relations
• Subtopic: International Relations
• Index: Politics & International Relations

Evaluation has become an important instrument for rational governance and is used in an increasing number of countries and policy fields. Recent developments at the global, national and local level are changing the conditions and functions of evaluation worldwide. This book examines current global development trends and changing demands for evaluation. It addresses issues surrounding professionalisation and globalisation, examining the need to strengthen accountability for social development in various different policy fields, regions and countries to improve governance and its impacts on social betterment. It also considers issues of quality, utility and further education and the upgrading of evaluation in a broad variety of different organisations, such as multilateral donor organisations, national public administrations, private consultancies, civil-society organisations, universities, and research institutes. With contributions from 30 different countries, this book combines a broad variety of viewpoints to examine the global future of evaluation.

Part I

Introduction

Ten Years of the Center for Evaluation: Review – Purview – Preview

Dieter Filsinger

This book is concerned with the future of evaluation in modern societies and is based on a conference of the same name held on the occasion of the ten-year jubilee of the Center for Evaluation (CEval). That international conference took place on 14 and 15 June 2012 at Saarland University in Saarbrücken, with more than 200 participants. What follows in this chapter is a textualized version of the encomium delivered on that occasion.

It is in the nature of an encomium, or eulogy, to be oriented toward the past, toward things a person or group of people have achieved. Whilst the future can only be talked about as a thought experiment, the past appears to be a thing that can be reconstructed with empirical reliability. But perhaps this differentiation between past, present and future is only valid to a limited extent. The chairman of the German Sociology Society (Deutsche Gesellschaft für Soziologie; DGS), Hans-Georg Soeffner, adapting an idea from the church father Augustine, very recently contributed a highly interesting thought on the matter: ‘The past, the present and the future are ways of addressing our memories and expectations; those memories and expectations – and the present itself – are always in motion, and always changing’ (Soeffner 2011, p. 138).

Ten years of the CEval at Saarland University. Isn’t that a rather short period of time for a jubilee celebration of international format? Is it possible that this is just a local happening, the importance of which is being overestimated? Are there not evaluation institutes that can look back on a much longer history? And how is tribute to be paid to the center in the context of the more recent history of evaluation in Germany? These questions cannot be answered exhaustively within this book chapter. Nevertheless, I shall attempt to do so.

In post-war Germany, science- and research-based evaluation has a varied, rather discontinuous history in which two waves can be observed. It was introduced in the context of the expansion of the welfare state and a reform policy which followed a new model, that of active, formative social politics (Kaufmann 2002; Filsinger 2007), in the framework of which educational and social reforms were launched in the late 1960s. We should think back here, for example, to the model experiments with comprehensive schools, the programs for the humanization of the world of work, or social services reform (Hellstern/Wollmann 1983, 1984b). The accompanying research projects which were placed alongside these (model) programs were heavily process-oriented and, under the influence of action research concepts, formative in nature (Hameyer/Haft 1977 for the schools sector). Practical and policy counseling played an important role. But the impact dimension, which has gained importance considerably in the evaluation discourse in recent years, was also taken into account, as shown for example by two articles in the magazine German Youth from 1976 and 1978, one on the subject of ‘impact analyses in youth work’, the other entitled ‘scientific accompaniment, impact analysis and success monitoring in the federal youth plan’ (Stackebrandt et al. 1976; Burger et al. 1978; also Hellstern/Wollmann 1979). Thus today’s paradigm, that of impact-oriented evaluation, is not entirely new, and that applies equally to the skepsis toward the control expectations associated with it (Kaufmann 1983).

It is true to say that evaluation was quite widely introduced with the experimental reform and model policies of the social-liberal federal government (since 1969) – as the social policy researcher and sociologist Franz-Xaver Kaufmann (1979) once referred to it; from evaluation in the Office of the Federal Chancellor to evaluation of local government policy, from programs in elementary education to social work concepts (Hellstern/Wollmann 1984). However, it would not be correct to refer to this here as a comprehensive implementation of evaluation in the period concerned.

Practical evaluations were also accompanied by a scientific reflection and methodological ‘perspectivation’ of evaluation research incorporating international experience, in particular the American evaluation discussion. At that time, only Weiss (1974) had been published in German.

The results of those reflections and concept discussions can be reviewed, for example in the anthology Accompanying Research in Social Education – Analyses and Reports on Evaluation Research in the Federal Republic (Müller 1978), the Handbook on Evaluation Research (Hellstern/Wollmann 1984), or in the anthology Experimental Politics – Reform Whim or Strategy for Learning (Hellstern/Wollmann 1983); (also Riedmüller et al. 1982; Institute for Social Work 1983; Lange 1983; Hopf 1983; Dietzel/Troschke 1988). In this context, the discourse about social science counseling and the use of social science knowledge, highly relevant for evaluation, deserves a mention of its own (Beck/Bonß 1989).

In their interim appraisal, Hellstern and Wollmann (1984a) arrive at the conclusion.

that the attempts to institutionalize and build capacities that measure up to the tasks at hand have so far failed to meet the demand. It is true that the outlines of a promising evaluation infrastructure are beginning to show thanks to the proliferation of research institutions close to the government, numerous research institutions in the länder, cities and towns, and the commitment of established and many younger research enterprises and university research institutions; the influence and the quality potential of evaluation research are however still low and would require additional training facilities and critical basic research in order to render the quality and influence of evaluation research more effective.

(Hellstern/Wollmann 1984a, p. 14)

However, no systematic anchorage of evaluation can be discerned in the methodological training at universities and tertiary education institutions.

Against this backdrop of disillusionment about the willingness and ability of the institutions system in the German Federal Republic to reform (Habermas 1985; Filsinger 2007) and the perceived limitations of scientifically accompanied policy-making, evaluation was not at first broadly institutionalized. Having said that, it does remain present in the scientific field of application-related research, which somewhat qualifies the theory of discontinuity formulated at the beginning. Publications include – even if they are only occasional – not only methodological articles on evaluation research (for example Marin 1978; Sieber 1984; Dewe/Wohlfahrt 1985; Heiner 1986; Beywl 1988), and in particular on the scientific assessment of intervention and reform projects (Brandtstädter 1990; Koch/Wittmann 1990), but also introductions (Wittmann 1985), a first textbook (Thierau/Wottawa 1990) and area-specific systematizations (Heiner 1996, 1998; Müller 1998; Badura/Siegrist 1999; Müller-Kohlenberg/Münstermann 2000). However, it was not until the first decade of the new millennium that a comprehensive theoretical, methodological and methodical systematization of the whole subject area was produced by Reinhard Stockmann and his colleagues.

With the introduction of ‘new management models’ in the 1990s (output-oriented management, quality management, obligations to produce evidence of effectiveness and efficiency) in public administration and the charity organizations, and against the backdrop of an increase in the number of social problem areas (long-term unemployment, the consequences of migration, marginalization in the social environment), evaluation is gaining markedly in importance. It would indeed, strictly speaking, be appropriate to talk of a second wave of evaluation, though it is in competition with other, business-management-oriented procedures (for example controlling). A kind of take off can be observed in connection with the PISA shock – the discussion on educational reform which began in the aftermath of the international comparative studies PISA, TIMMS and IGLU – and the labor market and social reforms, which have been evaluated comprehensively by order of parliament. As from the first decade of the new millennium, there has been an evaluation boom, and it is leading to a broad institutionalization of evaluation, but also to a broadening of perspectives. This includes the growing demand for monitoring systems in the context of the emerging trend toward reporting in the education and integration sectors (Filsinger 2008, 2014).

It may, on the one hand, be true to say that the founding of the CEval at its current size (an average of 12 externally funded research assistants) and its current structure could also be interpreted as an unintended consequence of the downsizing of the Department of Sociology at Saarland University; however, what is decisive is that Reinhard Stockmann recognized the ‘signs of the time’ and founded the center at the right moment, a center that can, without exaggeration, be described as unique in the Federal Republic of Germany. At almost breathtaking speed, the center has forged ahead with the project of the scientification and professionalization of evaluation, taking up on earlier scientific work and works from the first evaluation wave. Gerd-Michael Hellstern, Hellmut Wollmann and Hildegard Müller-Kohlenberg remain (co-) editors of the Zeitschrift für Evaluation and/or members of the CEval’s advisory committee. Their achievements have been more than remarkable.

Evaluation is only imaginable as an interdisciplinarily oriented field of expertise. Interdisciplinarity, indeed, is nothing less than constitutive. To that extent, it would seem risky to apply the usual criteria for the recognition of a discipline in its own right in the scientific community: a definable area of knowledge, basic research, a systematic knowledge basis (terminology, theories), research methods of its own, an identity of its own; and these must be supplemented by scientific associations of its own and media for scientific discourse. Professionalization, on the other hand, is unthinkable without scientific foundations and scientific training and further education.

Since we are not, at this point, talking about a debate on general principles about the status of evaluation among the multitude of social science disciplines, about the correlation between discipline and profession or about the extent to which evaluation can be professionalized, let us apply the above-mentioned criteria as a heuristic for analyzing and paying tribute to the achievements of the CEval.

The CEval has many years of practical experience in evaluation, in particular in development cooperation, but also in other fields such as environmental policy, vocational training and culture, to name but a few. The number of evaluation assignments and evaluation reports (194) is impressive, as is the volume of external funding. However, the focus in this eulogy is not on these achievements in practical evaluation, even though it is this practical evaluation work that has enabled skills to be built up which allow us to refer to the CEval as a competence center.

The emphasis should rather be placed on the fundamental work that Reinhard Stockmann and his staff have carried out on the development of theory and method in evaluation. The results are documented in a large number of publications (books, essays in magazines, articles), in some cases also available in English, Spanish and Chinese. In particular the fundamental principles of evaluation in social science terms from the year 2000 should be mentioned (Stockmann 2000), in which Stockmann not only systematically reappraised the international evaluation literature, but also established an evaluation model of his own. A seminal volume on the connection between evaluation and quality development and on impact-oriented quality management (Stockmann 2006), a comprehensive handbook of evaluation with practical guidelines for action (Stockmann 2007), and the introduction to evaluation submitted most recently (Stockmann/Meyer 2013) should also be mentioned. At the Waxmann Publishing Company, Stockmann is the originator of the series ‘social scientific evaluation research’ and has been its editor since 2002. He founded the Zeitschrift für Evaluation, now in its 11th year, and continues to be its executive editor. The editorial office is in fact registered at the CEval. He is also one of the founder members of the (German) Evaluation Society (DeGEval) and has played a decisive role in the shaping of that society, as have staff members of the center in various different capacities (working-group spokespersons, board members). The ‘standards for evaluation’ were developed with important contributions from colleagues from the CEval.

The center has been impressively successful in creating a systematic basis for science-based evaluations. That basis is the result of a clever combination of application-oriented contract research and pure development, which is probably only possible in the context of a tertiary education institution. Stockmann and his colleagues have understood very well how to tap the potential of practical evaluation work and contract research for pure development. Since there are, today, so many evaluation and praxis research institutes that one can hardly keep track of them any more, the tasks and the development potential of the CEval probably lie mainly, if not exclusively, in problem- and praxis-oriented basic research. A series of doctorates have already helped to set a course in this direction and, at the same time, have made a contribution to assuring the emergence of a new scientific generation.

The professionalization of evaluation is a task that could be said to have ground to a halt in the first phase of evaluation in Germany. A basic study course for evaluation did not seem to make sense, even if evaluation research should be, and in many places already is, a fundamental component of elementary research-methodological training in educational and social science study courses. After all, educational science and social education departments, for example, lay claim to evaluation as part of their own discipline as if it were the most normal thing in the world, and they seek to conceptualize it, though admittedly from a disciplinary point of view. It seemed to be nothing short of imperative that the CEval initiate a study course of its own in order to push ahead with the professionalization project. The idea of a Master’s program in evaluation came from the CEval. It was realized in collaboration with the then Catholic University of Social Work and its cooperation partner the Saarland University of Applied Sciences (HTW). That collaboration made it possible to extend the range of evaluation fields out beyond development cooperation, championed by the CEval, by adding the educational sector and the social services, and to strengthen the qualitative method training and finally complement it with economic evaluation.

The first Master’s program in Germany now a joint programme with the HTW, where CEval (Saarland University) and HTW each are responsible in equal parts for content and delivery is getting on a bit now as well. This year it too celebrated its ten-year jubilee. Having said that, the Master’s further education program is not the only activity in the sphere of advanced and further education. The further training course for skilled personnel in development cooperation is already older than the Master’s program and indeed played a part in its inspiration; the CEval has organized further education and training courses for a large number of institutions.

The CEval is a central actor in German evaluation research and has made a significant contribution to revitalizing it and helping to give it a distinctive image. The very considerable achievements of the CEval for Saarland University are probably appreciated most by its president. The new government of the Federal State of Saarland has the opportunity to rediscover the benefit of the center for Saarland and its development potential.

If we look at the manifold contributions of the CEval to the institutionalization, scientification and professionalization of evaluation, we can make the objective observation that this center has earned its reputation and should be acknowledged for its uniqueness. In particular, the way it has linked contract research to pure development and application-related research to basic and continuing training, and its outstanding commitment in scientific terms and in terms of professional policy justify such references to uniqueness. The author is not alone in making such an assessment. External evaluations confirm this finding. It is not necessary to have a low regard for the evaluation research achievements of other minds and institutes to pay tribute to the outstanding achievements of the CEval.

No one studying the history of the CEval will find it difficult to recognize ‘stages of development’. A more precise reconstruction could perhaps be made the subject of some Master’s theses. The last stage in that development for the time being has been the internationalization of the CEval. This assessment, however, is inaccurate, at least in as much as the international perspective was there all along, thanks to the fact that the CEval was so firmly anchored in development cooperation, not only in practical program and project evaluation, but also clearly visible in the way the Master’s program was put together. But in matters of the institutionalization of evaluation research and initial and continuing training in evaluation, the center has in recent years experienced a very clear jump forward in internationalization. The reformation of a Master’s program in evaluation in Costa Rica and the blended learning study course in English, currently being prepared, deserve particular mention here.

So what is there to come? What is the next stage of development? Discussing that would certainly go far beyond the bounds of a mere eulogy. The conference will surely provide some indications. One task a university evaluation research institute will have to fulfill in the future is the analysis and reflection of the consequences of the expansion and institutionalization of evaluation in almost all the societal areas of the German Republic and beyond, well aware that the CEval itself is a part of the field to be subjected to observation and critical analysis. However, it – the institute in question – will only be able to perform this reflexive task in an appropriate way if it can pursue questions of its own relatively independently of the evaluation market, that is to say independently of clients. This does however call for some appropriate basic features. For evaluation research in particular, which must refer to societal lifeworlds and practical problems and engage in interaction with the field actors, it is especially important to adopt an analytical perspective and maintain a certain distance from the subjects it is endeavoring to describe and analyze. With the topic of ‘ethics in evaluation’, introduced into the discussion just very recently by the CEval, this perspective is reinforced. Only in this way, through diligent empirical and self-reflexive work, can evaluation research make a (modest) contribution, aiming ‘to ease the hardship of human existence’ (Bertolt Brecht).

1

The Future of Evaluation: Global Trends, New Challenges and Shared Perspectives

Reinhard Stockmann and Wolfgang Meyer

Introduction

During the last decades, evaluation...

Table of contents

• Cover
• Title Page
• Copyright
• Contents
• List of Figures and Tables
• Foreword
• Acknowledgments
• Notes on Contributors
• Part I: Introduction
• Part II: The Role of Evaluation in Society
• Part III: Evaluation as a Profession: Global Trends
• Part IV: New Challenges for Evaluation: Practice and Utility
• Part V: New Challenges for Evaluation: Theory and Methods
• Part VI: The Future of Evaluation – Shared Perspectives?
• Bibliography
• Index