Long Term Perspectives in Evaluation

Increasing Relevance and Utility

Kim Forss, Ida Lindkvist, Mark McGillivray, Kim Forss, Ida Lindkvist, Mark McGillivray

• Publisher: Routledge
• Year: 2020
• eBook ISBN: 9781000167931
• Edition: 1
• Topic: Politics & International Relations
• Subtopic: American Government
• Index: Politics & International Relations

Long Term Perspectives in Evaluation is the first book to advocate the virtues of a long-term perspective for policy evaluation as well as to show how evaluations can take a longer time perspective than they usually do. To get there, it is necessary to understand the decision-making context of evaluations and study the obstacles and the resistance toward long-term perspectives – as knowledge of that will lay the ground for more effective advocacy.

The book is divided into three parts: the first section examines different aspects of methodology and methods. In the next section, authors present case studies of long-term evaluations, examine their own experiences of such evaluations and discuss difficulties, challenges and lessons learned. Cases discussed include: education sector reforms in Sweden, local governance reforms in Denmark, policy interventions in Southern Italy and Brazil, and Paris Declaration Principles of aid effectiveness such as Swedish aid to Tanzania, Vietnam, Laos and Sri Lanka. Finally, the third section sees the authors turn to a set of contextual issues and concluding remarks.

Bringing together a rich collection of insights and a renowned group of experts, Long Term Perspectives in Evaluation: Increasing Relevance and Utility, constitutes a significant landmark in the field.

1 Introduction

Kim Forss and Ida Lindkvist

There is no doubt that decision-makers need quick and correct information. The world is changing fast – and changing at a faster rate than ever. New technology, global interconnectedness and unprecedented levels of risk and uncertainty shape the demand for information. Policy-makers require a continuous flow of up-to-date information. They are likely to want strategic information, accurate and tailored to their needs. Several of the volumes on comparative policy evaluation that our group has written deal with this, for example: Can Governments Learn, From Studies to Streams, Evaluating the Complex and Evaluation in Turbulent Times.

Evaluation is a demand-driven process, and those who finance evaluation act in contexts where decisions need to be taken quickly. Public policy choices obviously differ from the world of international finance. Looking at another field of work, those who speculate on the rise and fall of currencies, stocks and raw materials are accustomed to situations where the time span from information to decision might be a matter of seconds (Castells, 1996, 2013). In public management a time horizon of months and years might be more common. Even so, in a historical perspective, that is certainly short term.

The aim of this book is to take a step back and put the spotlight on the long-term perspective. We do not argue that decision-makers do not need up-to-date information, delivered in time. Nor do we question that change, turbulence and complexity are the orders of the day. However, just because so many factors speak in favor of short-term considerations, quick feedback and rapid delivery, it is timely to advocate for the opposite, namely the virtues of a long-term perspective.

The advantages of a long-term perspective are several. First, it is obvious that many times it takes a long time for social systems to change. There are a number of reasons for that; social inertia and path dependence are some. The fact that social change happens through a series of small steps – muddling through – rather than through grand initiatives (Lindblom, 1959) also speaks in favor of a long-term approach. Evaluations that have such advantages provide better and more accurate policy support. It is also likely that such evaluations generate learning. It is often said that evaluations do not contribute much to learning, and if they do, it is usually single-loop learning. Long-term perspectives may help evaluation go beyond this and generate double-loop learning (Bateson, 1976). Longer time perspectives will inevitably increase the complexity of any inquiry, in a complex world evaluation needs to come to grips with the factors that make it difficult to understand an intervention and its effects. The first purpose of this book is thus to advocate for more long-term perspectives in evaluation.

How can evaluation as a practice approach long-term change and long-term impact? Evaluation is a discipline at the crossroads. It has its own body of knowledge, its own professional ethos and its methodological debates. It is also an open practice where evaluators borrow, steal and are inspired by methods in other scientific disciplines. Evaluation as a discipline belongs to the social sciences, and its methods borrow much from the closest neighboring disciplines – political science, management and administration, economics and sociology. But compared to much other social science research, one could argue that evaluation tends to be more utilization-focused and hence more subjected to demand for “immediate delivery.” The relations to other disciplines are discussed in the volume Mind the Gap, which our group produced some years ago. However, the disciplines that are considered to most thoroughly address the past are found in the humanities rather than among the social sciences. History, philosophy, archaeology and ethnography come to mind as sciences whose methodological debates concern what constitutes valid and reliable knowledge of long-term change and long-term effects.

The second purpose of this volume is to show how evaluations can take a longer time perspective than they usually do. This capacity-building ambition follows two tracks. First, we assume that the methodological debates in history etc. are not widely known in the evaluation community. The book introduces and reflects on some of the key themes from these disciplines. Second, although not common, there are examples of evaluations that take long-term perspectives and some of those who have led such evaluations share their experiences in case studies.

The dual purposes of this volume are thus advocacy and capacity-building. We argue that an increasing number of evaluations need to take a long-term view of policies and programs – for a number of good reasons. We demonstrate how that can be done by discussing methodology and by providing examples of successful practice. No doubt, the need for short-term perspectives and quick inputs to decision-making processes will remain. It is more a question of redressing the balance – more comprehensive knowledge and higher levels of learning make for better decisions. Even short-term evaluations will be better if some evaluations take a longer perspective. To get there, it is necessary to understand the decision-making context of evaluations, hence the book also studies the obstacles and the resistance toward long-term perspectives – as knowledge of that will lay the ground for more effective advocacy.

What Does a Long-term Perspective in Evaluation Mean?

In this volume we define/argue that having a long-term perspective involves the use of historical data to explain the past, the present, or predict the future, and/or the use of theoretical models that take into account how long-term change occurs. A long-term perspective may be purely empirical or theoretical, or both. Having a long-term perspective will also have consequences for choice of methods for data collection and analysis.

A theoretical model that takes a long-term perspective could, for example, involve causal models that require longer time-series to make sense of so-called slow-moving variables that in short-term evaluations would simply be seen as context or exogenous variables, such as institutions or culture. While longer time-series may be required to assess change when the underlying theoretical model has a long-term perspective, this is not a requirement for having a theoretical long-term perspective. Long-term theoretical models could specify what needs to happen in the short-term to have long-term consequences. In evaluation of environment interventions, this is often coined transformative change.

How Long is Long-term?

Many of the studies that we point to in the text cover between 30 to 60 years of implementation. Many a historian might laugh at that. Fernand Braudel, one of the founders of the “Annales” school of thinking, explained the emergence of capitalism over a period of 400 years (Braudel, 1982). It took so many centuries to shape the social, economic and cultural system we call capitalism. Braudel also inquired into the identity of France, and that made him and his associates go back to prehistoric and Roman times to analyze waves of change as well as structures that change slowly – if at all (Braudel, 1986). But in his study of the Mediterranean in the age of Philip II (sixteenth century), Braudel analyzed change over a few decades only (Braudel, 1972).

Whether a perspective is long or short depends on what is being analyzed. The references to Braudel show that it is the nature of the questions and hence what you need to explain that imposes a time perspective. Historians deal with change across centuries, but they also analyze, for example, what triggered a certain event. Explanations of why the First World War started usually go back a few decades in time, but not more. To make it more complicated, it is not uncommon that several time scales are used in an inquiry. It is often necessary to look at both short-term and long-term factors to get a complete picture of how and why change occurs – and what the effects are.

In much of public administration, interventions are meant to have an effect within a few years’ time. To go beyond that, and look at a decade or more, would be to introduce a long-term perspective. Some of our cases describe bilateral development cooperation, where long term means going back to the origins and apply, for example, a 50-year perspective. As a rough estimate, we would suggest that anything beyond 10 years would be a long-term perspective, though most of the examples we point to look at a few decades more, and some a little less. The significant benefits of the longer perspectives seem to appear with that extension of a time horizon.

Using History to Explain the Past, the Present, or to Predict the Future

The uniting theme of the chapters in this book is that time unfolds over a longer period than it does in most evaluations. Three types of “unfolding time” can be discerned:

1. An intervention took place and came to an end. There was an evaluation at the time and its findings have become established wisdom – the lessons learned – about the intervention and its results. Many years later, a new study takes place, and in the light of new evidence, that established wisdom is challenged.

2. Policies evolve over time. In sectors such as education, defense and labor market development, governments are active through policies, programs and projects, working through one or several agencies. Evaluation is institutionalized at government levels, within organizations, and through external and independent agencies. Taking a longer-term perspective in such sectors involve analyzing policy-making, implementation and effects over several decades – and often up to and including the present moment.

3. In many types of interventions, the desired social changes take a long time to appear. The European Union’s programs for regional development are a case in point; the various projects for regional growth are implemented over a span of 3 to 10 years, but for backward regions to rise and show indications of sustained higher than average growth rates take longer than this. Similarly, private sector development follows time scales other than typical project cycles. While the return on an investment would normally be calculated over an 8 to 10-year period, the question of whether firms accomplish transformative change, grow, diversify, manage business cycles, etc. cannot be observed and assessed in less than a few decades.

These situations have the long-term perspective in common, but the approach might vary. The first case is a relatively rare phenomenon and it is not the main thrust of what a long-term perspective involves and what we discuss in this book. However, we have witnessed a few interesting cases. What was first proclaimed a success, turned up to have undesirable side effects and in the long run the positive results eroded. In other cases, what was first seen as a failure turned out to lead to sustainable change and to generate unexpected and positive benefits. It was probably coincidental that assessments were reversed but, on the other hand, the more complex and nuanced perception of change was to be expected.

Evaluating a policy field over several years as in the second case is different from coming back to a project that was implemented, came to an end, and then time has passed. When a policy field is evolving the evaluation must write its history, so to say, and analyze both implementation and results, in context. Taking a historical perspective means going back in time and following development within the policy domain. In the first case, the intervention has come to an end, and the challenge is to trace results. In the third case, the emphasis is also on the results and the context of implementation. The difference between the first and the third type of situation is that, in the third situation, it is simply not possible to measure results until time has passed.

Theoretical models alone or combined with historical data can be used in evaluation to address evaluation questions about sustainability and likely long-term impact. To take evaluations of environmental interventions as an example; evaluating the effectiveness of interventions that address long-term environmental change by necessity need to rely on long-term models and the type of change that needs to occur in the short term for long-term changes to materialize. Through models and historical data, evaluators can assess the likelihood of long-term change by looking at consequences of short-term changes in their model and in data.

Regardless of whether historical data are available, the underlying theoretical framework for an evaluation may benefit from taking a long-term perspective. An evaluation of an intervention that targets a social phenomenon depending on slow-moving variables needs to consider how the intervention will affect these variables or is affected by them. An intervention targeting a phenomenon heavily influenced by culture needs to consider the long-term consequences of the intervention on culture, or by culture. Similarly, the nature of the targeted change must be considered at least theoretically, even when data is not available. For example, if change is likely to occur at a tipping point, then even short-term evaluation relying on short-term data needs to consider whether short-term results are unlikely to persist due to the nature of change.

We also have the case where policy change is expected to have a major impact immediately. What would a long-term perspective add? Systems theory suggests that all systems tend toward an equilibrium. They can change and change fast, but after a while they return to equilibrium. A longer time perspective in evaluations help us get a truer picture of the effects of social interventions. Sometimes a short time perspective exaggerates the importance of immediate effects, at other times they do not reveal the effects that are incremental, accumulating and enduring.

Benefits of a Long-term View

We have suggested a few potential benefits of extending the range in evaluations, but there are more. The chapters in this book serve to substantiate for these claims. We realize that they are examples only; examples that can point to possible ways to make better use of evaluation resources. This book appears to be the first publication devoted to the time perspective in evaluation, and hence we can claim that our cases generate propositions about the benefits and advantages of taking a long-term perspective. Let us look at them, one at a time.

(1) Help detect the aggregated effects of policy change. An example from evaluations of the education sector in Sweden illustrates what this means. The decline of public education has a long story and goes back many decades, to the 1980s. With the start of international benchmarking the sense of decline took new shape, but it had been present many years before that. Various indicators of results, and many evaluations, pointed to problems in, for example, teachers’ compensation, school management, curriculum development, etc. The political debate has been lively; education is by no means a neglected sector. Hence there has also been a long series of reform proposals, and these have usually also been evaluated – each reform at a time. But the decline has continued. In 2012 the Swedish government appointed an inquiry into school reform, and after 2 years of study the committee presented its conclusions (Lewin, 2014). Rather than pointing at specific examples of failed or partially successful reform, the report argued that the long history of reforms was a problem in its own right. The educational system never had a chance to properly implement any change, it was overwhelmed by one wave of policy reforms after the other. In order to detect this pattern it was necessary to go beyond each specific reform and to analyze change over a span of 20 to 30 years.

(2) Uncover demi-regularities. Evaluators struggle to uncover causality, that is, finding and proving regular connections between causes and effects. But what is “regular”? One interpretation is that a regularity implies that an effect always (or at least with a very high degree of probability) follows an intervention, for example, that realty prices go up when interest rates fall, or inflation increases when fiscal policy regimes are expansive. However, as Lawson (1997) points out, there are also demi-regularities. They are explained as follows:

the occasional, but less than universal, actualisation of a mechanism or tendency, over a definite region of time-space. The patterning observed will not be strict as countervailing factors sometimes dominate or frequently co-determine the outcomes in a variable manner. But where demi-regularities are observed there is evidence of relatively enduring and identifiable tendencies in play.

(Lawson, 1997, p. 204)

Demi-regularities are often concealed and it requires a longer time-series, and a longer perspective to detect them.

(3) Show multiple and interconnected causal patterns. The notion of multiple causes implies that events are unlikely to be caused by one single previous act but are more likely to be the result of multiple causes. Juvenile delinquency is caused by a huge number of social, economic and cultural factors. Interventions can be based on addressing one factor, and that is perhaps not in itself a problem – as long as th...

Table of contents

• Cover
• Half Title
• Series Page
• Title Page
• Copyright Page
• Table of Contents
• List of figures
• List of tables
• Notes on contributors
• Foreword
• 1 Introduction
• 2 Historians’ Debates on Time and the Nature of Historical Research
• 3 Counting the Long Term in Evaluation
• 4 Contribution Analysis and the Long-term Perspective: Challenges and Opportunities
• 5 Using a Wide Lens to Take a Long View: How Integrating Systems Thinking into Evaluations Can Assist in Taking a Long-term Perspective
• 6 Reforms of Local Governments in Denmark
• 7 Reform Takes Time, but Politics Won’t Wait: The Scope for Long-term Evaluation in the Education Sector
• 8 Looking from a Local Place to Take the Long View in Evaluation
• 9 Complex Development Aid Evaluations: The Aid Quality Evaluation Framework
• 10 Why Take a Long View? How Formal and Informal Rules Influence the Choice of Time Perspectives in International Development Evaluations
• 11 Long-term Causal Inference
• 12 From Measuring Impact to Understanding Change
• 13 Concluding Remarks on Evaluating the Long Term: It Works but Can be Made to Work Better!
• Index