Evaluation as a discipline, in recent decades, has progressed from providing answers to questions like economic viability, cost-effectiveness, and efficiency to questions that are critical for effective planning, financing, design, implementation, and success of a program. Today evaluation can provide valuable assistance in defining a problem, identifying program targets, designing of interventions, identifying winners and losers and organizational strengths and weaknesses, assessing the quality of interventions and performance of program delivery and its impact, suggesting modifications and alterations, and ultimately guiding a program or project to its successful end. Evaluation is no longer just a tool for program or project appraisals but also an important tool in operational management. It is not just a snapshot of the end; it is also the means to the end (see also Williams and Giardina 1993). This book provides an easily comprehensible and comprehensive survey of tools of analysis that are used in the evaluation literature to evaluate public projects, programs, policies, and policy analysis and advice. The following paragraphs provide an overview of the book.
Chapter 2 by Deb and Shah provides a brief survey of the program evaluation methods, their objectives, strengths, and weaknesses. The methods have been presented in a manner that reflects the changes in outlook towards public programs and the changing role of evaluation. Accordingly, we have first presented methods like cost-benefit analysis, cost-effectiveness analysis, the social marginal cost of funds analysis, and data envelopment analysis that are appropriate for guiding efficient allocation and utilization of resources. This is followed by a discussion on multiple-objective evaluation, which is much more holistic. Apart from efficiency, it addresses issues like the relevance of a program, effectiveness of a program in achieving its objectives and sustainability of the program benefits. Newer multi-criteria approaches such as the Iron Triangle, Alternate Service Delivery Framework (ASDF), and the Results-Oriented Management and Evaluation (ROME) are briefly sketched. The Iron Triangle notes three important constraints faced by public managersâtime, cost, and quality. The ASDF brings into sharper focus the role of government as a catalyst in managing and coordinating service delivery by government and beyond government providers. ROME is particularly noteworthy for recognizing the role of citizens as the principals and various orders of governments as their agents. It provides an integrated approach to managing for results, citizens-based evaluation of those results, and the processes to hold the government to account by citizens. Finally, the authors outline the theory-based evaluation approach. It is a relatively new concept in evaluation literature where the focus is not just on whether a program succeeds or fails but also on how and why a program succeeds or fails.
The chapter concludes that the methods discussed serve different purposes. Methods like cost-benefit analysis, cost-effectiveness analysis, and data envelopment analysis address the question of efficiency in the allocation and utilization of funds. In multiple-objective evaluation, the emphasis is more on accountability in public sector programs, the effectiveness of programs, and the sustainability of program benefits. The evaluation is concerned with issues like identification of program beneficiaries, assessing their requirements, tailoring of interventions to meet those requirements, monitoring of interventions to ensure that the appropriate interventions are being delivered to the participants, and finally the overall effectiveness of the interventions in achieving the program objectives. So, a multiple-objectives evaluation plays a big role in program planning, design, and implementation. A theory-based evaluation assigns an even bigger role to an evaluation in public programs. It goes deeper into the mechanism through which the interventions bring about the desired effects. It analyzes the causal links between interventions and outcomes. So, instead of passing a summative judgment on whether a program succeeded or failed in achieving its objectives, the theory-based evaluation shows why it succeeded or failed. Thus, it contributes to the development of more effective programs in the future.
The authors argue that evaluation is different from other social research in the sense that it derives its questions from policymakers, program sponsors, program managers, and stakeholders. So, the applicability of any specific evaluation method depends on the questions that the evaluator has been asked to address. When the evaluation question is deciding upon alternative interventions aimed at producing similar effects, cost-effectiveness analysis might be more suitable. But it is not useful at all when the problem is prioritizing among different programs addressing different problems. A cost-benefit analysis will be more appropriate in that case. Similarly, data envelopment analysis might not have the valuation problems associated with the cost-benefit or cost-effectiveness analysis. But its applicability is limited to comparing efficiencies of similar programs only. For programs whose efficacies have already been established, multiple objective evaluations might be enough for performance evaluations. However, for pilot studies or for programs that have not been tested before, a theory-based evaluation is much more desirable. But it is also more time consuming and more expensive than any other evaluation methods. So, the choice of the evaluation method would also depend upon the availability of time and resources.
Chapter 3 by Robin Boadway summarizes the principles used to evaluate projects (such as individual investment projects, general expenditure programs, and the implementation of government policies) from an economic point of view. Following the principles of welfare economics, the objective of project evaluation is to measure the costs and benefits to individuals in society. The chapter begins with some theoretical perspectives on cost-benefit analysis and then proceeds to discuss the difficulties of implementing such an analysis and practical ways of dealing with these problems. He describes project evaluation as âan art, though one with scientific underpinnings.â
Broadly, the measurement of costs and benefits amounts to a measurement of individualsâ âwillingness to pay.â Two methods that have been developed in the theoretical literature are âcompensating variationâ and âequivalent variation,â depending on whether one wishes to use final or initial prices (respectively) for goods affected by the project. In addition, one must consider that projects that have costs and benefits spread over time must use a common set of prices to adjust for inflation and the time value of money. Policymakers may also wish to add distributive weightsâin other words weighting more heavily improvements in the incomes of the poorest. Once the discounted stream of costs and benefits is summed (yielding the Net Present Value, or NPV), the decision rule for project approval is simply determined by a (positive) sign of the NPV. Finally, the risk and uncertainty of outcomes should also be included. Other, similar techniques for project evaluation include the benefit-cost ratio and the internal rate of return methods. However, these alternative measures have some problems and may rank projects differently than the NPV criterion.
While the NPV method is in principle the same as is used by the private sector to guide the choice of investment decisions, the implementation of this procedure differs in some important ways when applied to the public sector. This is because the public sector must take into consideration: the marginal social values or shadow prices (rather than just market prices, since markets may be distorted) of inputs including labor, capital, and foreign exchange, and the impact of externalities such as pollution, general equilibrium effects of the project, valuation of intangible benefits and costs (such as time saved due to public transport), excess burden of public financing (due to distortions of the tax systemâthe so-called marginal cost of public funds, or MCF), the social discount rate, and social considerations (such as equity or protection of special groups). Each of these issues is considered in turn in this chapter.
Chapter 4 by Bev Dahlby probes more deeply into an evaluation tool that received brief treatment in Chaps. 1 and 2âthe Marginal Cost of Public Funds (MCF ), or the loss to consumers and producers caused by raising an additional dollar of tax revenue. Taxes impose a cost on the economy if they alter taxpayersâ consumption, production, and asset allocation decisions, leading to a less-efficient allocation of resources. Raising an additional dollar of tax revenue costs the private sector more than a dollar if the allocation of resources in the economy becomes more distorted. The marginal cost of public funds, MCF, is a measure of the cost imposed on the private sector in raising an additional dollar of tax revenue. Dahlby argues that the marginal cost of public funds should be used in evaluating the opportunity cost of financing public sector expenditures. It also provides a guide for tax reform by revealing which taxes impose the greatest welfare losses in generating additional revenues. The MCF can also be used to measure the gains from tax reforms that shift the burden from the high-cost tax bases to ones with lower costs.
In this chapter, Dahlby introduces the concept of the marginal cost of public funds (MCF ), examples of how the MCFs can be measured, and examples of how they can be applied to guide tax reform and public expenditure policies. In keeping with the theme of this volume, the emphasis is on using the MCF as a tool for public policy analysis. Two practical examples demonstrate the use of this evaluation tool in public policy analysis and evaluation. The first is an assessment of the gains from a tax reform that shifts some of the tax burdens from corporate to the personal income tax base in Alberta, Canada. The second example illustrates how the MCF can be used to derive optimal matching rates for intergovernmental infrastructure grants in a federation. In presenting these applications, the author makes a convincing case of the critical importance of the MCF as a tool for the evaluation of tax and expenditure policies.
Theory-based evaluation traces the factors that contributed to a specific outcome for the project. This a challenging task for project evaluation as it requires establishing a causal chain based upon theoretical considerations and analyzing data on various links in this causal chain. Ewa Tomaszewska in Chap. 5 provides a guidebook for those interested in conducting case studies in project evaluation determining the impact of various projects in combating corruption. The author provides a conceptual framework and data requirements for such evaluation case studies for privation programs, judicial and legal reforms, civil service reforms, trade liberalization, tax administration reforms, and direct anticorruption activities (anti-corruption agencies), the Office of Ombudsman, transparency rules and decentralization.
The impact of corruption on public service delivery performance and poverty alleviation is widely recognized. A wide consensus has also recently emerged that corruption is a symptom of failed governance and hence curtailing corruption requires addressing the causes of misgovernance. Nevertheless, the menu of potential actions to curtail corruption is very large so a framework is needed that provides guidance on ordering potential actions. Prioritization of various actions depends on both the conceptual and empirical views of what works and what does not work in the context of particular countries. Such a framework is also needed for evaluating country anti-corruption programs and policies. Chapter 6 by Huther and Shah proposes a framework for such evaluations. The chapter concludes that path dependency is critical in determining the relative efficacy of various anti-corruption programs. For example, in a largely corruption-free environment, anti-corruption agencies, ethics offices, and ombudsmen serve to enhance the standards of accountability. In countries with endemic corruption, the same institutions serve a function in form only and not in substance. Under a best-case scenario, these institutions might be helpful, but the more likely outcome is that they help to preserve the existing system of social injustice. Successful anti-corruption programs are those which address the underlying governance failures, resulting in lower opportunities for gain and a greater likelihood of sanctions. Thus, programs must be targeted to a countryâs existing quality of governance. Past experiences of the industrialized world confirm these conclusions since, without exception, these countries did not achieve a reduction in corruption by introducing technocratic solutions but, rather, by encouraging a sense of public duty among officials through accountability for result...