Handbook on Impact Evaluation

Quantitative Methods and Practices

Shahidur R. Khandker, Gayatri B. Koolwal, Hussain A. Samad

• Publisher: World Bank Publications
• Year: 2009
• eBook ISBN: 9780821380291
• Edition: 0
• Topic: Economics
• Subtopic: Development Economics
• Index: Economics

Public programs are designed to reach certain goals and beneficiaries. Methods to understand whether such programs actually work, as well as the level and nature of impacts on intended beneficiaries, are main themes of this book. Has the Grameen Bank, for example, succeeded in lowering consumption poverty among the rural poor in Bangladesh? Can conditional cash transfer programs in Mexico and Latin America improve health and schooling outcomes for poor women and children? Does a new road actually raise welfare in a remote area in Tanzania, or is it a 'highway to nowhere'? This book reviews quantitative methods and models of impact evaluation. It begins by reviewing the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It then focuses on the experimental design of an impact evaluation, highlighting its strengths and shortcomings, followed by discussions on various non-experimental methods. The authors also cover methods to shed light on the nature and mechanisms by which different participants are benefiting from the program. For researchers interested in learning how to use these models with statistical software, the book also provides STATA exercises in the context of evaluating major microcredit programs in Bangladesh, such as the Grameen Bank. The framework presented in this book can be very useful for strengthening local capacity in impact evaluation among technicians and policymakers in charge of formulating, implementing, and evaluating programs to alleviate poverty and underdevelopment.

Table of contents

• Contents
• Foreword
• Preface
• About the Authors
• Abbreviations
• Part 1 Methods and Practices
• Part 2 Stata Exercises
• Answers to Chapter Questions
• Appendix: Programs and .do Files for Chapter 12–16 Exercises
• Index
• Box 2.1 Case Study: PROGRESA (Oportunidades) in Mexico
• Box 2.2 Case Study: Assessing the Social Impact of Rural Energy Services in Nepal
• Box 2.3 Case Study: The Indonesian Kecamatan Development Project
• Box 2.4 Case Study: Monitoring the Nutritional Objectives of the FONCODES Project in Peru
• Box 2.5 Case Study: Mixed Methods in Quantitative and Qualitative Approaches
• Box 2.6 Case Study: An Example of an Ex Ante Evaluation
• Box 3.1 Case Study: PROGRESA (Oportunidades)
• Box 3.2 Case Study: Using Lotteries to Measure Intent-to-Treat Impact
• Box 3.3 Case Study: Instrumenting in the Case of Partial Compliance
• Box 3.4 Case Study: Minimizing Statistical Bias Resulting from Selective Attrition
• Box 3.5 Case Study: Selecting the Level of Randomization to Account for Spillovers
• Box 3.6 Case Study: Measuring Impact Heterogeneity from a Randomized Program
• Box 3.7 Case Study: Effects of Conducting a Baseline
• Box 3.8 Case Study: Persistence of Unobserved Heterogeneity in a Randomized Program
• Box 4.1 Case Study: Steps in Creating a Matched Sample of Nonparticipants to Evaluate a Farmer-Field-School Program
• Box 4.2 Case Study: Use of PSM and Testing for Selection Bias
• Box 4.3 Case Study: Using Weighted Least Squares Regression in a Study of the Southwest China Poverty Reduction Project
• Box 5.1 Case Study: DD with Panel Data and Repeated Cross-Sections
• Box 5.2 Case Study: Accounting for Initial Conditions with a DD Estimator—Applications for Survey Data of Varying Lengths
• Box 5.3 Case Study: PSM with DD
• Box 5.4 Case Study: Triple-Difference Method—Trabajar Program in Argentina
• Box 6.1 Case Study: Using Geography of Program Placement as an Instrument in Bangladesh
• Box 6.2 Case Study: Different Approaches and IVs in Examining the Effects of Child Health on Schooling in Ghana
• Box 6.3 Case Study: A Cross-Section and Panel Data Analysis Using Eligibility Rules for Microfinance Participation in Bangladesh
• Box 6.4 Case Study: Using Policy Design as Instruments to Study Private Schooling in Pakistan
• Box 7.1 Case Study: Exploiting Eligibility Rules in Discontinuity Design in South Africa
• Box 7.2 Case Study: Returning to PROGRESA (Oportunidades)
• Box 7.3 Case Study: Nonexperimental Pipeline Evaluation in Argentina
• Box 8.1 Case Study: Average and Distributional Impacts of the SEECALINE Program in Madagascar
• Box 8.2 Case Study: The Canadian Self-Sufficiency Project
• Box 8.3 Case Study: Targeting the Ultra-Poor Program in Bangladesh
• Box 9.1 Case Study: Poverty Impacts of Trade Reform in China
• Box 9.2 Case Study: Effects of School Subsidies on Children’s Attendance under PROGRESA (Oportunidades) in Mexico: Comparing Ex Ante Predictions and Ex Post Estimates—Part 1
• Box 9.3 Case Study: Effects of School Subsidies on Children’s Attendance under PROGRESA (Oportunidades) in Mexico: Comparing Ex Ante Predictions and Ex Post Estimates—Part 2
• Box 9.4 Case Study: Effects of School Subsidies on Children’s Attendance under Bolsa Escola in Brazil
• Figure 2.1 Monitoring and Evaluation Framework
• Figure 2.A Levels of Information Collection and Aggregation
• Figure 2.B Building up of Key Performance Indicators: Project Stage Details
• Figure 2.2 Evaluation Using a With-and-Without Comparison
• Figure 2.3 Evaluation Using a Before-and-After Comparison
• Figure 3.1 The Ideal Experiment with an Equivalent Control Group
• Figure 4.1 Example of Common Support
• Figure 4.2 Example of Poor Balancing and Weak Common Support
• Figure 5.1 An Example of DD
• Figure 5.2 Time-Varying Unobserved Heterogeneity
• Figure 7.1 Outcomes before Program Intervention
• Figure 7.2 Outcomes after Program Intervention
• Figure 7.3 Using a Tie-Breaking Experiment
• Figure 7.4 Multiple Cutoff Points
• Figure 8.1 Locally Weighted Regressions, Rural Development Program Road Project, Bangladesh
• Figure 11.1 Variables in the 1998/99 Data Set
• Figure 11.2 The Stata Computing Environment
• Table 11.1 Relational and Logical Operators Used in Stata