Part 1:
An Overview of Methods, Measures and Programs
The three chapters in this section lay the groundwork for the analysis: Chapter 1 begins with a discussion of methods of program evaluation. The omitted variables, endogeneity, and selection biases that frequently complicate non-experimental evaluations are explained, along with many of the problems involved in conducting social experiments. The chapter also outlines several approaches that have been developed for dealing with these problems. All involve strong assumptions, and the methods are not equally suitable for all problems. It is important to choose assumptions that are plausible in the context of the problem to be addressed, and to check that the results are robust to changes in these assumptions.
Chapter 2 makes the point that child well-being is a multi-dimensional concept and discusses possible measures. Examples of data sets that contain these measures are given in the Data Appendix.
Chapter 3 provides an overview of the main federal programs that benefit children. The largest federal programs for children are AFDC, Food Stamps, and Medicaid. Expenditures on AFDC have fallen over time, expenditures on Food Stamps have risen somewhat, and expenditures on Medicaid have shot up due to the increasing costs of medical care. Since 1975, the fastest growing programs in terms of both caseloads and expenditures have been WIC and the Earned Income Tax Credit, which can be thought of as a transfer program for working parents. Head Start also showed rapid growth. These patterns demonstrate the shift away from unrestricted transfers in the form of AFDC payments and towards more restricted transfers and programs targeted directly to children.
1.
METHODS FOR EVALUATING WELFARE PROGRAMS
Families on welfare are poor and likely to be disadvantaged in other respects. Hence, it should not be surprising to find that their children also tend to be disadvantaged. In fact, it is possible that a familyâs participation in a welfare program could increase the well-being of a child substantially, and still leave that child worse off than an average child. In order to isolate the effects of welfare programs on children, we need to control for all relevant differences in the backgrounds and characteristics of participants and nonparticipants.
The standard way to do this has been to control for observable differences, such as differences in parental education and income, using Ordinary Least Squares regression (OLS), a procedure that is available in virtually all statistical software packages (see Theil, 1971, for a discussion of the properties and potential biases of OLS). This procedure attempts to compare participants and non-participants who have the same observable characteristics.
OLS estimates of program effects will be unbiased as long as the mean of any unobserved variables (which make up the âerrorâ term) is zero, and as long as unobserved variables are not correlated with the observable characteristics included as explanatory variables in the regression. These are important limitations because even the most comprehensive data set is likely to omit some potentially relevant characteristics of the parent or child.
Suppose for example, that children in families on welfare are likely to attend inferior schools, and that the quality of school is unmeasured, or poorly measured. Assume also that welfare participation has no effect on test scores, but that scores increase with school quality. If we estimate an OLS regression of test scores on welfare participation and omit school quality, we may erroneously conclude that welfare participation has a negative effect on test scores because the estimated coefficient on the indicator for welfare participation will incorporate the negative effect of inferior school quality. This problem is referred to as omitted variables bias. Note that if school quality were not related to welfare participation, then omitting it would not cause any biasâonly omitted variables that are correlated with program participation cause problems.
A related concern is that other key variables may be determined jointly with program participation and with the outcome in question. Such variables are said to be endogenous. For example, suppose we wish to determine the effect of AFDC participation during pregnancy on birthweight. Good prenatal care is associated with higher birthweights, and women on AFDC may have access to better prenatal care because they are covered by Medicaid. However, if we include both AFDC participation and adequacy of prenatal care in our model of birthweight, we may erroneously conclude that only prenatal care affects birthweight. The problem arises because prenatal care is treated as if it were a fixed, pre-assigned variable, rather than a variable that is chosen by the mother and affected by AFDC participation.
A third problem arises when researchers try to make inferences about the effects a program would have on a broad population of participants using only information about people who are selected into the program. For example, broad bi-partisan support exists for extending the Head Start program to serve children in all poor families. If current enrollees were a random sample of all poor children, then we would generalize from their experiences to the population of all poor children.
However, faced with limited resources and discretion about who gets into the program, it is unlikely that program administrators will choose a random sample of all applicants: They may choose either to target their resources to the neediest children, or to target them to relatively better off children who are judged to be most likely to benefit. Once again, if we can observe the criteria that are being used to select children into the program, then we can control for these characteristics when evaluating program effects. But in most cases, it is likely that selections are being made using variables that researchers do not observe.
Given that we are unlikely to ever have data sets that include all of the relevant variables, a number of approaches have been developed to deal with the problems of omitted variables bias, endogeneity, and selection. These include social experiments, instrumental variables techniques, ânatural experimentsâ, selection corrections, and fixed or random effects estimators. These approaches are not all equally suitable for all questions or data sets. The purpose of this chapter is to provide an overview that emphasizes the underlying assumptions, and possible pitfalls involved in the use of each technique. References are given in each section for those who wish to see a more technical treatment.
A. Experiments
In principle, the effects of welfare programs could be evaluated in the same way that we evaluate the effects of a new drug: by conducting a randomized experiment. People would be randomly assigned to a treatment group and a control group. Random assignment would insure that there were no systematic differences in either the observable or unobservable characteristics of the two groups. Hence, the effect of the treatment could be deduced by comparing the means of the two groups.
One might wonder why it is necessary to randomly assign treatment and controls? Why not find a group of similar people who are not in the program and compare them to the âtreatmentâ group? This procedure is sometimes called a âquasi-experimentâ. Lalonde (1986) shows that this strategy can produce very misleading estimates. He begins with data from a true randomized experiment, which means that he knows the effect of the treat...