Applied Policy Research

Concepts and Cases

J. Fred Springer, Peter J. Haas, Allan Porowski

• Publisher: Routledge
• Year: 2017
• ISBN:
  9781135215408
• Edition: 2
• Topic: Politica e relazioni internazionali
• Subtopic: Politica pubblica

Where many textbooks on policy research focus on methodological and statistical theories, leaving students to wonder how they will apply those theories to future policy positions, this innovative textbook takes theories of policy research and puts them into practice, demystifying the subject by translating it into real-world situations in which students can actively engage. Beginning with an orientation and overview of policy research, outlining the processes of policy analysis and evaluation from start to finish, Applied Policy Research, 2e walks students through an examination of case studies to demonstrate how these theories play out in real policy situations. New to this edition:

• 
  
  

• A rewritten Part I that includes several new chapters incorporating the latest developments in applicable policy research design, implementation, and products to provide a framework for conducting policy research.

• 
  
  

• A matrix at the start of Part II to easily identify how each of the fifteen case-study chapters correspond with concepts and topics presented in Part I, showing the reader where to look for a specific real-life example of a given topic or concept. Each case is drawn from real instances of policy research to provide students with an opportunity to consider and learn how to grapple with the challenges posed by the needs of public programs and agencies. Cases include local, state, and nonprofit agencies as well as federal-state-local intergovernmental "hybrids."

• 
  
  

• Each chapter is presented in a uniform format: (1) a detailed description of a policy research problem; (2) a discussion of the unique challenges posed by the problem; (3) a description of the policy research techniques used; (4) a summary of the outcomes or conclusions associated with the research as it was conducted; and (5) conclusions about the implications or lessons for policy research.

Illustrative figures help students understand the stages of policy research, and end-of-chapter tools such as discussion questions, assignments and activities, and case studies "at a glance" help students master not only the particulars of each case but the broader skills needed in future research. Applied Policy Research, Second Edition will be essential reading in all policy research courses with a focus on practical outcomes and student preparation for public service.

Part I

Principles of Policy Research

J. Fred Springer, Peter J. Haas, and Allan Porowski

1 Introduction to Policy Research

How is policy really made? Popular views of the workings of American government are characterized by diverse and sometimes contradictory stereotypes—politicians cutting deals in back rooms, bureaucrats putting rules before public interest, and technocrats who coldly pursue numbers and logic to the exclusion of people. This book addresses policy analysts’ role in this process, painting a realistic picture of what applied research brings to policy making. It cuts through the stereotypes that policy research is a purely technocratic endeavor and portrays the important place that analysts play in the give and take of policy making in American democracy.

Although policy analysts may not top the list of visible government workers, policy research has some harsh critics. Former U.S. Senator William Proxmire’s political calling card was his Golden Fleece award, awarded from 1975 to 1988. The senator often made the award to what he saw as particularly useless publicly funded studies. For example, the first award was to the National Science Foundation’s $84,000 study to find out why people fall in love (http://content.wisconsinhistory.org/cdm/ref/collection/tp/id/70852). From a different perspective, advocates for data-informed innovation still lament the “chilling effect on research” that the Golden Fleece Award exerted (Catmull and Wallace, 2014). Political criticism of policy-relevant research has not abated in the nearly thirty years since the Golden Fleece awards. Senator Tom Coburn recently ridiculed the National Science Foundation’s funding of a study concerning the impact of climate change on sea life. The study measured the effects of water temperature on the stamina of shrimp trudging along a moving surface. While this study produced important scientific results on climate change and ocean ecology, widely disseminated criticism made “shrimp on a treadmill” shorthand for government waste.

As is frequently the case with public debate, selective understanding of evidence and facts are in part responsible for contradictory positions. One of our major purposes in writing this book is to provide a more balanced, practical, and realistic understanding of how policy research is conducted and how it influences public decisions. This discussion may not enflame passions in the same way as incendiary commentary about politics, but it will lead to a more realistic understanding of the role that policy research plays in public decisions. This perspective should be particularly useful to students and practitioners of politics and public administration.

To provide a balanced discussion of ways in which policy research is actually produced and used in public decision making, we will include case studies that focus on policies and programs at the local and state levels, as well as studies that encompass the workings of the federal government. Multimillion-dollar studies by the federal government make excellent fodder for critics of policy research, but they are not typical of most policy research activity. The work of the policy analyst is usually done at a more modest level, financed by a small budget and limited resources in less visible settings: in state, county, or municipal government or even in the nonprofit sector, which implements many policies. The case studies and discussion presented in this text reflect the diverse programmatic milieu in which public employees and decision makers operate.

This chapter is the first of five that define policy research and introduce a framework for understanding its purpose, methods, and application. We begin by addressing five fundamental questions:

1. What is policy research? We will discuss how policy research differs from other social research and introduce common terms used to describe policy research.
2. How is policy research conducted? We will discuss basic procedures and methods that are used in policy research.
3. Who does policy research? We will discuss the kinds of jobs that policy researchers hold and the kinds of skills their work requires.
4. Why do policy research? We will discuss the purposes, mandates, and incentives that motivate policy researchers.
5. Does policy research improve public decision making? We will discuss who uses policy research, why it is used, and how it may help.

What Is Policy Research?

Although it is referred to in different ways, many employees (and consultants) of public agencies engage in policy research. “Policy research” is a catch-all term used to describe the many information-gathering and processing activities that public agencies engage in to facilitate decision making. It is both a skillful process and an evidence-based information product (Majchrzak and Markus, 2014, p. 2). Policy research may be referred to as policy analysis, program evaluation, needs assessment, performance monitoring, or other terms depending on the context in which it occurs. For example, policy analysis typically refers to the analysis of policy during the decision-making process, while program evaluation typically refers to the analysis of a program that puts a policy in place. More will be said about these distinctions, how they differ, and what they share in the pages ahead.

Policy research can happen at any point in the policy cycle, which has five steps (Fischer, Miller, and Sidney, 2006):

• Agenda Setting: Before a policy can be formed, the problem that is to be addressed by the policy must be defined and balanced against other competing priorities. Policy research can be conducted at this stage to assess the severity of the problem to be addressed, to assess which competing policy issue needs to be addressed first, or even to establish whether the problem exists in the first place.
• Policy Formulation: Before policies are passed, they must be formulated to address the problem in question. Policy researchers may do comparative research to assess the different costs and effects of each policy alternative (policy analysis) or research the key components of the policy to ensure that it is specified and will be implemented with fidelity.
• Decision Making: The actual passage of a policy may be guided by policy research—or it may not. While decision making is traditionally a political process, policy researchers may be drawn in to testify on pending legislation or help guide a bureaucratic process. The decision not to enact a policy can be considered policy making as well.
• Implementation: Once a policy is agreed on, there are a multitude of opportunities for policy research to assess how well it is being put into action. These implementation studies may assess how closely the policy as enacted conforms to policy intentions, examine how efficiently it is being put into action, or monitor performance benchmarks, such as the number of clients served.
• Evaluation: Using scientific methods to determine whether policies or programs actually produce their intended effects is a common type of policy research (Rossi et al., 2004). Evaluation takes many forms and is often required when policies are put into effect. Policy or program evaluation can assess short-term outcomes of a program, such as determining whether a new preschool program results in stronger reading readiness. It can also assess the long-term impacts of a policy—for example, whether the preschool program results in better reading performance among students in grades K–12. It can also assess the balance of costs and benefits of some programs. For example, the evaluation of a new business tax incentive can involve the calculation of its total economic benefit and comparing that figure to the total cost of public investment in the incentive.

Policy research must take many forms because policies take many forms, including statutes, laws, regulations, decisions, and government programs (Birkland, 2015). Despite the complexity of defining policy research, it has two clear distinguishing factors:

1. Policy research is problem driven (Birkland, 2015, referring to the work of Peter May). In other words, policy research is initiated by real-world problems that need to be solved.
2. Policy research is designed to produce actionable results (Hakim, 2000). Because policy research is expected to help solve real-world problems, the information it produces must have clear implications for what can be done—it must be “actionable.” Policy researchers do not make policy decisions. They do provide information that elected officials and public personnel can consider along with other inputs and considerations (e.g., politics, cost, feasibility, credibility of the research, effects of alternative strategies). This role in the decision-making process makes policy research particularly exciting—and complex.

Throughout this volume, you will see that the utility of policy research findings is not determined by the rigor of the underlying research or even the magnitude of impacts detected. Its usefulness also depends on framing and delivering findings in a way that helps decision makers formulate and deliver their arguments and ultimately to make decisions.

How Is Policy Research Conducted?

“Real-world” problems are complex. People respond differently to the same situations, and programs may be implemented differently across states, communities, or even agency offices. Outcomes may differ dramatically in different contexts, and differences in outcomes across contexts may be attributable to multiple interwoven influences. The good news is that the tools and methods available to policy researchers to account for this complexity are constantly improving—and their number is expanding. Aspiring policy researchers should be prepared to embark upon a lifetime of learning as methods and tools evolve. While the full scope of prominent methods and tools available to policy researchers cannot be conveyed in this introductory chapter, the following examples hint at their diversity and evolution.

• Experimental research: Experimental research is a long-standing pillar of scientific rigor in policy research. Experimental designs require random assignment of individuals or groups (e.g., classrooms) to either receive services or benefits enacted by a policy (the treatment group) or to not receive the intervention (the control group). By comparing data on subjects randomly assigned to the treatment or to the control group before and after the treatment is received, policy researchers can provide strong evidence on whether the intervention had an effect. This rigorous research method provides a model for determining whether policies or programs produce their intended impacts.
• Quasi-experimental research. By requiring the random assignment of the intervention, the experimental model clearly requires more control of the study setting than can be achieved in most policy settings. Quasi-experiments overcome ethical, legal, and logistic barriers to randomizing people by forming a comparison group similar to the treatment group. Comparison groups can be developed prior to the start of the study in a number of ways not requiring randomization, including recruiting clients on a wait list, selecting students in a similar school, or matching persons on demographic or other characteristics of treatment group members. The point of both experimental and quasi-experimental methods is to establish a comparison group to ascertain what would have happened in the absence of the intervention and compare that result with the outcomes of those who received an intervention. Quasi-experimental designs are far more common in policy research than experimental methods because they allow for adaptation of experimental logic to the realities of conducting studies in the real world of policy making and implementation.
• Rapid-cycle evaluation: To be useful in the rapidly changing world of policy setting and revision, information must be available when decisions need to be made. Rapid-cycle evaluation is a framework for conducting a rapid sequence of experimental or quasi-experimental research that tests whether operational changes improve results (Gold, Helms, and Guterman, 2011). Unlike traditional experimental and quasi-experimental studies, rapid-cycle evaluations typically take months rather than years to complete, and they tend to focus on incremental changes in operations instead of measuring the effects of entire programs. These rapid-cycle evaluation methods are gaining in popularity, since they get answers into policy makers’ hands much more quickly, which in turn increases the likelihood that the research will influence decisions.
• Nonexperimental research: Nonexperimental methods involve gathering information on intended policy effects before, during, and/or after an intervention from a single group of participants with no comparison group. Though this type of research is considered to be less scientifically rigorous than experimental or quasi-experimental research, it is still often used and can produce actionable information. This is especially true when quantitative (numeric) data is combined with qualitative (nonnumeric) information gathered in the field to help stakeholders understand why or how a policy or program has—or does not have—desired effects.
• Secondary data analysis: Using data already collected by programs or regularly collected for other purposes (e.g., surveys conducted to measure unemployment), policy researchers can assess changes in outcomes over time. These secondary data analyses can sometimes support quasi-experimental policy research (e.g., matching program participants to available data on nonparticipants before an intervention started and following their outcomes over time), or these analyses can be nonexperimental (e.g., drawing upon program records to assess trends among program participants only).
• Survey research: Policy researchers may survey beneficiaries of a program to determine whether the program had the intended outcomes. Surveys are popular and relevant because they can be tailored to nearly any situation. They provide direct answers to a variety of questions about how policy impacts individuals (e.g., “customer satisfaction” questions to determine how programs can better serve the public). This ability to get information straight from the source makes survey research widely used among policy researchers.
• Systematic reviews: Policies are much more defensible if they are grounded in evidence-based research. Policy researchers may review evidence across multiple studies to identify whether compelling evidence exists for the adoption of a program, policy, or practice. Meta-analysis is a commonly used statistical technique for combining findings across multiple studies. In recent years, many systematic reviews and repositories of evidence have been sponsored by both federal and private organizations, including the U.S. Department of Education’s What Works Clearinghouse (WWC), the U.S. Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR), and the U.S. Department of Health and Human Services’ National Registry of Evidence-Based Programs and Practices (NREPP).
• Site visits: When trying to understand the inner workings of a program or a policy, there is no substitute for seeing it in action. Site visits to interview the “front lines” of staff involved in service provision are nearly always enlightening experiences. “Ambient observation,” a fancy term for getting the feel of a place, can contribute significantly to the policy researcher’s understanding of how and why a program is shaped by its context. Whereas quantitative research can be useful for measuring the magnitude of a program or policy’s effect, site visits add important information for understanding how, why, and in what situations effects are being produced.
• Implementation studies: The field of implementation science is growing rapidly, and it is often coupled with experimental, quasi-experimental, and qualitative research to determine both whether the program being evaluated was implemented with fidelity (i.e., as intended), and what core components of the program are most important for achieving intended outcomes. Early implementation science focused on programs, but it is increasingly being applied to achieving systemic policy objectives such as integrating services and collaborating across agencies.
• Cost-benefit or cost-effectiveness analyses: A logical focus of policy research is to determine the “bang for the buck” that a certain policy or program will have. Whereas cost-benefit analysis focuses on the economic benefit of a program by quantifying the dollar value of all costs and benefits, cost-effectiveness analysis compares the costs of a program with the raw outcomes produced (e.g., cost per life saved). The difference between cost-benefit and cost-effectiveness analysis, then, lies in how benefits are...