Advocacy and Policy Change Evaluation

Theory and Practice

Annette Gardner, Claire Brindis

• Publisher: Stanford Business Books
• Year: 2017
• ISBN:
  9781503602335
• Edition: 1
• Topic: Politique et relations internationales
• Subtopic: Plaidoyer politique

This is the first book-length treatment of the concepts, designs, methods, and tools needed to conduct effective advocacy and policy change evaluations. By integrating insights from different disciplines, Part I provides a conceptual foundation for navigating advocacy tactics within today's turbulent policy landscape. Part II offers recommendations for developing appropriate evaluation designs and working with unique advocacy and policy change–oriented instruments. Part III turns toward opportunities and challenges in this growing field. In addition to describing actual designs and measures, the chapters includes suggestions for addressing the specific challenges of working in a policy setting, such as a long time horizon for achieving meaningful change.

To illuminate and advance this area of evaluation practice, the authors draw on over 30 years of evaluation experience; collective wisdom based on a new, large-scale survey of evaluators in the field; and in-depth case studies on diverse issues—from the environment, to public health, to human rights. Ideal for evaluators, change makers, and funders, this book is the definitive guide to advocacy and policy change evaluation.

Part 1

Useful Theories and Conceptual Models

CHAPTER 1

POLICY AND POLICYMAKING

Making a Difference

INTRODUCTION

Policy and policymaking permeates and shapes our daily lives, from mandating funding for public schools to regulating the disposal of hazardous materials. No one is untouched by public policies, and when well thought out and implemented, they are potent vehicles for social betterment. The policy process and its outcomes are the raison d’etre of government and a lens on the ongoing debate about the nature of societal problems and appropriate solutions. Public policymaking is also increasingly viewed as a venue for individuals, organizations, and groups to intervene and achieve system-wide change that heretofore was limited to the privileged few. But engaging in policy change, be it organizing a community to act on its own behalf or examining the implementation of a newly minted measure, bill, or budget, is a daunting undertaking under any circumstance. Understanding the challenges as well as opportunities for change are necessary first steps to designing a successful advocacy and policy change (APC) evaluation. In this chapter, we review the scholarship on public policy and describe the concepts that are important to advocacy and policy change evaluation practice, including the nature of policy change. We focus on public sector policymaking or decisions made by federal, state, local, and municipal governments, be it laws, regulatory measures, or funding priorities though many of the same principles apply outside of the public sector.

Through a tailored review of public policy scholarship, we aim to provide a handy reference for evaluators who are new to the policymaking process, and/or evaluating it and wanting for a bird’s eye view. While theory building is outside the purview of many advocacy and policy change evaluations, evaluators can still use theory to understand a complicated policy venue: the type of policy, models of the policy change mechanism, and the different policymaking venues. Be forewarned, this is a changing and somewhat chaotic arena, with vague boundaries and a weak theoretical foundation. Characterizing the steps that a policy goes through before it becomes law is not a particularly difficult task. However, understanding the political actors and institutions and their roles and relationships at each stage of the decision-making process is a different matter since they are less transparent and possibly in flux as environmental factors change, such as a change in administration. (Please note: These models and definitions speak primarily to U.S. public policymaking. However, we are intentionally broad in our inclusion and description of these concepts since many political systems share the same components of government, elections, organized interests, and a decision-making process.)

With this groundwork laid, we provide recommendations for incorporating policymaking concepts and models into evaluation practice. While incorporating a policy change model into an evaluation design can be challenging since stakeholders may have competing theories of change, some policy frameworks, such as the policymaking stage model, can be readily adapted to many different types of evaluations in different contexts. This and other models will be described in this chapter. As with many other types of evaluations, articulating and facilitating a shared understanding of the assumptions about the policymaking process is key in setting the parameters of an advocacy and policy change evaluation.

Last, to ground this discussion to the reality of policymaking, we tap into the findings from the 2014 Aspen/UCSF APC Evaluation Survey completed by 106 APC evaluators, as well as describe and compare the real-life policy issues that were targeted by advocates in the six advocacy and policy change evaluation cases that were developed for this book. While not representative of the universe of policy arenas and issues, the cases illustrate the diversity of policy types, geographic levels of decision-making, and venues where policymaking takes place.

WHAT IS POLICY?

There is no one definition of “public policy,” and it can be narrowly or broadly defined. For example, one classic political science definition, “Public policy is whatever governments choose to do or not to do” (Dye 2002, 1), is too broad to be of use to evaluators. It belies the complexity of the actual decision-making process. It also overlooks the role that values play and that in many cases, policymaking is deciding which values will prevail, such as whether responsibility for implementation of a new program should be done at the federal or state level or left to the private sector (Kraft and Furlong 2010). Political scientist Thomas Birkland (2001, 20) provides a list of attributes that are commonly used to define policy and that is more helpful for framing an advocacy and policy evaluation design:

• Policy is made in the public’s name.

• Policy is generally made or initiated by government.

• Policy is what government intends to do (is purposive) on behalf of the public.

• Policy is what the government chooses not to do.

• Policy is interpreted and implemented by public and private actors.

These attributes—the how, what, and where policy is made—and their application to evaluation practice are described in the next sections.

THE POLICYMAKING PROCESS

We look to the political science arena for models that we believe will help evaluators develop a more robust understanding of the policymaking process and the contextual factors that shape it. As explained by political scientist Thomas Dye (2002), these models describe a distinct way of thinking about policy, but they are not mutually exclusive and they can be combined to explain a policy’s trajectory. These models overlap in that they focus on representation, as well as the distribution of power and likelihood of incremental or radical policy change. They explain how public and private institutions, organized interests, and policymakers interact to produce and implement policies. In their broadest sense, they characterize the nature and role of individuals and groups, the role of information and beliefs in decision-making, the level of action, and activities at various stages of the policy process (Schlager and Blomquist 1996). Increasingly, public policy models take a systems approach and detail the relationships among these components and provide evaluators with a foundation on which they can base a theory of change as well as develop a sound contextual analysis.

The most encompassing model describes the policy process as a set of stages or the “policy cycle” (hereafter referred to as the “policy stage model”) framework. While not a perfect description of the policymaking process, we suggest that all evaluators incorporate aspects of this model into their evaluation design, particularly in the development of a theory of change and/or logic model. Drawing on the literature, we have created an annotated stage model that describes the opportunities for evaluators to better understand the policy landscape as well as key activities and events that can be incorporated into an APC evaluation logic model. Originally premised on the application of systems theory to explain the policy process, the stage model is a chronological rendering of interdependent stages in the policymaking process. Collectively, the stages are the subject matter of the policy studies arena within political science, and the goal is to study the procedures and processes by which policy is made (Theodoulou 1995; Dye 2002). Each of these stages has its own body of scholarship and research questions, with some attracting more attention than others.

Stage 1, Problem Recognition, is the stage when the public makes demands for government action, and policy problems are recognized by policymakers as requiring action. If recognized as legitimate, problems, such as global warming, workplace safety, or equity issues, then become issues. At this stage, it is important to determine who identifies a problem, the role of public opinion and the media, and the nature of the problem, such as whether it is a new problem (say, a crisis), or broad and systemic (say, lack of employment opportunities). Polling information, media coverage, and policymaker communications are useful for understanding the history, context, and saliency of the problem to the public, advocates, and decision-makers. It is also helpful to understand the length of time it takes for a problem to reach a critical point and policymaker acknowledgement that action is justified. It is a worthwhile investment in effort since in all likelihood the debate and voices raised at this stage will have a ripple effect and shape the remaining the stages.

Stage 2, Agenda Setting, is considered the “make or break” stage, and if problems do not make it on the policy agenda, the process stops there. During this stage, an issue is given serious consideration by policymakers, launching the search for a policy solution. It is useful to specify who identifies the issues that gain policymakers’ attention, such as professional associations, the media, decision-makers, influential individuals, as well as the mechanism for narrowing the set of issues that policymakers will act on. Additionally, issues that are kept off the agenda are worth noting, such as those like strict gun control that are opposed by powerful private sector interests.

Stage 3, Policy Formulation, is the development of acceptable policy proposals to address the problem(s) identified during Stage 1 by interest groups, policymakers, and think tanks. It is the stage when policy analyses are conducted to determine the relative merits of one policy over another with the intent to improve policies. This stage provides detailed information on the understanding of the causal sequence between the problem and the policy solution.

Stage 4, Policy Adoption, is the act of selecting which policy proposal will be enacted into law by decision-makers or the courts. It is also the stage of Policy Legitimation or the process of building support for adoption, including bargaining, competition, persuasion, and compromise. A policy’s progress can be tracked as it winds its way through hearings, voting, and signing.

Stage 5, Policy Implementation, is when a bill becomes a law and is translated into guidelines or rules and regulations by bureaucracies. New legislation, such as agency activities and public expenditures, is implemented through public programs and involves federal, state, or local government. This stage provides information on the legal and technical aspects of a policy and a blueprint for action that can be monitored and used in assessing achievement of policy goals. Recently, there has been increased use of implementation science to assess Stage 5 since a poorly executed policy may be the same or worse than no policy at all. This is also a stage where there is the possibility of greater equality of representation across advocates, such as the public comment period and participating on agency committees. However, the evidence on legislative and administrative lobbying at the federal and state levels in the United States suggests that while lobbyists of all types are very active during Stage 5, business interests dominate both arenas (Boehmke, Gailmard, and Patty 2013).

Last, Stage 6, Policy Evaluation, is the systematic evaluation of a policy—its actual impacts, costs, and whether or not it achieved its intended results. It is an opportune time to assess whether or not policy was implemented as intended and with the necessary resources and system capacity to fulfill the policy’s intent. The objective is to inform policymakers on whether or not they made the right choice as well as the effects of the policy, and should be familiar territory to APC evaluators. Standard evaluation design principles apply, but the context poses particular challenges, such as government concerns about negative findings and limited use of findings by decision-makers and government agencies (Dye 2002). Evaluator Carol Weiss (1999) and other evaluators describe the interface between evaluation and public policy and the limited avenues and opportunities for educating policymakers. While the information-rich environment and the confluence of interests and ideology precludes full consideration of evaluation findings by policymakers, there is still the opportunity to play more than a symbolic role and educate decision-makers and the public on a policy’s impact.

The literature suggests high agreement about the stages of the model to the extent that these are the primary stages, though even this varies somewhat, such as having Agenda Setting as Stage 1, instead of Problem Identification. There are also suggestions for expanding the stage model further, such as adding a new Stage 1, Building Advocacy Capacity, to describe the groundwork that is developed prior to the identification of a problem (Brindis, Geierstanger, and Faxio 2009).

There are issues with the stage approach that evaluators should be mindful of. Political scientists Hank C. Jenkins-Smith and Paul A. Sabatier (1994) point out that the model may be “descriptively inaccurate” for policies that go through the stages in a different order or bypass one or more stages. For example, the role of media in influencing public and policymaker opinion may straddle Stage 1, Problem Recognition, and Stage 2, Agenda Setting. Or, depending on the saliency and conflict surrounding an issue, the media may be active at all stages. For example, the 2010 Affordable Care Act, or Obamacare, attracted widespread public and policymaker attention and figured prominently in the media pre- and postpassage. Additionally, these stages correspond to decision-making points, but not all initiatives advance to a decision. Critics also argue that a stage approach is purely descriptive and not a causal model that explains why something happens or when something is likely to happen.

However, while not a perfect characterization of how policy is crafted, the stage model is nonetheless a descriptive and flexible heuristic. At a minimum, we recommend that you align your policy issue with the stage (or stages) of the model and examine the scholarship for that particular stage. For example, if you are looking at advocacy postpassage, focus on the policy implementation and policy evaluation literature. Second, examine the analytical aspects of the stage model that can inform your situation, in particular the Policy Formulation and Policy Evaluation stages, when policy analyses are likely to be undertaken and can provide detailed information on policy options and the likelihood of policy sustainability (Chelimsky 2014). These are also the stages that APC evaluators are well positioned to improve the quality of a policy or program.

Last, the stage model is also useful for identifying decision points that are opportunities for advocates to make themselves heard, such as during the public comment period. These decision points also represent evaluation opportunities. For example, assessing the level of funding allocated to a policy during the Policy Implementation stage may reflect the political savvyness of the advocacy community, as without sufficient resources, the policy passage may remain largely a paper tiger. Advocates are highly knowledgeable of the policymaking process and its opportunities, making it even more important that evaluators familiarize themselves with the decision points of the process.

Policy Change Models

Increasingly, in APC evaluation practice, it is important to give some thought to the models that characterize the change aspect of policy change, particularly when developing a theory of change. A policy can undergo significant change as it winds its way through the decision-making process and may not resemble its original form by the time it is adopted. However, upon closer examination, we see that this is really a much more nuanced discussion. Policy change may be incremental and consist of small changes in existing policies or passage of noncontroversial policies. Described as “muddling through” by political scientist Charles E. Lindblom (1979), the incrementalism model explains why policies are rarely terminated. Nor are radically new policies passed. This is due in part to the stability of a policy arena, a desirable feature, but it is also reflects the difficulties in bringing about more radical change. Overlapping policy jurisdictions, contested problems and solutions, policymaker reluctance to support new and potentially risky policy shifts, and ideological stalemates, particularly in market-oriented systems all act as significant impediments to reform. U.S. health care reform and the passage of the Affordable Care Act was a lesson in incrementalism (and patience), taking forty-plus years to pass. Immigration reform has proceeded slowly as well in the United States.

While a useful approach to studying the small steps that lead up to a policy change, the incremental model lacks a strategic orientation to policy change and does not account for the radical reforms that do take place or need to take place, such as climate change policy. Moreover, policy change can at times be swift and result in transformative change. This can be due to events, conflict, and economic crises, such as the 9/11 attacks and the 2008 Great Recession. There are a few models that describe this sudden form of policy change, including the “cyclical” model, such as the shifting back and forth between privileging public interests (government expansion) over private interests (minimal gov...