Implementation Monitoring and Process Evaluation

Ruth P. Saunders

• Publisher: SAGE Publications, Inc
• Year: 2015
• Print ISBN: 9781483308098
• eBook ISBN: 9781506305035
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

This practical guide helps readers understand and use the steps that program planners and evaluators take in implementing and monitoring a new program, policy, or practice in an organizational setting. The book covers the entire process, from planning, to carrying out the plan, and summarizing, reporting, and using the results. A wide range of real-world examples in the book are drawn from health, education, non-profit organizations, and public administration, and an extended case study, Your Turn boxes, and worksheet templates help readers apply concepts to their own projects. Ideal for practitioners, researchers, and students, this book can be used as a primary text for a process evaluation or an implementation monitoring course or as a supplemental text in a broader program evaluation course.

Step 1 Conduct Preimplementation Planning

Learning-Objectives

By the end of the chapter you will be familiar with

1. The basic program planning and evaluation planning process
2. Important considerations for enhancing the effectiveness of the program planning and evaluation planning process

The planning team, which consists of program planners and evaluation planners, must develop the new program, policy, or practice before it can be adopted, implemented, and evaluated in the organizational setting. This optimally happens through a multistep, systematic planning process in which the planning team develops a written plan, prospectus, or proposal that is then submitted for approval or funding. Many resources describe and prescribe the basic program planning and evaluation planning processes for program development in detail (see Resource Box 1.1), so this chapter provides an overview of this process.

Ideally, the program and evaluation plans are developed together in a process best described as iterative, building, and evolving over time, and both are completed prior to program, policy, or practice implementation. This is the ideal scenario, however; reality is far messier. For example, sometimes detailed planning has not been conducted prior to initiating implementation, or no program plan exists prior to developing an evaluation plan. In these cases, the evaluation members of the planning team must work with the implementers to understand the key elements and objectives of the program before developing an evaluation plan. Even though this situation is common, it is not optimal, particularly for implementation monitoring planning that relies on knowledge of many of the details about the program, policy, or practice. Therefore, if preimplementation planning has not been conducted or has only been partially conducted, it will be important for the planning team to conduct this basic planning step before proceeding.

Effective program, policy, or practice innovation plans include an assessment of priority audience needs and interests, preferably using an established framework. The priority audience is the intended program, policy, or practice beneficiaries. This systematic assessment may be oriented toward “objective” needs of the priority audience, such as needs identified through state or national data or a population-specific assessment or survey. Conversely, it may be oriented toward “perceived” needs of the priority audience that are identified through a participatory process or a survey with the population of interest. The former tend to be more expert driven in practice and minimally involve stakeholders, whereas the latter tend to be more facilitative, participatory processes with greater stakeholder involvement. Many methods combine elements from both of these approaches. The material presented in this textbook applies to both expert-driven and stakeholder-driven approaches in all organizational settings, and also pertains to evaluation research and evaluation practice approaches.

Resource Box 1.1

Program Planning and Evaluation Planning

Program Planning

Bartholomew, L. K., Parcel, G. S., Kok, G., & Gottlieb, N. H. (2006). Planning health promotion programs: An intervention mapping approach (2nd ed.). San Francisco, CA: Jossey-Bass.

Green, L. W., & Kreuter, M. W. (2005). Health program planning: An educational and ecological approach (4th ed.). New York, NY: McGraw-Hill Publishing.

Issel, L. M. (2008). Health promotion program planning and evaluation: A practical, systematic approach for community health (2nd ed.). Sudbury, MA: Jones & Bartlett Publishers.

RE-AIM program planning model. Retrieved from http://www.re-aim.org/

Evaluation Planning

Grembowski, D. (2001). The practice of health program evaluation. Thousand Oaks, CA: Sage.

King, J. A., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to assess program implementation. Thousand Oaks, CA: Sage.

McDavid, J. C., Huse, I., & Hawthorne, L. R. L. (2013). Program evaluation and performance measurement: An introduction to practice (2nd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (2011). Essentials of utilization-focused evaluation. Thousand Oaks, CA: Sage.

Pawson, R. (2013). The science of evaluation. Thousand Oaks, CA: Sage.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed). Thousand Oaks, CA: Sage.

Ryan, K. E., & Cousins, J. B. (Eds.). (2009). The SAGE international handbook of educational evaluation. Thousand Oaks, CA: Sage.

Participatory and Stakeholder Approaches to Program and Evaluation Planning

Centers for Disease Control and Prevention. (2012, September). A framework for program evaluation. Retrieved from http://www.cdc.gov/eval/framework/index.htm

Chen, H.-T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd ed.). Thousand Oaks, CA: Sage.

Chinman, M., Imm, P., & Wandersman, A. (2004). Getting To Outcomes™ 2004. Promoting accountability through methods and tools for planning, implementation, and evaluation. Rand Technical Report. Retrieved from http://130.154.3.8/pubs/technical_reports/TR101.html

Fetterman, D. M., & Wandersman, A. (2005). Empowerment evaluation principles in practice. New York, NY: The Guilford Press.

Minkler, M., & Wallerstein, N. (Eds.). (2008). Community-based participatory research for health: From process to outcomes. San Francisco: CA. Wiley.

An Overview of the Basic Program Planning and Evaluation Planning Process

Overview of Program Planning Process

As depicted by the vertical arrows in Figure 1.1, basic program planning is a series of assessment activities occurring at multiple levels accompanied by planning activities appropriate to that level and culminating in the development of an objective. Planning typically begins with the activities shown at the top of Figure 1.1, in which the planning team assesses needs and establishes outcome objectives. Outcomes may be related to distal or proximal conditions or factors. For example, health outcomes may include distal changes in conditions such as diabetes or heart disease or risk factors such as total serum cholesterol or high blood pressure. Alternatively, health outcomes may include proximal behaviors such as physical activity or diet that contribute to outcomes. Similarly, in education settings, distal outcomes may include graduation rates or employment, whereas proximal outcomes may include academic performance or skills. Depending on the population, scope, and time frame of the program, policy, or practice change, project outcomes may focus on the behavior, performance, and/or skill levels or on more distal conditions and outcomes. For example, in the Lifestyle Education for Activity Program (LEAP) the primary outcome was proximal: physical activity behavior in adolescent girls. The distal health condition, a cardiovascular disease, does not manifest until many years beyond adolescence and was therefore outside the scope of LEAP. See Green and Kreuter (2005) for more detailed guidance on planning health-related programs.

The next level of assessment and planning addresses factors that influence the selected outcome of interest. In LEAP, this entailed identifying influences on physical activity in adolescent girls, which was accomplished primarily through examining evidence in published literature. It revealed the importance of targeting factors such as perceived self-efficacy and social support for physical activity, as research shows these factors are consistently associated with physical activity in youth. That is, youth who have confidence in their ability to overcome common barriers to physical activity and who have social support and encouragement for being physically active are more likely to be physically active.

Following the selection of important factors that influence the outcome of interest, the next step in planning is to identify intervention strategies that are designed to impact these factors. This level of assessment includes examining evidence in both published literature and theory to identify appropriate methods and strategies. The LEAP team selected confidence-building and fun approaches that enabled girls to socialize during physical education class in order to develop physical activity self-efficacy and social support (Ward et al., 2006). Bartholomew and colleagues provide a detailed planning process for identifying theory- and evidence-based program methods and strategies (Bartholomew et al., 2006).

The final level of assessment and planning, depicted in the bottom arrow in Figure 1.1, pertains to project or program structure and processes that must be in place to get the work done in the setting. In organizational settings, this refers to organizational resources and materials, including staff members who are needed for program implementation. In LEAP, this level included the two full-time LEAP intervention staff who worked with schools, as well as school resources such as space, scheduling, and appropriate equipment. That is, the school provided physical education within an appropriately equipped gymnasium, and qualified personnel such as physical education teachers and other school staff who were trained to carry out LEAP conduct the activities with the high school girls.

Overview of Evaluation Planning Process

Each level of assessment and planning, as shown in Figure 1.1, has a corresponding evaluation approach, depicted in the boxes to the right of the arrows. The outcome evaluation (top box in Figure 1.1) addresses questions, objectives, or aims about the extent to which the intervention impacts the primary outcome. The impact evaluation (second box from the top in Figure 1.1) addresses questions, objectives, or aims about the extent to which the intervention has a positive impact on factors that influence outcomes. Assessing the effects of the intervention on the outcome and influence variables is also considered a summative evaluation. Outcome and impact evaluation—both of which are examples of effectiveness evaluation and are beyond the focus of this textbook—deserve additional comment here. Various evaluation frameworks define these two terms differently. In the PRECEDE/PROCEED (Green & Kreuter, 2005) framework, distal effects are considered to be outcomes; that is, they take long periods of time to be realized, as in chronic diseases that take nearly a lifetime to develop. In contrast, the CDC evaluation framework (Centers for Disease Control and Prevention, 1999) labels distal effects as impacts. In this textbook, both words are used together (outcome/impact) to refer to the proposed indicator for program effectiveness at distal and proximal levels.

Implementation monitoring (third box from the top in Figure 1.1) refers to methods used during active program implementation to monitor the ongoing implementation process and context. Plan quality and resource assessment (bottom box in Figure 1.1) encompass methods employed during preimplementation to assess the quality of the program plan and materials as well as the technical and cultural competence of program providers. Implementation monitoring and process evaluation methods may be used in a formative evaluation to further develop a program and/or to keep a developed program on track, although it is possible to use implementation monitoring and process data in a summative manner, as well.

Figure 1.1 Overview of Sequence of Program Planning and Evaluation Planning Process

Important Considerations for Enhancing the Effectiveness of the Program Planning and Evaluation Planning Process

Three important considerations for improving the quality of the program, the evaluation planning processes, and the planning products of these processes are as follows:

• Effective program, policy, or practice change entails extensive collaboration with people from the implementing settings.
• Intervention planning is optimally guided by a conceptual framework that includes an understanding of the causal mechanisms that will enable the program, policy, or practice to produce expected outcomes/impacts.
• Program planning, implementation, and evaluation are best conducted within collaborative interdisciplinary teams.

Effective program, policy, or practice change entails extensive collaboration with people from the implementing settings.

Research from diverse fields provides consistent evidence for the importance of collaborating with sett...

Table of contents

• Cover
• Half Title
• Acknowledgements
• Title Page
• Copyright Page
• Brief Contents
• Detailed Contents
• Preface
• Acknowledgments
• About the Author
• Publisher Note
• Introduction, Overview, and Perspectives
• Phase I Basic Program Planning
• Step 1 Conduct Preimplementation Planning
• Phase II Implementation Planning
• Step 2 Describe the Program, Policy, or Practice, Organizational Setting, and Broader Context
• Step 3 Determine Strategies for Facilitating Adoption of the Program, Policy, or Practice at the Organizational Level
• Step 4 Establish Complete and Acceptable Delivery/Installation of the Program, Policy, or Practice
• Step 5 Develop Strategies for Facilitating Program, Policy, or Practice Implementation and Sustainability
• Step 6 Develop or Update the Action Model, Action Plan, and Logic Model to Integrate Planning
• Phase III Implementation Monitoring Planning
• Step 7 Develop Initial Implementation Monitoring Questions
• Step 8 Choose Implementation Monitoring Methods and Compile the Comprehensive Implementation Monitoring Plan
• Phase IV Implementation
• Step 9 Implement the Program, Policy, or Practice and Use Implementation Data for Formative Purposes
• Step 10 Collect and Manage Implementation Data
• Phase V Analysis/Synthesis, Reporting, and Use
• Step 11 Analyze and Synthesize Implementation Monitoring Data
• Step 12 Report and Use Implementation Data
• Future Directions
• Glossary
• References
• Index
• Publisher Note