Practical Program Evaluation for Criminal Justice

Gennaro Vito, George Higgins

• Publisher: Routledge
• Year: 2014
• ISBN:
  9781317521914
• Edition: 1
• Topic: Law
• Subtopic: Criminal Law
• Index: Law

Practical Program Evaluation for Criminal Justice shows readers how to apply the principles of fiscal responsibility, accountability, and evidence-based practice to criminal justice reform plans. Unlike other policy-based texts, which tend to focus more on implementation than assessment, this book provides applicable, step-by-step instruction on determining an initiative's necessity prior to its adoption (reducing the risk of wasting resources), as well as how to accurately gauge its effectiveness during initial roll-out stages. The book gradually introduces basic data analysis procedures and statistical techniques, which, once mastered, can be used to prove or disprove a program's worth. Lastly, the book introduces the types of stakeholders who should review evaluation results for quick action, as well as how to best structure reports to ensure their buy-in.

1
Getting Started with Program Evaluation

Keywords

systematic review
Maryland Report
meta-analysis
Campbell Collaboration (Crime and Justice Group)

Chapter Outline

Introduction

Administrator and Evaluator

Strengths and Weaknesses of Program Evaluation

Evidence-Based Practices

Maryland Report

“What Works”

“What's Promising”

“What Doesn't Work”

Meta-Analysis

Campbell Collaboration (Crime and Justice Group)

Summary

Discussion Questions

References

Program evaluation is the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of implicit or explicit standards, as a means of contributing to the improvement of the program or policy.

—Carol Weiss (1998)

Introduction

The aim of program evaluation is to determine the effectiveness of an intervention. In this book, we are interested in the effectiveness of crime prevention programs, whether they are aimed at individuals (treatment of offenders or victims, prosecution of career criminals) or communities (crime prevention programs), and the operations of the components of the criminal justice system (police, courts, corrections—both juvenile and adult).

The measurement of efficiency is the key problem faced by an evaluation researcher. Accountability can only be established if the evaluation measures are valid indicators of performance. Of course, the measures must also be readily available. Evaluation of crime programs is a significant exercise that can have a direct impact on society. Effective crime programs are vitally needed to deal with the crime problem and all the facets of public safety, such as rising crime rates, fear of crime among the public, crowded jails and prisons, and government expenditures. The pressure for information to deal with these issues is evident. The role of evaluation research in criminal justice is to provide evidence about the effects of programs and policies that are designed to enhance public safety in a manner that is accessible and informative to policy makers (Lipsey, 2005, p. 8). This text will provide an introduction to the research methods and guidelines necessary to conduct a successful evaluation.

Administrator and Evaluator

As Adams (1975, p. 5) has indicated, there are two significant actors in the evaluation process: the program administrator and the evaluator. The administrator must be committed to research and the creation of an organizational climate that encourages the production, reporting, and use of the findings of the evaluation. The evaluator must guide the conduct of the research process from beginning to end: its structure and methodology, use of valid measures of performance, and maintaining the balance between theory and practical, applied policy implications of the results. These two actors must work together and communicate well to produce a valid and reliable program evaluation. Each has different sources of expertise: the administrator knows what the program was designed to do and how it should operate, and the evaluator has methodological skills and knows how to design accurate research projects.

Suchman (1974, p. 5) identified three major aspects of the demand for evaluation research:

• The social problem (in our case, crime).
• The service agencies (the components of the criminal justice system).
• The public (who seek protection from crime).

Again, the emphasis is on the determination of the worth of programs and policies designed to prevent crime. Of course, this is why the term evaluation is so pertinent. In this text, we will be concerned with the determination of the research methods and techniques that can accurately assess the value of a crime program.

Monitoring is necessary to establish accountability for program results. Program evaluation informs the monitoring process. Traditionally, such monitoring assumes two forms: external and internal. Public agencies, like those in the criminal justice system, are externally accountable to political authorities and must regularly report to them about activities and processes. Internally, criminal justice agencies must provide useful, decision-oriented information on the program compliance of their facilities to flag problems before they become crises so that timely adaptations can be implemented (Sylvia & Sylvia, 2012, p. 25). Ultimately, evaluation research can inform and improve the operations of criminal justice programs and enhance service delivery.

Strengths and Weaknesses of Program Evaluation

The evaluation program is both a political and scientific process. The two issues are tied together inextricably. The scientific validity of the study colors and determines the value and objectivity of the research findings and their policy implications. This is "applied" research in that the primary objective of program evaluation is to determine whether the crime prevention program is reaching its goals—defined and desired results.

Evaluation research is different from other research in seven basic ways (Weiss, 1998, pp. 6-8):

1. Use for decision makings: Use results of evaluation research are designed with use in mind. The evaluation should provide a basis for decision making in the future and provide information to determine whether a program should be continued and expanded or terminated.
2. Program-derived questions: The research questions are derived from the goals of the program and its operations and are defined by the evaluator alone. The core of the study is administrative and operational: Is the program accomplishing what it is designed to do? Is it reaching its "target population"—that is, the clients that the program was supposed to serve? Does the program make effective and efficient use of its resources, both physical and financial?
3. Judgmental quality: Objectivity requires that the evaluator focuses on whether the program is achieving its desired goals. It is imperative that these goals are stated in a clear and measurable fashion that accurately documents effective performance.
4. Action setting: The most important thing going on is the program, not the research. The program administrator and staff control access to information, records, and their clients. The research must deal with this reality and construct research designs that are feasible in the real-world setting.
5. Role conflicts: The administrator's priority is providing program services, which often makes him or her unresponsive to the needs of the evaluation. Typically, the administrator believes strongly in the value and worth of the program services. The judgmental nature of the findings and the establishment of accountability are often viewed as a threat to both the program and the administrator personally. The possibility of friction between the program and research and the administrator and evaluator is almost inevitable. Programs are often tied to both the ego and professional reputation of the administrator and staff. Some programs (e.g., Drug Abuse Resistance Education, or D.A.R.E.) are politically attractive, and as a result have lives of their own that defy objectivity and rational assessment. Negative outcomes are not always accepted in a rational manner. In fact, one of the great ironies of evaluation research is that negative findings often fail to kill a program and positive results seldom save one. This is due to the fact that so many crime prevention programs are tied to availability of grant funding. The presence or absence of funding often determines program survival regardless of the program evaluation research findings.
6. Publication: Publication of evaluation research is vital to the establishment of a base of information on effective crime prevention programs. To be published, the research must be carefully designed and executed and the statistical analysis must be valid and accurate.
7. Allegiance: The evaluator is clearly conflicted on this aspect. He or she has obligations to the organization that funds the study, to the scientific requirements of research objectivity, and to work for the betterment of society through the determination of program effectiveness. These obligations can be contradictory and the researcher must face this reality. For example, program officials often need real-time assessments of tactics as they unfold. If the evaluator discovers problems during the process evaluation of program implementation that might jeopardize its success, then the evaluator has an obligation to alert program officials without compromising ethical concerns (Joyce & Ramsey, 2013, p. 361).

Thus, research findings have utility in that they can guide future programming and crime policy. However, the results of evaluation research are not always clear and unequivocal, even when the research design is valid and vital. Findings are often small in magnitude and effect and may be influenced by forces (both social and political) that cannot be isolated from those of the program itself. Thus, program administrators and policy makers typically consider the evaluation research results in combination with other factors such as public opinion, cost, availability of staff and facilities, and possible alternatives (Weiss, 1998, pp. 3-4).

Determination of effectiveness is the crucial difficulty facing the evaluator. The practical problems of obtaining valid data and faithfully executing the research design often conflict with the realities of operating the crime prevention program. The evaluator never has complete control over the research process because of the need to administer the program. Program operations drive the evaluation and its design. The relationship between the needs of the evaluation and the program must be continually balanced. Ultimately, the evaluation research design must be flexible enough to address this relationship. Cooperation between the program administrator and evaluator must be firmly established and maintained. The research must be tied to the changing nature of daily program operations.

The ultimate aim of evaluation research is to guide rational policy making that is based on valid research findings. The rationale is that programs that have been deemed effective will be expanded to other areas and locations and those that fail will be terminated and abandoned. In recent years, evidence-based best practices have grown significantly. Here, the emphasis is on rationality. Ideally, there should be a systematic, evidence-based foundation for criminal justice policies and programs that will increase both the accountability and effectiveness of the criminal justice system (Mears, 2010, p. 2). The search for what works has extended across all areas of the criminal justice system (both adult and juvenile)—police, courts, and corrections.

Evidence-Based Practices

One form of analysis that categorizes, analyzes, and summarizes the research information on a particular criminal justice policy or program is the systematic review. Systematic reviews "use rigorous methods to locate, appraise, and synthesize findings from criminal justice program evaluation studies. Typically, they give their criteria for including studies, conduct an extensive search to find them, code their key features (especially their methodology), and provide conclusions of their review" (Welch & Farrington, 2001, p. 161).

For example, Braga (2001) conducted a traditional, narrative review of studies of the effects of hot-spot policing on crime. He reviewed nine studies—five randomized experiments and four nonequivalent control group quasi-experiments. Overall, the research findings revealed that focused police efforts (aggressive disorder enforcement, directed patrols, proactive arrests, and problem solving) can prevent crime in high-risk places that feature concentrations of crime without resulting in crime displacement effects. However, the study was unable to determine exactly what types of police interventions are most preferable in controlling crime at hot-spot locations.

Maryland Report

Probably the most pertinent example of a systematic review is the "Maryland Report." This report was written in response to a request from the U.S. Congress to review existing research on criminal justice programs and identify those determined to be most effective (Sherman et al., 1997, p. 4). The report reviewed more than 500 crime prevention program evaluations a...