Program Evaluation and the Management of Government

Eliot Freidson, Ray Rist

• Publisher: Routledge
• Year: 2020
• eBook ISBN: 9781000678789
• Topic: Politics & International Relations
• Subtopic: International Relations
• Index: Politics & International Relations

This book appears at an opportune time in the history of evaluation. Its detailed and up-to-date account of the organization and use of evaluation in eight Western, democratic countries—Canada, Germany, United Kingdom, United States, Denmark, Holland, Norway, and Switzerland—shows how evaluation functions at different levels of development. Focusing on the national or federal level of government, this volume presents a systematic and comparative view of eight nations at different stages of the development, institutionalization, and utilization of evaluations. All of these original contributions have been written by academics and government officials involved in the production and use of evaluation findings. Each shows how their respective country has moved to institutionalize evaluation at the federal level, and each explores the reasons for that institutionalization. Among them are managerial accountability, the increased complexity of the decisions facing policymakers, federally sponsored social change that needs to be tracked and assessed, and the increasing recognition that political power comes to those who possess such information. Program Evaluation and the Management of Government is tightly integrated. The contributions share coherence, a common analytic framework and use of key terms, resulting from the authors' three-year dialogue as members of the Working Group on Policy and Program Evaluation sponsored by the International Institute for Administrative Sciences located in Belgium. Their shared commitment to working together has given us the first systematic effort to assess evaluation across such a large number of countries. It will be of interest to applied social scientists and policymakers, especially those interested in comparative research.

II
Countries in the First Wave of Evaluation Development

1
Policy and Program Evaluation in the Government of Canada

R. V. Segsworth

Introduction

In a series of background documents prepared for a 1985 seminar on program evaluation, the government of the Province of Quebec raised the question: “What’s new after twenty years?” (Program Evaluation Branch [PEB] 1985). The implication that a serious program evaluation policy and function has existed in the Public Service of Canada for more than two decades is somewhat misleading. It could be argued that program evaluations had been conducted virtually from Confederation. Parliamentary committees, central agencies, line departments, and the cabinet had examined and reviewed policies and programs over the years. Royal commissions and, more recently, “colored” papers and task forces were used frequently to examine and evaluate the effectiveness of various government policies and programs (Doerr 1981). On the other hand, it is equally clear that such efforts were sporadic and of uneven quality. There is, however, evidence that suggests that a serious attempt to establish an internal, comprehensive, ongoing program evaluation capability and process began in the 1960s.

The Development of Evaluation Policy

The reports of the Royal Commission on Government Organization (Glassco Commission) stressed, among other things, the need to justify expenditures. The introduction of planning, programming, and budgeting (PPB) in the late 1960s was perceived to be a step in this direction. The 1966 Treasury Board publication, Financial Management, pointed out the need to monitor the progress of programs, to evaluate the effectiveness of operations, and to assess the performance of government activities. The 1969 PPB Guide called for an “information system on each program to supply data for the monitoring of achievement of program goals and to supply data for the reassessment of the program objectives and the appropriateness of the program itself.” The evaluation implications and concerns were evident.

Departments were encouraged to establish planning and evaluation units during the late 1960s and early 1970s and to use such units to carry out the kinds of evaluation studies suggested by PPB. The Treasury Board Secretariat set up its Planning Branch in 1969. This agency, amongst other responsibilities, carried out sophisticated evaluations and policy reviews, developed evaluation capacity and support for departments and attempted to show the way in an effort to encourage more and better evaluation activities within departments.

The success of such efforts was marginal. Despite the prime minister’s apparent support for such “rationalist” attempts (Adie and Thomas 1982), program evaluation within the public service of Canada was largely unsuccessful. Treasury Board studies in 1974 and 1976 “found that little progress had been made in evaluating programs” (Program Evaluation Branch [PEG] February 1985, 8). “The bulk of staff years nominally assigned to evaluation were routinely utilized in broad planning activities, consultancy roles and specialized ad hoc activities” (Jordan and Sutherland 1979, 588). Dobell and Zussman noted that a “solid decade—almost two—has gone into changing the words and the forms. Yet even the most dedicated do not argue that evaluation efforts have led to decisive results or significant government action” (1981, 406).

Dobell and Zussman argue that three factors held back the success of program evaluation efforts: the lack of agreed theory and purpose, departmental resistance, and the failure of the system to consider the information needs of the user. Michael Prince suggested that “probably the most fundamental factor affecting policy-advisory groups in attempting to organize their program evaluation role is that there is no overall process in the federal government” (1979, 295).

This failure to evaluate was noted with concern by an increasingly influential external agent—the auditor general of Canada. Over a long period of time, his annual reports expressed concern regarding financial management practices within the government of Canada. His 1976, 1977, and 1978 reports specifically pointed to a need to improve the program evaluation function within the public service.

Parliament acted to support the auditor general. The 1977 Auditor General’s Act allowed the auditor general “to call Parliament’s attention to cases in which the government had not established satisfactory procedures to measure and report on the effectiveness of its programs” (McNamara 1979, 1544).

This officer of Parliament was quick to respond to his new mandate. By 1978 he reported that “a review of 23 programs in 18 departments has disclosed few successful attempts to evaluate the effectiveness of programs. The scope and quality of effectiveness evaluation will have to be increased significantly before management, the Government and Parliament, each with its respective interests, can be reasonably informed on the achievements of public programs.” Earlier, in 1977 he recommended the appointment of a “chief financial officer for the government” (Jordan and Sutherland 1979, 585–86).

The Royal Commission on Financial Management and Accountability (Lambert Commission), appointed in 1976, reported in 1979. It noted concerns similar to those raised by the auditor general. The Lambert Commission put great emphasis on the need to improve program evaluation. It recommended a cyclical review of government programs every five years and a policy which would ensure the capacity within government to conduct such evaluations at reasonable standards of quality.

The government had little choice but to respond to such pressures. The Office of the Comptroller General (OCG) was created in 1978. The first incumbent, Harry Rogers, described his program evaluation responsibilities as being

• to develop and promulgate the Treasury Board’s policy and guidelines on program evaluation;
• to be collector and disseminator of information which will aid in the performance of program evaluation;
• to provide advisory services on the application to specific programs of program evaluation;
• to review departmental and agency compliance with the Treasury Board’s policy. (Rogers, April 1979)

Treasury Board policy had been outlined in Circular 1977–47, which was issued on 30 September 1977. The general statement of policy was that “departments and agencies of the federal government will periodically review their programs to evaluate their effectiveness in meeting their objectives and the efficiency with which they are being administered”. The remaining six pages of the circular provided guidelines on the policy.

The period from 1978 to 1981 was one in which the Program Evaluation Branch (PEB), together with the comptroller general, attempted to give substance to the 1977 policy. A serious effort was undertaken to avoid much of the departmental resistance which had characterized the previous effort. Mr. Rogers adopted what he called a “co-operative approach” (OCG Nov. 1979). This strategy of “the soft sell of program evaluation in an era of expenditure restraint” (Doern and Maslove 1979, 3) involved extensive consultation with senior line department officials as well as outside experts. In addition, the PEB staff prepared, circulated, discussed, and amended background and discussion papers (Office of the Comptroller General [OCG] Oct. 1980). By the fall of 1981, the 1977 policy had taken shape and it was outlined in two important publications: Guide on the Program Evaluation Function (PEB 1981a) and Principles for the Evaluation of Programs by Federal Departments and Agencies (PEB 1981b).

These two documents are the basis of much of what follows. They provide a definition of program evaluation in the government of Canada. They outline the roles of key actors in the process and they discuss the process which is required. In essence, these two publications describe, in considerable detail, official policy regarding program evaluation.

A Definition of Policy and Program Evaluation

Program evaluation has not been viewed in the same manner in Canada as it has in the United States for two reasons. There is less of a history of evaluation as a significant part of social-science research in Canada than in the United States and as a result, government has taken more of a leadership role than has been the case in the United States. The fact that Canada is a parliamentary system with a fusion rather than a separation of the executive and legislative branches also necessitates a somewhat different approach to program and policy evaluation. It is seen as one of many elements in the management cycle—reviewing and monitoring program performance (see figure 1.1). The basic purpose of program evaluation, therefore, is to provide clients and users with relevant and timely information and analysis to assist them to make better resource allocation and program improvement decisions. In addition, it is believed that good evaluation studies will improve accountability within government. As a result of this approach to program evaluation, less emphasis is placed on truly scientific studies that attempt to provide definitive statements about program outcomes, effects, and impacts, or all three.

FIGURE 1.1 Management Review and Monitoring Functions

At the same time, the policy suggests a fairly comprehensive examination of a number of fundamental subjects in each evaluation study conducted. There are four basic program evaluation issues involved: program rationale, impacts and effects, objectives achievement, and alternatives.

Program rationale raises the question of whether the program makes sense. Two further questions follow from this concern. The first involves an assessment of the extent to which the objectives and mandate of the program continue to be relevant. The second refers to the extent to which outputs and operational activities are logically linked to the attainment of program objectives.

Impacts and effects focuses on results and asks what has happened. In this element, impacts and effects, both intended and unintended, are to be determined. In addition, the evaluator is expected to describe relationships of the program under review with other relevant programs which may affect or duplicate its outputs.

Objectives achievement concerns itself with the question of whether the program has achieved what was desired. In addition, the evaluation study is expected to demonstrate how appropriate results were achieved.

Finally, the evaluator must examine alternatives. Two issues confront him. The first is whether more cost-effective programs might be created to achieve the desired results. The second is an assessment of alternative delivery systems to see if more economical means might be employed to achieve the objectives established for the program.

The Structures of Policy on Program Evaluation

In order for this type of evaluation to be conducted regularly and thoroughly, the policy establishes well-defined responsibilities for a number of key actors in the process. Other factors, including political pressures and needs, require the participation of other actors. Six structures have important roles to play in the evaluation process and in ensuring that program evaluation policy is implemented. They are the OCG, the deputy heads of departments and agencies, evaluation units within departments, Treasury Board, cabinet policy committees, and the auditor general.

The Office of the Comptroller General has a system-wide responsibility for program evaluation. This office, and in particular, the PEB, is to ensure that the 1977 policy on program evaluation is implemented throughout the Public Service of Canada in appropriate departme...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication Page
• Contents
• Figures
• Tables
• I Introduction
• II Countries in the First Wave of Evaluation Development
• III Countries in the Second Wave of Evaluation Development
• IV Overview and Synthesis
• Index
• About the Editor
• Addresses of Authors