Audit and Evaluation
Audit and evaluation are essential instruments for accountability. They help explain what organizations and individuals have done, how well they have done it, and what has been achieved as a result. Both support oversight by managers, boards, and legislatures so that these parties can be provided with information on whether or not actions have, for example, been carried out correctly, with the desired results, and that value for money has been achieved. There are numerous other mechanisms available for rendering similar accounts including inquiries, reviews, investigations, monitoring systems, and supervision associated with a broad array of performance management and measurement techniques (Lonsdale, Wilkins, & Ling, 2011). However, audit and evaluation are of particular interest because of the professionalization that has taken place in recent decades, the widely noticed expansion of such practice (Power, 1997), and the institutionalized approaches that have been applied to examining questions of performance.
Performance audit, internal audit and evaluation are distinct areas of practice most often located in separate units or organizations within governments around the world. Unlike traditional financial audit, which has its own specific focus on the accuracy of financial statements, there can be a crossover between internal audit, performance audit and evaluation in the programs and topics that are examined and the methodologies used. There can also be transfers of professional practitioners between the different activities, most often from evaluation into audit, but also in other directions.
This book examines the potential for crossover between the three professional groups and their work, as well as related incentives and potential barriers. By “crossover” we mean the sharing or exchange of approaches between the different practices which help in creating synergies or common understanding between practitioners from different backgrounds. The focus is on the public sector, but many of the observations, particularly in relation to internal audit, apply more generally, including to the private sector.
In addition, in many countries there is an increasing blurring of the boundary between public and private sector provision, with many public services now delivered by private companies. As a result, many such companies are finding they are increasingly subject to the scrutiny expectations traditionally associated with public bodies. This makes the subject matter and concepts discussed in this book of relevance to people in a wide range of settings.
Audit and evaluation are distinct activities that operate under their own statutes, policies, conventions, and professional standards. However, they do not operate in silos from each other. They have much in common, including principles of practice, methodologies, areas of work, and challenges. At the same time, however, there are significant differences. These include levels of formality and professionalization, rights of access to information, the extent to which their work is designed to be publicized and becomes public, and the extent to which those carrying out the work can expect to be able to secure a response to their recommendations.
The visibility of the different types of work varies. There is often a greater public awareness of performance audit results than other, equally important, results from evaluation or internal audit. This is partly a reflection of the purpose of performance audit, which is concerned with public accountability and transparency, which is often far less the case for internal audit or evaluation. When the head of a national audit office – the statutory body that undertakes much of this work – issues a performance audit report, there is often media attention and public parliamentary or legislative scrutiny. In comparison, internal audit reports and evaluations are more often internal tools instigated by heads of organizations or by their program managers to provide advice and guidance on program performance. What these reports provide, and how they are produced, are important for recipients, who have them in their management tool kit to improve performance.
Understanding the differences and similarities between the work and the reports they generate matters to the managers and users of the reports, as well as to audit and evaluation practitioners. These practitioners may be looking to achieve efficiencies and improvements in the ways they do their work by relying on, or collaborating with one another, which in turn has the potential to make for more insightful and less burdensome analysis. With a better understanding of what others are doing and what methods have worked elsewhere, opportunities exist for practitioners of all kinds to learn from each other and to strengthen each practice. Improved understanding of the different practices also provides opportunities for professionals to move between them more easily.
Against this background, the purpose of this book is four-fold. First, it is to examine what the differences and similarities are of current practices with a view to gaining a better understanding of each of them. Second, to examine the practices and the opportunities they present for crossover and collaboration. Third, to examine the challenges to practices in the current environment of financially constrained public services. Fourth, to examine organizational initiatives that have been taken to enhance the opportunity for crossover or greater collaboration.
For practitioners, the book is designed to provide insight on how to bridge some of the divides, while for users of the products, it will help generate a better understanding of the activities and services that are either imposed on them or are freely available, and also help to stimulate their optimal use. Specific cases will be included to illustrate what has worked or not and suggest reasons why.
This introductory chapter provides the context for the remaining chapters in the volume by laying out some definitions and concepts, some of the major players, themes, and structure of the Book.
Definitions
The Oxford dictionary defines audit as:
An official inspection of an organization’s accounts, typically by an independent body. 1.1 A systematic review or assessment of something.
The broad definition includes the audit of financial statements, performance audit (sometimes called value for money audit), and internal audit. In this book, we include in our analysis performance audit and internal audit, the two forms of audit that have most in common with evaluation in that they are designed to focus on performance gaps, and identify ways of rectifying or closing those gaps by analyzing available evidence and making recommendations for improvement.
Authoritative definitions of each activity are as follows:
Performance audit refers to an independent, objective, and reliable examination of whether government undertakings, systems, operations, programs, activities or organizations are operating in accordance with the principles of economy, efficiency, and/or effectiveness and whether there is room for improvement. (International Organization of Supreme Audit Institutions (INTOSAI), 2016b, p. 2).
Internal Audit is an independent, objective assurance and consulting activity designed to add value and improve an organization’s operations. It helps an organization accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of risk management, control, and governance processes (Institute of Internal Auditors Research Foundation, 2017).
Program Evaluation refers to the process of determining the worth or significance of an activity, policy or program. [It is] as systematic and objective as possible, of a planned, ongoing, or completed intervention (Organisation for Economic Co-Operation and Development (OECD), 2010).
These are generally-accepted definitions, each drawn from well-informed practitioner sources that carry weight in their fields. Those for performance audit and internal audit are from international institutions, reflecting a broad consensus built over recent decades, often through formal consultation and research processes. They may be subject to local interpretation but are widely recognized. Both types of audit also have international, national, and regional professional institutes, which represent and inform practitioners, and often further elaborate on the standards and practice by producing guidance and operating protocols.
The program evaluation definition does not have the same broad consensus. As Morra Imas and Rist (2009, p. 8) point out, there are many different definitions that reflect different purposes or contexts, with the result that the practice of evaluation is not as prescribed or limited as audit. The absence of a consensus by an international body of evaluation practitioners is a significant difference between audit and evaluation.
A further distinction between the three practices relates to the defined standards of practice and how they are regulated. All three have such standards but, as with the definition of the activities, there is much more consensus for audit than for evaluation. For performance audit, there is an agreed framework of international standards within which regional and individual organizations, mandated to carry out performance audits, have further developed standards and guidance. Internal audit has a single international set of standards and guidance that also serve as a national set of standards and guidance. Evaluation, on the other hand, has multiple standards that include those drawn from different disciplines. This is further elaborated in Chapters 2 to 5 in Part I.
There is no certification for program evaluation or program evaluators, while audit has different forms of certification. Auditors who are qualified accountants (certified, chartered or management accountants, for example, depending on which institute’s qualification has been followed) are required to meet institutional standards, as well as audit standards, and are subject to discipline or loss of their designation if they fail to comply.
Performance audit organizations that adhere to international and regional standards do not provide certification specifically for individual performance auditors, some of whom may be designated accountants but others could be evaluators or from other disciplines. Most will instead require completion of specified training or evidence of specialist knowledge as well as on-the-job experience. Further, external audit organizations commit to adhering to agreed performance audit standards and require adherence by each of their performance audit practitioners. Internal audit has a voluntary certification program available that internal auditors are encouraged to obtain, in addition to following additional staff training. Both forms of audit are subject to forms of peer review or external scrutiny to help identify areas for improving their practice.
A challenge for the practice of performance audit is to ensure that the multidisciplinary evaluators who participate in it understand and can apply the performance audit standards. These standards are more procedural and prescriptive about the components of a performance audit than the requirements for systematic inquiry for evaluation, and are supplemented with much more practitioner guidance and process, developed to meet the needs of the regime within which the Audit Office operates. Some of this may seem alien to those not used to it, as well as overly bureaucratic and procedural, but is justified by audit bodies by the very public nature of their work and the expectations of transparency about how they conduct their business.
Performance audit, internal audit and evaluation all, in their different ways, support accountability, with challenges for both in answering the increasingly complex questions about the value of public policies and programs in democratic countries (Bemelmans-Videc, Lonsdale, & Perrin, 2007). There are important distinctions to be made, however, in how they are positioned with respect to accountability relationships. Performance auditors in national audit institutions serve an existing accountability relationship between government and their legislature, which is often long-established, highly formalized and viewed by some as unnecessarily punitive (LeClerc et al., 1996, p. 224).
This is less the case for internal audit and evaluation in government since they work inside public institutions, usually reporting to a board, committee or the head of the agency or department. Internal auditors in the private sector serve the accountability relationship between management and the governing board of governors. Not all of this work is publicly available, and as a result the consequences of the findings – for individuals and organizations – are likely to be worked out within institutions, rather than being visible, for example, via public hearings or press briefings as in the case of performance audit.
Even within the definition and structural differences, there are commonalities between all or some of these activities. Performance audit and internal audit, for example, have a common root in accounting and in the formal rendering of an account for the allocation and utilization of public resources. Internal audit and program evaluation are at times co-managed or co-located in government organizations since they both support the head of the organization, unlike performance audit. Performance audit and program evaluation both deal with issues of program or policy effectiveness.
In general, all three activities aim to translate their scrutiny and analysis of evidence presented on performance into practical recommendation for change – and hopefully improvement – within the activities or bodies examined, and possibly within wider categories of organizations, where the identified lessons may have similar resonance.
