Recurring Issues from a Decade of Evaluation : Lessons for the IMF

International Monetary Fund. Independent Evaluation Office

• Publisher: INTERNATIONAL MONETARY FUND
• Year: 2014
• eBook ISBN: 9781484376850

NONE

Chapter 1. Introduction

1. This evaluation aims to help the IMF enhance its effectiveness by identifying major recurring issues from the IEO’s first 20 evaluations and assessing where they stand. These issues are tendencies that the IEO has found in specific instances and that have affected the IMF’s performance in all of its core areas of responsibility: surveillance, lending, and capacity development. Almost all of them have been frequently discussed within the IMF as requiring institutional attention. While the Fund has addressed these issues in specific instances, their recurrence in different contexts in multiple IEO evaluations suggests that they are intrinsic to the nature of the IMF, with deep roots in its culture, policies, and governance arrangements.

2. The evaluation has been prepared in response to the 2013 External Evaluation of the IEO, which proposed that the IEO prepare a review of “generic and substantive issues” that are not “encapsulated in specific recommendations” but deserve monitoring. The External Evaluation further proposed that the review “should focus on major generic issues identified by the IEO rather than [being] an exhaustive review of specific actions” (Ocampo, Pickford, and Rustomjee, 2013, p. 28). This proposal received broad support from the Executive Board, Management, and staff when the report was discussed at the Board in March 2013.

3. The External Evaluation proposed such a review as a means to strengthen the follow-up process for Board-endorsed IEO recommendations.¹ At present, this process consists of (i) Management Implementation Plans (MIPs) for those IEO recommendations endorsed by the Board, and (ii) Periodic Monitoring Reports (PMRs) to track the implementation of those recommendations. The External Evaluation found that the follow-up process has several weaknesses. One is “conflicts of interest for Management, which has the triple responsibility of overseeing the summing up of the Board discussion, preparing the subsequent implementation plan, and monitoring its application” (Ocampo, Pickford, and Rustomjee, 2013, p. 24). The external evaluators recommended that the preparation of PMRs be moved to the Office of Internal Audit in order to separate Management’s implementation and monitoring functions. This change has been approved by the Executive Board to take effect in 2014 (IMF, 2014).

4. Another weakness the external evaluators identified in the follow-up process is that it has become a “box-ticking” exercise, in which IEO recommendations are turned into “a series of specific actions” that tend to dilute their substance; there is no monitoring of broad policy conclusions and concerns raised in IEO reports. The implementation of an IEO recommendation is no longer tracked once the IMF staff judges, and the Board concurs, that the benchmarks for implementation noted in the MIP have been met or are progressing to timely completion.² This may in part explain why “recommendations deemed by the Fund to have been met or on track for completion tend to be raised again in subsequent IEO reports” (Ocampo, Pickford, and Rustomjee, 2013, pp. 23–24).

5. In line with the External Evaluation’s proposal, the present report focuses on key issues that have recurred in past IEO evaluations, rather than on specific IEO recommendations and their implementation. IEO recommendations, even when endorsed by the Board, do not preclude the IMF from addressing the identified issues in an alternative way. By highlighting the recurring issues, this report aims to advance the overall effort to enhance the follow-up process, the need for which has been recognized by the Board. This approach also accords with the suggestion made by the Managing Director, in her response to the External Evaluation, that it would be useful to “refocus the follow-up process … on the broader policy objectives.”³

6. Though some of the issues highlighted in this report may not be fully solvable, recognizing them and understanding their root causes is a first step in moving to address them in a fundamental way. Without strategic efforts to get to the bottom of the problems, the IMF will keep attempting to address the same issues in different contexts without finding permanent solutions.

7. The rest of the report is organized as follows. Chapter 2 discusses the framework of evaluation, including the identification of recurring issues, the evaluation questions, and the sources of evidence. Chapter 3 applies the framework to issues identified by the evaluation team as most frequently recurring in five areas, namely: (i) Executive Board guidance and oversight; (ii) organizational silos; (iii) attention to risks and uncertainty; (iv) country and institutional context; and (v) evenhandedness. Chapter 4 presents conclusions and issues for Board consideration. Annexes 1 and 2, respectively, present a complete list of IEO findings related to the five groups of issues and a selective chronological summary of relevant IMF initiatives and decisions adopted during 2008–13.

Chapter 2. The Evaluation Framework

A. Identifying Issues for the Evaluation

8. Previous authors have identified high-level, generic issues in the IMF’s performance that have been repeatedly highlighted by IEO evaluations (Lamdany and Edison, 2012; Salop, 2012; Reichmann, 2013).⁴ Instead of attempting to distill high-level findings from multiple evaluations, this evaluation takes a bottom-up approach, first identifying findings from past IEO evaluations and then grouping them according to common themes. Using this approach allows us to track how similar issues have recurred in different contexts and what the IMF may have done over time to address them.

9. The evaluation team concentrated on evaluation findings about aspects of the IMF that are not topic-specific. For example, a finding about the nature of interdepartmental collaboration in producing research is considered as non-topic-specific because it concerns a broader organizational issue; by contrast, a finding about the technical quality of IMF research is specific to the topic of one particular evaluation. The team identified more than 300 non-topic-specific findings in the IEO’s first 20 evaluations.

10. The non-topic-specific findings pertain to common themes, which are for the purpose of this evaluation classified as recurring issues. Some of the recurring issues are closely related to each other, so that an evaluation finding may simultaneously pertain to two or more issues (e.g., organizational silos within the IMF and integration of different strands of work; candor and attention to risks and uncertainty). Themes on which the past evaluation findings are most numerous are considered to be the most recurrent.

11. The present evaluation identified recurring issues in 14 areas, each containing a dozen or more related findings from multiple evaluations. These in turn can be grouped under the following categories:

Institutional, organizational, and governance concerns

• Accountability and monitoring frameworks
• Corporate governance
• Executive Board guidance and oversight
• Organizational silos

Analytical shortcomings

• Attention to risks and uncertainty
• Content and value-added
• Country and institutional context
• Integration (e.g., macro-financial, multilateral/ bilateral)

Cognitive and cultural traits

• Candor
• Mindset (e.g., group-think, intellectual capture)
• Operational practices (e.g., staff turnover)

Relationship with member countries

• Engagement with authorities
• Evenhandedness
• Outreach

12. As recommended by the External Evaluation, the present report concentrates selectively on those issues that have been identified most frequently in past IEO evaluations:⁵

• Executive Board guidance and oversight. The Executive Board has in some instances fallen short of providing clear guidance and effective oversight of the institution;
• Organizational silos. The IMF has in some instances found it difficult to integrate work across different parts of the institution;
• Attention to risks and uncertainty. The IMF has in some instances paid insufficient attention to risks and uncertainty in surveillance and program design;
• Country and institutional context. The IMF has in some instances provided insufficient country specificity and institutional context in its analytical work and policy advice; and
• Evenhandedness. The IMF has in some instances been seen as lacking evenhandedness in its analysis or treatment of member countries.

13. The selection of these issues does not mean there are no others of valid concern to the IMF; it only means that these issues were the ones most frequently found in the context of the topics the IEO selected for its evaluations. Had the IEO selected alternative topics, other issues might have been identified or found to be just as recurrent. By using recurrence as the selection criterion, moreover, the present review may be excluding some important issues from consideration.

14. Even so, the topics selected by past IEO evaluations cover much ground, ranging from surveillance to crisis management, from research to governance, and from advanced to low-income countries. The fact that the issues identified by the evaluations have recurred across a wide range of contexts must indicate their importance and relevance. Almost all of them have been frequently discussed within the IMF as requiring institutional attention by the Executive Board, Management, and staff. Though the identified issues have not been found in all IEO evaluations, they have recurred often enough to be of concern, especially from the point of view of individual countries for which the incidence of one such weakness could have major consequences.

B. Evaluation Questions

15. For each of the five groups of issues, this evaluation addresses the following sets of questions:

• In what manner and in what context has the issue surfaced in the IMF’s work, as identified by successive IEO evaluations?
• What has the IMF done to address the issue, irrespective of whether there was a specific IEO recommendation?
• If action has been taken, has the situation improved? In what, if any, areas of the IMF’s work does the issue remain outstanding? To what extent is the issue so inherent to the nature of the IMF, or to what the IMF does, that it will likely remain a challenge and require continued attention?

C. Sources of Evidence

16. Building on the findings obtained from IEO evaluation reports, the evaluation team gathered evidence on the current status of the ident...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Contents
• Foreword
• Abbreviations
• Executive Summary
• 1 Introduction
• 2 The Evaluation Framework
• 3 Recurring Issues from IEO Evaluations
• 4 Conclusions and Issues for Board Consideration
• Boxes
• Sources Consulted
• Statement By the Managing Director and The Chairman’s Summing up
• Statement by the Managing Director
• The Chairman’s Summing Up
• Footnotes