Research at the IMF : Relevance and Utilization

International Monetary Fund. Independent Evaluation Office

• Publisher: INTERNATIONAL MONETARY FUND
• Year: 2013
• eBook ISBN: 9781616351540

NONE

Chapter 1: Introduction

1. This evaluation examines the research produced at the IMF between 1999 and 2008.¹ It focuses on the relevance and utilization of the research, particularly as seen by authorities in member countries, and also examines technical quality and the management of research activities. The evaluation identifies ways to improve the relevance, quality, and management of IMF research. Research is defined broadly to capture most analytical publications of the IMF, ranging from surveillance-oriented output—such as selected issues papers (SIPs) prepared for Article IV consultations and the analytical chapters of the World Economic Outlook (WEO) and Global Financial Stability Report (GFSR)—to more academically-oriented output—such as working papers (WPs) and external journal publications. During the evaluation period these outputs comprised a large body of research, about 650 publications annually, at a cost of about 10 percent of the IMF budget.

2. Research plays an important role in the IMF. It contributes to the development and updating of conceptual models and tools that form the basis for the IMF’s analysis. High-quality, relevant research is, therefore, critical to sustain the credibility of IMF policy advice and program design. IMF research also contributes to furthering global knowledge in areas relevant to the IMF’s purposes. The possibility of engaging in research activities also helps to attract academically oriented, talented economists and to keep staff skills up to date.

3. This evaluation uses as a reference point the 1999 External Evaluation of Research in the IMF (Mish-kin and others, 1999) that was prepared at the request of the Executive Board by a team of outside experts.² The focus of that evaluation was on the organization and quality of the research. The Executive Board agreed with its finding that there was “substantial room for improvement in the overall quality of the IMF’s research.” Directors endorsed many of the report’s recommendations, including the need for greater coordination of research activities and that more of the research should be in areas where it could add the most value: namely research on developing and transition economies, a stronger emphasis on the financial sector, and cross-country work. Progress has since been made in many of these areas and Annex 1 describes the status of implementation of the report’s recommendations.

4. The following are some of the main questions addressed in this evaluation:

• How relevant was IMF research to authorities in member countries, IMF staff, and other stakeholders? How was it utilized? Was the thematic coverage adequate? Was it widely read in member countries? Within the IMF? How familiar were authorities and other stakeholders with its findings and messages? Did it play a role in policy-making? In the dialogue between IMF staff and authorities?
• What was the technical quality of IMF research? Did it generate new knowledge or broaden the understanding of policy frameworks? Did it allow for alternative perspectives?
• How were IMF research activities organized and managed? How were activities prioritized and coordinated? How was research reviewed? Was dissemination effective?

5. The remainder of the report is organized as follows. Chapter 2 discusses the scope and methods of the evaluation and describes trends in research product lines across IMF units and over time. Chapter 3 discusses the relevance and utilization of IMF research. Chapter 4 presents findings on the technical quality of research. Chapter 5 describes the management of research activities. Chapter 6 presents the main conclusions and recommendations. This report is accompanied by four background documents providing technical background on how the evidence for this evaluation was gathered (i.e., describing how surveys and interviews were conducted and how their results were integrated into the overall analysis). Seven background papers, summarized in Annex 4, present the assessment of peer review panels on specific research product lines, for example, WEO/GFSR and SIPs, as well as a citation analysis.

Chapter 2: Evaluation Framework and Background

6. This chapter lists the IMF product lines considered to be part of research for the purpose of this evaluation, and describes the methods and sources of data used in this study. It also provides information on the composition of research by product line, originating IMF department, and coverage.

A. What Is IMF Research?

7. The evaluation adopted an enumerative definition of research to capture the variety of analytical outputs produced across all IMF departments, including the Research Department (RES). For this evaluation, research is defined to cover a wide spectrum of products, ranging from surveillance- and policy-oriented publications (hereafter “surveillance-oriented output”) to more academic-oriented publications (hereafter “academic-style output”). Surveillance-oriented output comprises the applied research usually prepared as part of the IMF’s operational work, mainly surveillance and program work. Academic-style output refers to work that seeks to generate new knowledge and techniques or to broaden and deepen the understanding of policy frameworks.

8. The following are the main types of IMF publications considered research in this evaluation:

Surveillance-oriented publications

• The World Economic Outlook (WEO) is a semiannual flagship publication of the IMF. The WEO is produced in the Research Department (RES) with inputs from other departments. It is a key vehicle for communicating the IMF’s multilateral surveillance messages and has a wide audience, including among central banks, government officials, financial institutions, think tanks, and academia. The research component of the WEO is contained in individual analytical chapters covering economic issues of importance to the world economy.
• The Global Financial Stability Report (GFSR) is another semiannual flagship publication of the IMF. The GFSR is produced in the Monetary and Capital Markets Department (MCM). It has a wide audience of policymakers, especially in central banks, as well as in financial institutions and academia. The research component of the GFSR is contained in individual analytical chapters focused on financial issues of importance to the world economy.
• The five Regional Economic Outlooks (REOs) are semiannual publications produced by the five area departments to provide a perspective on the macroeconomic outlook for each region. Policymakers are the REOs’ main intended audience. Analytical chapters address topics of importance to the corresponding region.
• Selected issues papers (SIPs) are background papers that accompany Article IV consultation documents and are issued by area departments. They contain analytical work and are often used to inform policy discussions between the staff and the authorities. The main audience for SIPs is authorities in the corresponding member countries.
• Occasional papers (OPs) feature nontechnical analyses of economic and financial subjects of current importance to the IMF’s work. Topics include broad policy questions and country-specific issues. OPs are often authored by a large team of IMF staff and their intended audience is policymakers.
• Policy discussion papers/staff position notes (PDPs/SPNs) showcase new policy analysis and research by IMF departments. They are nontechnical and aim at a broad audience, including IMF staff involved in mission work and readers outside the IMF who are interested in economic policy issues.

Academic-style publications

• Working papers (WPs) feature original research by IMF staff, consultants, and guest scholars, and are generally more technical than other research products. Their intended audience includes academics as well as policymakers.
• External publications feature original research in refereed journals, including the IMF’s own journal, IMF Staff Papers. Typically this research is published in some different form before being submitted for external publication.
• Books and conference volumes are usually based on conferences and seminars held or sponsored by the IMF and cover a wide variety of topics. The intended audience varies, but it includes the wide academic community, as well as country authorities.

9. The two categories of research output were assessed along the same dimensions of relevance and utilization, technical quality, and management. But for each category the evaluation used somewhat different methods and metrics, given their different goals and that they were directed to different audiences. In practice, the evaluation found that the distinction between the two types of outputs was not sharp. Most WPs and other academic-style outputs dealt with policy-relevant issues, even if they were not specifically written for surveillance activities. Similarly, many surveillance-oriented outputs, particularly WEO chapters, generated new knowledge and introduced significant innovations. In fact, there was significant overlap between these two categories, for example, some papers first issued as SIPs were later converted into WPs and eventually published in academic journals.

10. Country authorities and many other audiences typically viewed research products, especially surveillance-oriented publications, as broadly representing the views of the IMF. This was the case even though all research outputs contained disclaimers explaining whose views they reflected, and clearly stating that they did not necessarily reflect the views of the IMF, its Executive Board (the Board), or its member countries.³

B. Methods and Sources of Data

11. The evaluation used a variety of methods of analysis and several sources of evidence, which allowed for triangulation of findings and contributed to the robustness of conclusions.⁴ The main sources of information for this report were:

• Document review—mainly reflected in six background papers presenting the assessment of peer review panels on specific product lines, for example, WEO/GFSR and SIPs (Annex 4 presents the abstracts of the background papers);
• Interviews—more than 350 semi-structured and open interviews of authorities, staff, and other stakeholders (Background Document IV explains the selection of interviewees, presents the interview templates, and describes how the results were aggregated by type of country);
• Surveys—two surveys were conducted—one of authorities and one of staff (see Background Document I available at www.ieo-imf.org); and
• Technical and statistical work, including a citation anal...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Contents
• Foreword
• Abbreviations
• Executive Summary
• 1 Introduction
• 2 Evaluation Framework and Background
• 3 Relevance and Utilization
• 4 Technical Quality of IMF Research
• 5 Management of IMF Research
• 6 Conclusions and Recommendations
• Boxes
• References
• Statement by the Acting Managing Director, Staff Response, and the Acting Chair’s Summing Up
• Footnotes