Chapter 2: Recently Completed Evaluations and Follow-Up on Past Evaluations
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2012
During FY2012, the IEO issued the report on its evaluation of Research at the IMF: Relevance and Utilization, which was discussed by the Executive Board on June 13, 2011.
Research at the IMF: Relevance and Utilization
This evaluation examined the research produced at the IMF between 1999 and 2008, focusing on the relevance and utilization of the research, particularly as seen by authorities in member countries. It also examined the technical quality and the management of research activities. Research was defined broadly to capture most analytical publications of the IMF, ranging from surveillance-oriented output to more academically-oriented output—together comprising a body of research of about 650 publications annually, at a cost of about 10 percent of the IMF budget.
Findings and conclusions
The evaluation found that the IMF produced a vast body of research that was widely read, included a large number of high-quality and very useful publications, and was appreciated by country authorities and the research community. This was particularly true for the World Economic Outlook (WEO) and Global Financial Stability Report (GFSR), but also for many other publications. Given the attention they received from member country authorities, IMF research publications played a significant role in policymaking and were relied upon by authorities in their discussions with IMF staff. IMF research was also influential among other international organizations, academics, and think tanks.
At the same time, the evaluation found that there was significant scope to improve the relevance and quality of IMF research, and hence enhance utilization.
First, the relevance of research was often hampered by the lack of early consultation with country authorities on research themes and by a lack of sufficient country and institutional context. Also, authorities indicated that some important issues, such as macro-financial linkages and aspects of monetary policy, were not adequately covered. A majority of country authorities reported that they were not consulted on the choice of topics for Selected Issues Papers (SIPs), which also used analytical frameworks poorly suited to their countries’ situation and did not reflect a good knowledge of local institutions. In regions other than Africa, authorities indicated that generic regional analysis and an absence of in-depth comparative or cross-country research limited the usefulness of Regional Economic Outlooks (REOs). And many country authorities were also disappointed by the small amount of collaboration on research projects between local researchers and the IMF.
Second, the technical quality of IMF research publications was quite diverse. The WEO, GFSR, and external publications were generally of high and consistent technical quality. On the other hand, the quality of SIPs and Working Papers (WPs), which were not subject to a rigorous quality review, was lower and more variable. Some WPs made a contribution to the literature and were extensively cited, and most other WPs and SIPs were of adequate quality. The quality of other WPs and SIPs, as well as some analytical chapters of the REOs, however, was below satisfactory. This was found to be a serious concern, as most country authorities perceived these publications as having been closely reviewed, if not endorsed, by the IMF and accordingly took their findings and recommendations into account in policy analysis. Low-quality publications also negatively affected the reputation of all research. The evaluation found that short time frames and limited resources contributed to the low quality of some publications, particularly SIPs, as did the absence of IMF-wide quality standards or a uniform review process.
Third, there was a widely held perception that IMF research was message-driven. About half of the authorities held this view, and more than half of the staff indicated that they felt pressure to align their conclusions with IMF policies and positions. Policy recommendations provided in some research publications did not follow from research results. Further, a number of country authorities and researchers noted that IMF research tended to follow a pre-set view with predictable conclusions that did not allow for alternative perspectives—detracting from the quality and credibility of studies and reducing their utilization.
In addition, the evaluation found that it was difficult for country authorities and IMF staff alike to distill relevant findings and policy implications from IMF research, given its large volume and the lack of a simple way to search through the IMF’s research outputs. And finally, the evaluation found IMF-wide leadership of research lacking. Research activities were highly decentralized, with very limited coordination across departments and no mechanism to set IMF-wide priorities or quality standards. Collaboration among staff across departments was limited and mostly based on personal relationships. Many agreed that there was scope to better prioritize research activities across the IMF and to improve quality assurance.
The evaluation presented four main categories of recommendations aimed at addressing the shortcomings identified.
To enhance the relevance of research, the IMF should:
- Conduct a periodic strategic review of the function and uses of its research product lines to establish whether they should be strengthened, redesigned, or discontinued. An indicative medium-term research agenda should be prepared in consultation with member countries and the Executive Board. This agenda should be made publicly available. It should not be seen as excluding research on other themes and areas.
- Make it standard practice to consult with authorities on research topics and to discuss preliminary results with authorities and other in-country experts, in order to enhance the country and institutional context of research, particularly SIPs. Longer country assignments by mission members would also enable greater familiarity with country conditions, as would collaboration with authorities on research projects.
To enhance the technical quality of analytical work, the IMF should:
- Allocate adequate time and resources to each research project, even if this leads to fewer publications.
- Establish clear standards for technical quality of different research products. Strengthened quality assurance and review processes should be introduced to this end. For example, WPs could be subjected to a well-structured external peer review, to help promote greater openness to new and alternative ideas, as well as to weed out low-quality products.
- Strengthen incentives to improve the quality of research. For example, Management should clarify that staff annual performance evaluations will assess the quality of research as well as the quantity.
To promote openness to alternative perspectives, the IMF should:
- Allow researchers to explore issues without preconceived conclusions or messages. The Board, Management, and senior staff should actively foster an environment that encourages innovative research and should establish incentives for staff to pursue such research. After a thorough quality review, staff should be able to publish WPs and other academic-style products even when the results of their analysis are not well aligned with messages in surveillance documents.
To improve the management of IMF research, IMF Management should:
- Designate a senior staff member to be the leader and advocate of research activities across the IMF. This leader would be responsible for coordinating research activities across the IMF, including by setting standards for quality review processes and publication policies and promoting openness to alternative perspectives. The coordinator should report annually to member countries and the Board on research priorities and achievements.
Management and staff response
IMF Management found the IEO evaluation to reflect a balanced assessment of Fund research and welcomed its constructive recommendations. The Acting Managing Director also noted progress made by staff since the conclusion of the review period in tackling relevant policy issues in their research work.
IMF staff welcomed the overall findings and agreed that a number of key issues required further consideration, including suggestions for periodic strategic reviews of research, better allocation of resources to research projects, enhanced review processes, and vigilance against the risk of message-driven research. They noted that it is particularly important that IMF researchers not feel that they need to toe a line or tilt empirical results. At the same time, they expressed concern that a medium-term research agenda may limit flexibility and could be quite bureaucratic, that consultation on research topics with authorities should vary by product, and that coordination of research could stifle individual research efforts.
Executive Board discussion
The IMF Executive Board discussed the evaluation on June 13, 2011. Following are key excerpts from the Summing Up of the Board discussion.
Executive Directors welcomed the IEO report, noting that it provides a balanced assessment of the quality, relevance, management, and utilization of IMF research. …
Noting that IMF research is of uneven quality and perceived to be message-driven, however, Directors saw scope for enhancing the relevance and technical quality of the analytical work, openness to alternative points of view, and coordination of research activities across the institution. Directors looked forward to considering concrete steps to take forward the IEO recommendations, complementing efforts underway.
Directors broadly shared the main IEO findings. They concurred that, while global and core macroeco-nomic issues were adequately covered in IMF research, up until 2008, there were some gaps in the coverage of macro-financial linkages and capital account issues. They acknowledged, however, that efforts since then have narrowed these gaps, and urged staff to build on this progress. Directors also noted gaps in country-level research, especially for low-income countries where the influence of IMF research on policymaking is greatest. …
Directors considered worrisome the finding that there is a widely held perception that IMF research is messagedriven, or that policy conclusions do not always follow from the analysis. While recognizing that research produced by the IMF will inevitably carry policy messages—especially surveillance-oriented research—they considered it critical for the credibility of the institution that the conclusions of in-house research are not biased by the IMF’s position on the subject or excessively influenced by other work done internally, and, conversely, that its policy advice is grounded on robust analysis. In this context, many Directors underlined the importance of addressing concerns about the internal culture and institutional values—identified in previous IEO evaluations—with a number of Directors regarding staff diversity in terms of academic background and professional experience as critical in this regard. More broadly, Directors stressed that IMF research should aim primarily at improving the analytical tools for the IMF to carry out its core mission.
Directors agreed on the need for improved dissemination of IMF analytical work, allowing a wider group of stakeholders to distill relevant lessons and increasing its contribution to the policy debate. …
Directors broadly endorsed the main recommendations of the IEO, and looked forward to further analysis and discussion in the context of the forthcoming Management Implementation Plan.
Directors generally saw merit in conducting a periodic strategic review of research products. Management and staff were encouraged to focus on how best to allocate resources among the various research product lines … and to strengthen quality controls, the internal review process, and incentives to enhance the technical content of research, while taking into account budget constraints. More specifically, some Directors suggested re-examining the value-added of Regional Economic Outlook reports, while a few highlighted their usefulness to intended audiences. Most Directors also supported the IEO recommendation to set an indicative medium-term research agenda, possibly in consultation with member countries and the Executive Board, although a few Directors were not in favor of Board involvement in the agenda-setting process. At the same time, Directors underscored the need to retain adequate flexibility for staff to take on independent research projects.
Directors broadly supported the IEO recommendation to consult more with country authorities on research topics prepared for bilateral and regional surveillance, particularly for Selected Issues Papers. Recognizing that staff and the authorities do not always agree on the prioritization of issues, Directors agreed that staff should remain free to research the issues that they feel are most important. Longer country assignments for mission members could facilitate collaboration with authorities and enhance familiarity with country-specific conditions.
Directors agreed on the need to improve the management of IMF research and were open to the various proposals to achieve that objective. …
Management Implementation Plan and follow-up
On June 18, 2012, IMF staff circulated to the Evaluation Committee a MIP on the IEO evaluation of Research at the IMF: Relevance and Utilization—a year after this report was discussed by the Executive Board. The MIP will be reviewed by the Evaluation Committee at a date to be determined, and will subsequently be considered by the full Board.
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07
An initial MIP for this evaluation was prepared in March 2011. Following an initial discussion of this proposed MIP in April 2011, the Evaluation Committee decided to postpone formal consideration of an implementation plan until after completion of the Triennial Surveillance Review (TSR) then under preparation. IMF staff subsequently outlined plans for moving ahead in other areas (e.g., diversity, Summing Up process) that were not under discussion in the TSR.
A revised implementation plan was prepared by staff in December 2011, following the issuance of the TSR. This revised MIP was discussed by the Evaluation Committee in January 2012. At this meeting, Directors emphasized the need for further steps to address the conclusions of the IEO evaluation and also expressed a desire for more information about Management’s strategic vision on internal reforms. The Committee also decided that the MIP should be considered at a meeting of the full Executive Board. The MIP was issued to the full Board for its consideration on February 14, 2012.
On May 25, 2012, after the close of FY2012, the Board discussed the revised MIP, together with a statement by the Managing Director describing some recent initiatives undertaken to address cultural and structural weaknesses identified in the IEO evaluation. Directors generally considered that the Managing Director’s statement and the proposed implementation plan complemented well the action plan for the TSR, and together should help enhance the effectiveness of Fund surveillance.
Directors emphasized that a comprehensive long-term approach is needed to tackle the shortcomings highlighted in the IEO report. They broadly supported the specific proposals in the implementation plan, and welcomed Management’s statement on an ambitious agenda to break down silos and promote diverse views and candor, further advancing initiatives under way. Directors considered that the MIP, together with Management’s proposed agenda, provided a good start and encouraged Management and staff to continue to build on them, and where appropriate, engage the Board in the process.
Directors focused their discussion in particular on actions to: encourage candor and diverse views; speak up to power; integrate financial sector issues into mac-roeconomic assessments; break down silos; and deliver a clear and consistent message on the economic outlook and risks. Key action items included establishment of a Board Working Group on Summings Up Review, which was expected to provide its recommendations to Management before the 2012 summer recess.
Directors stressed the importance of monitoring and verifying progress on all these fronts in the context of future PMRs. They noted that changing the institutional culture is a continuous, long-term process and looked forward to revisiting this issue in one year. The Managing Director also recognized that some Directors believed that the incremental steps already under way and proposed may still not fully address remaining concerns or more fundamental problems—including governance reforms. She pledged to monitor progress and adapt over time.
A Public Information Notice with the full Summing Up of the Board discussion was posted on the IMF website on June 6, 2012.