Information about Western Hemisphere Hemisferio Occidental
Chapter

Annex 4: Abstracts of IEO Background Papers33

Author(s):
International Monetary Fund. Independent Evaluation Office
Published Date:
March 2013
Share
  • ShareShare
Information about Western Hemisphere Hemisferio Occidental
Show Summary Details

“Evaluating the Quality of IMF Research: A Citation Study,” by Joshua Aizenman, Hali Edison, Larissa Leony, and Yi Sun

This study compares the performance of IMF research to that of seven peer institutions in terms of publication in refereed journals and of citations in working papers of the comparator group during 1999–2009. Publication and citations are commonly used as indicators of quality of research. The comparator group is composed of Bank of Canada, the Board of Governors of the Federal Reserve, Federal Reserve Bank of New York, Federal Reserve Bank of San Francisco, Inter-American Development Bank, Organization for Economic Cooperation and Development, and the World Bank. IMF working papers were found to be cited less often than those of the Federal Reserve, but more than those of the other comparators. Citations by the IMF and other international organizations are underestimated because the data used for the analysis excluded publications in developing countries and in languages other than English.

“Review of IMF Research on Monetary Policy Frameworks,” by Kenneth Kuttner, Petra Geraats, and Refet Gürkaynak

This study examines the technical quality of a sample of 60 IMF working papers on monetary policy frameworks. It found that the quality and value-added of IMF research on monetary policy frameworks varied considerably. Most of the working papers issued in 1999–2008 posed interesting policy-related questions and many were very skillfully executed. Some were cited extensively and made major contributions to the literature. Yet many of the papers were substandard, raising the possibility that some of the IMF’s advice might rest on less than rock-solid research. Many of the flaws in the weaker papers stemmed from a lack of proficiency with empirical methods, or from failure to articulate a well-focused research question within the context of a coherent and appropriate theoretical framework, or from less than full and detailed description of the data and methods used to generate the results. The paper offers recommendations on screening, feedback, and documentation that could help address these weaknesses.

“Review of IMF Research on Tax Policy,” by Robin Boadway, Christopher Heady, and Henrik Kleven

This study examines the technical quality of a sample of 60 IMF working papers that focus on revenue and tax policy. It found significant variability in the quality of those papers. The papers were generally well motivated and focused on policy issues that were relevant for many countries. The papers were generally well written and mostly set within the context of the relevant literature. But many fell short in the analytical execution of the research, including the formulation of the model, the innovativeness of the approach, and the empirical or theoretical analysis. This resulted in lower scores for value added than for exposition. Fund researchers’ reliance in some areas of research on a limited number of established and sometimes dated approaches may reflect an overly inward-looking approach to research. The study offers recommendations for research program management.

“An Examination of the Quality of a Sample of 60 Selected Issues Papers,” by Marcelo Selowsky and Marko škreb

This study reports on an assessment of the technical quality of a sample of 60 selected issues papers (SIPs), which were prepared as part of IMF Article IV consultations. About one-third of the evaluated papers were found to be better than satisfactory by both readers; they included very good and excellent papers. Good papers addressed well-defined and relevant questions and exhibited knowledge of country context—they made intuitive use of economics and the technique used matched the question. To be effective, these papers need to address policy issues in a way that can be understood by the economic community in the country in question. Approximately half of the papers were judged as satisfactory but exhibited specific elements of weakness. Finally, 12 percent of papers were judged to be unsatisfactory by both readers. SIPs prepared for advanced countries were typically found to be better than those for low-income countries. Common factors were identified in weak papers. They had a cryptic definition of the issue to be addressed and the relevance to the country was often not convincing; they showed a weak knowledge of basic institutional country context and often lacked the minimum data needed to address the issue. They exhibited an excessive eagerness to apply a quantitative technique without a good explanation of the economics behind the technique. These papers seem to have been prepared hurriedly and with authors having spent too little time in the respective country. The evaluation offers recommendations to improve the quality of SIPs.

“An Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” by Miguel A. Kiguel

This study examines the technical quality of the analytical chapters of the two IMF flagship reports, the WEO and the GFSR, issued during 2004–08. It found that most of these chapters provided good quality analysis of relevant macroeconomic and financial topics. It found that WEO chapters were stronger when they dealt with areas within the core mandate of the Fund, and that the GFSR chapters had greatly improved since 2007. The weaker chapters in both reports tended to lack clear analytical frameworks. The policy advice in both reports tended to be predictable and did not always follow from the analysis. Neither the WEO nor the GFSR chapters achieved an integrated view of macroeconomic and financial developments and neither foresaw the severity of the 2007–08 crisis.

“Review of IMF Regional Economic Outlook Reports, 2003–09,” by Peter J. Montiel

This study evaluates the semiannual Regional Economic Outlook. It covers all 44 REOs issued between 2003 and 2009. It found the policy analysis contained in the publications to be uneven—better in the latter years. Many of the analytical chapters were satisfactory, but some were weak. They were undermined by a tendency to advocate very specific, highly prescriptive policies rather than identifying policy options and the trade-offs often involved in conflicting economic objectives. Some reveal specific policy predilections that are not necessarily shared by the profession at large, without justifying these policies. Many of the instances of vagueness, unsubstantiated claims, missing analysis, incoherent analysis, and incorrect analysis found by the evaluation seem to arise from an uncritical acceptance of conventional wisdom. Exposing these documents to outside review before they are issued might help to address this problem.

“IMF Research on Macro-Financial Linkages,” by Gerard Caprio, Jr.

This study reviews the research on macro-financial linkages at the IMF since 2005. It found that from 2005 to mid-2007, the amount of IMF research on macro-financial issues was limited and the rather small amount of such research that was potentially relevant was not particularly well integrated with the Fund’s operational work. From mid-2007 through 2008, IMF research and operational work began to concentrate more on macro-financial-related issues. Still, even by the end of 2008, the research effort in this area was insufficient relative to its importance to the IMF operational needs. Moreover, the study found that IMF operational work made little use of Fund research on macro-financial linkages, and the main messages from research in this crucial area remained difficult to discern.

References

    Aizenman, Joshua, HaliEdison, LarissaLeony, and YiSun, 2011, “Evaluating the Quality of IMF Research: A Citation Study,” IEO Background Paper No. BP/11/01 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Banerjee, Abhijit, AngusDeaton, NoraLustig, KenRogoff, and EdwardHsu, 2006, “An Evaluation of World Bank Research, 1998–2005” (Washington: World Bank).

    • Search Google Scholar
    • Export Citation

    Bayoumi, T., 2004, “GEM: A New International Macroeconomic Model,” IMF Occasional Paper No. 239 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    Boadway, Robin, ChristopherHeady, and HenrikKleven, 2011, “Review of IMF Research on Tax Policy,” IEO Background Paper No. BP/11/03 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Botman, D., D.Laxton, D.Muir, and A.Romanov, 2006, “A New-Open-Economy-Macro Model for Fiscal Policy Evaluation,” IMF Working Paper No. 06/45 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    Caprio, Gerard, Jr., 2011, “IMF Research on Macro-Financial Linkages,” IEO Background Paper No. BP/11/07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Goodfriend, Marvin, ReinerKönig, and RafaelRepullo, 2004, “External Evaluation of the Economic Research Activities of the European Central Bank” (Frankfurt: European Central Bank).

    • Search Google Scholar
    • Export Citation

    Independent Evaluation Office of the International Monetary Fund (IEO), 2004, Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Independent Evaluation Office of the International Monetary Fund (IEO), 2005, The IMF’s Approach to Capital Market Liberalization (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Independent Evaluation Office of the International Monetary Fund (IEO), 2006, Multilateral Surveillance (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Independent Evaluation Office of the International Monetary Fund (IEO), 2007, The IMF and Aid to Sub-Saharan Africa (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Independent Evaluation Office of the International Monetary Fund (IEO), 2011, IMF Performance in the Run-Up the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    International Monetary Fund (IMF), 2003, World Economic Outlook, April (Washington: International Monetary Fund).

    International Monetary Fund (IMF), 2004, World Economic Outlook, April (Washington: International Monetary Fund).

    International Monetary Fund (IMF), 2005, World Economic Outlook, October (Washington: International Monetary Fund).

    International Monetary Fund (IMF), 2007, World Economic Outlook, April (Washington: International Monetary Fund).

    International Monetary Fund (IMF), 2008a, World Economic Outlook, April (Washington: International Monetary Fund).

    International Monetary Fund (IMF), 2008b, World Economic Outlook, October (Washington: International Monetary Fund).

    Kiguel, Miguel A., 2011, “A n Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” IEO Background Paper No. BP/11/05 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Kumhof, M., and D.Laxton, 2007, “A Party without a Hangover? On the Effects of U.S. Fiscal Deficits,” IMF Working Paper No. 07/202 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    Kumhof, M., D.Laxton, D.Muir, and S.Mursula, 2010, “The Global Integrated Monetary and Fiscal Model (GIMF)—Theoretical Structure,” IMF Working Paper No. 11/34 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    Kuttner, Kenneth, PetraGeraats, and RefetGürkaynak, 2011, “Review of IMF Research on Monetary Policy Frameworks,” IEO Background Paper No. BP/11/02 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Laxton, D., and P.Pesenti, 2003, “Monetary Rules for Small, Open, Emerging Economies,” Journal of Monetary Economics, Vol. 50, No. 5 (July), pp. 1109–46.

    • Search Google Scholar
    • Export Citation

    Meyer, Laurence, MartinEichenbaum, DouglasGale, AndrewLevin, and JamesMcAndrews, 2008, “External Review of Economic Research Activities at the Bank of Canada” (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation

    Mishkin, Frederic, FrancescoGiavazzi, and T.N.Srinivasan, 1999, “Report of the External Evaluation Committee on the IMF’s Economic Research Activities” (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    Montiel, Peter J., 2011, “Review of IMF Regional Economic Outlook Reports, 2003–09,” IEO Background Paper No. BP/11/06 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    Office of Evaluation and Oversight, Inter-American Development Bank (IDB), 2006, “Evaluation of the IDB’s Studies,” RE-323, November (Washington: Inter-American Development Bank).

    • Search Google Scholar
    • Export Citation

    Selowsky, Marcelo, and Markoškreb, 2011, “An Examination of the Quality of a Sample of 60 Selected Issues Papers,” IEO Background Paper No. BP/11/04 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation

    St-Amant, Pierre, GregTkacz, AnnieGuérard-Langlois, and LouisMorel, 2005, “Quantity, Quality, and Relevance: Central Bank Research, 1990–2003,” Bank of Canada Working Paper 2005-37, December (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation
33

The Background Papers are available at www.ieo-imf.org.

    Other Resources Citing This Publication