Chapter 3: Cross-Cutting Issues: The Framework for Follow-Up on IEO Evaluations
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2012
The effectiveness and soundness of the institutional framework for follow-up on IEO evaluations continued to be an area of concern during FY2012. The issue was discussed by the Board, the Evaluation Committee, Management, and staff, in addition to being the focus of much attention during the conference that took place on December 6, 2011, to mark the IEO’s first decade of operations. Concerns have been expressed both about the extent of implementation of IEO conclusions and recommendations endorsed by the Board, as well as the quality and consistency of monitoring of this implementation. Key questions include:
- Do Summings Up of Board discussions accurately reflect the views expressed during the Board discussion of IEO evaluations?
- Are MIPs well aligned with Summings Up and with Board views? Do they take into account the broad conclusions and recommendations endorsed by the Board, in addition to specific actions?
- Are there sufficient systems in place for Management and staff to monitor the implementation of these conclusions and recommendations?
- Does the PMR enable the Board and other stakeholders to understand the extent of implementation?
The current framework for following up on IEO evaluations was put in place in January 2007. This followed the 2006 Report of the External Evaluation of the IEO, which identified a “clear need for more systematic monitoring of follow-up” and called on the Executive Board to take a lead role in this regard. The Board subsequently adopted a framework with two key components: (1) a forward-looking implementation plan for Board-endorsed IEO recommendations (MIP), to be provided by Management to the Board soon after discussion of the evaluation; (2) a periodic monitoring report (PMR), to be presented to the Board about once each year, on the status of implementation of actions agreed to implement Board-endorsed IEO recommendations. These reports were also intended to indicate difficulties in implementing the original plan and propose remedial or substitute actions whenever appropriate.
As noted in past Annual Reports, the Evaluation Committee and other Executive Directors have raised concerns about the follow-up process since this framework was put in place. In its discussion of the first PMR in January 2008, the Executive Board considered that monitoring would benefit from greater specificity and clarity of the follow-up actions required—including in their articulation in Board discussions and Summings Up, as well as in MIPs. In reviewing the Third PMR, Directors noted that, on some occasions, concerns raised by IEO that were broadly shared by the Board had not been addressed in implementation plans, since a specific recommendation had not been explicitly endorsed. Some Directors have also noted that more needs to be done in some cases to achieve the broader policy objectives underlying specific IEO recommendations, despite the fact that agreed implementation actions may have been completed. In discussing the Fourth PMR (as noted in Chapter 1), the Evaluation Committee requested that the subsequent PMR provide updates as necessary on broader issues raised by previous evaluations.
The need to strengthen the follow-up process also received some attention from IMF staff in FY2012. In the revised MIP for the Financial and Economic Crisis evaluation, IMF staff agreed, as a first step, to include a comprehensive analysis of all Board-endorsed IEO recommendations in future PMRs. In discussing the MIP, Directors raised a number of concerns about the process for monitoring different strands of work to follow up on this evaluation. In addition, there was also a call for Management to develop concrete and measurable norms to monitor and improve the follow-up of the IEO’s recommendations more broadly.
In an effort to help shed light on the extent of implementation of its recommendations as well as on how implementation is being monitored, the IEO undertook a study this year focused on seven evaluations completed in 2005–09.4 The study reviewed the implementation of high-level recommendations that were endorsed, at least in part, by the Board.5 The study concluded that most IEO recommendations had been implemented, to different degrees, but that issues remained with respect to many of these recommendations. Across the seven evaluations in the review sample, implementation appeared to have proceeded consistent with the Board position for about one-quarter of the high-level, Board-endorsed recommendations. For an additional 50 percent, some action had been taken, but there were issues with implementation; for instance, some actions were taken but efforts stalled before the desired results were achieved. For about 15 percent, minimal or no direct follow-up action was found.6 And for about 10 percent, insufficient information was found to assess the status.
The large share of Board-endorsed recommendations with issues still outstanding raises questions about the follow-up process, particularly given that the study examined only those recommendations that had been endorsed by the Board and addressed by MIPs. It also raises questions about the current monitoring system, since recent PMRs have concluded that all key performance benchmarks specified in earlier MIPs had either been met or were on track for timely completion and that there is no need to review current and past performance benchmarks in the following PMRs.7
The IEO study of implementation also analyzes the issues behind the delayed and/or halting implementation of a few recommendations. Some recommendations called for action on challenging issues, for which IMF staff may need time to reflect to identify solutions; other recommendations may encounter resistance from within or outside the IMF. The examples in the study are illustrative and not necessarily representative, but they may help identify weak aspects of the follow-up framework, and may assist in contemplating necessary changes.
Going forward, the IEO is considering revisiting the implementation of past recommendations in a more systematic way. The modalities for this exercise are not yet clear, but the IEO is exploring the idea of revisiting one or two of its early evaluations as a pilot project in the context of the FY2013 Annual Report. Such an exercise could examine the continued relevance of lessons from the corresponding evaluation in the current context, whether recommendations remain useful, and the extent to which they have been implemented.
This study will be published as part of the forthcoming IEO book “Independent Evaluation at the IMF: The First Decade.”
The IEO made 117 high-level and about 160 less critical recommendations in the 18 evaluations it issued during 2002–11. About 85 percent of the high-level recommendations were endorsed to some degree by the Board. In the 7 evaluations in the sample, there were 41 high-level recommendations of which 38 were endorsed by, or received partial or nuanced support from, the Executive Board (a higher proportion than in other IEO evaluations).
In some instances, IMF staff had linked particular action to certain recommendations, but the IEO concluded that these did not address the relevant concerns.
Third and Fourth Periodic Monitoring Report on the Status of Implementation Plans in Response to Board-Endorsed IEO Recommendations, October 9, 2009 and March 14, 2011, respectively. The Fourth PMR indicated that “the next PMR will provide further updates as necessary on broader issues raised in the context of this report.”