4 Strengthening the Performance of International Financial Institutions1
- International Monetary Fund
- Published Date:
- May 2006
This chapter examines the performance of the international financial institutions (IFIs) in taking action to fulfill their responsibilities and accountabilities within the Monterrey compact and in providing financial and technical support to developing countries. Last year’s report examined the mandate of individual IFIs; discussed their instruments of support to developing countries; and considered their performance in terms of standard indicators of lending shares and trends, support to regional and global programs, and progress with transparency and management for development results. This year’s report selectively deepens this analysis by considering information specifically collected for it and by updating information drawn from standard indicators and evaluation reports. On the basis of this information, the chapter not only assesses the performance of IFIs but also identifies remaining challenges.
IFI Financial Resources in Support of the Development Agenda
In the past 15 years, external financing directed to developing countries underwent a fundamental shift. During that time the increase in private sector flows—now the most important source of external financing in developing countries—was unprecedented. In 2003 the financial support provided by multilateral development institutions was about 10 percent of total lending and grants from both private and public institutions. Financial support (loans and grants) from the five largest multilateral development banks (MDBs) represented almost half of the financial support provided by bilateral donors. In 2005 total lending disbursements by IFIs—the five MDBs and the International Monetary Fund (IMF)—amounted to $32 billion. Recent trends in these financial flows, their composition, and their selectivity are analyzed below.
MDB Lending to Low-Income Countries
Global Monitoring Report 2005 noted a sharp increase in concessional lending (commitments) to low-income countries over the period 1999–2004. This increase was driven by volumes from the International Development Association (IDA) of the World Bank Group. However, in 2005 total MDB commitments to low-income countries fell from the high levels observed in 2004. In the case of IDA, this reduction in commitments was on the order of one-third (after a 50 percent increase the previous year) and was in part associated with resource constrains at the end of the IDA13 (13th replenishment of IDA) period.
To better reflect the trends in the flow of resources to low-income countries, this section focuses on actual disbursements rather than on lending commitments alone. Figure 4.1 shows a clear upward trend in disbursements by MDBs to low-income countries, and to Africa in particular, in the period 1999–2004. It also shows a drop in disbursements on the order of 2 percent (close to 4 percent for Africa) during 2005—a reduction much smaller than that observed in commitments, but a reduction nonetheless.
FIGURE 4.1Gross disbursements by MDBs, 1999–2005
Source: Staff of the big five multilateral development banks.
Note: Data are for calendar years.
This slowdown in disbursements is common across MDBs. In the case of IDA, it is associated with a spike in disbursements for policy lending in 2004; in 2005 these disbursements returned to 2003 levels. On the other hand, disbursements under investment lending continue a clear upward trend, as illustrated in figure 4.2.
FIGURE 4.2Trends in IDA investment and development policy lending, 1998–2005
Source: World Bank financial information.
A striking trend in development financing is the increase in lending to fragile states by IDA. This trend is explained by lending to Afghanistan, the Democratic Republic of Congo, and Nigeria, to which total disbursements rose from around $27 million in 2001 to more than $950 million in 2005.
MDB Lending to Middle-Income Countries
The disbursement volumes in figure 4.1 also confirm what appears to be a long-term trend toward contracting demand in middle-income countries (MICs). As a result, net disbursements to middle-income countries by MDBs continue to be negative, although less so than in 2003 and 2004.2 Factors underlying this trend include improved middle-income-country creditworthiness and the associated reduction in interest rate differentials with commercial lenders, as well as strong fiscal positions in several large MICs. The trend may also reflect the “cost of doing business” with multilateral institutions, for example, increased compliance (safeguard) costs for borrowers.3 This overall trend does not apply equally to all MICs. In the case of the International Bank for Reconstruction and Development (IBRD), for example, reduced demand by countries that have achieved investment grade accounts for an important part of the decline.
Composition of MDB Lending
The shares of investment and policy lending change significantly on a year-to-year basis—mostly because of sharp fluctuations in policy lending, as noted above. For the period 2003-5, policy lending, on average, represented approximately one-third of MDB disbursements. This share is the highest for Latin American and Caribbean countries and the lowest for countries in Asia. On the other hand, the sectoral composition of lending in 2005 does not show significant changes with respect to the overall trends noted in Global Monitoring Report 2005.
IMF financial support to member countries experiencing protracted balance-of-payments difficulties is given on nonconcessional terms from IMF’s General Resources Account (GRA) and through concessional loans to low-income countries under the Poverty Reduction and Growth Facility (PRGF). In general, GRA net flows are mainly dependent on the needs of large middle-income countries in the context of economic crises and are consequently erratic on a year-to-year basis. Net PRGF lending is less erratic but also substantially affected by the needs of larger low-income recipients. Although net GRA flows were negative overall in the period 1999–2005, positive net loans totaling $28.5 billion were made in the period 2001-3. These loans mainly reflected new borrowing by Argentina, Brazil, and Turkey. PRGF net lending in the period 1999–2005 amounted to negative $150 million, though disbursements in the period 2002-4 peaked with initiation of arrangements with Pakistan and Bangladesh and refinancing of the arrears of the Democratic Republic of Congo.
The IMF is continuing to refine and strengthen its support of low-income members in their efforts to achieve macroeconomic stability, growth, poverty reduction, and the MDGs. In this context, the Fund has recently taken steps to adapt its instruments and facilities:
- To improve the effectiveness of PRGF arrangements, the Fund is striving to ensure that design is consistent with the objectives laid out in Poverty Reduction Strategy Papers (PRSPs). Strong links among the PRGF, PRSPs, and the MDGs should make the Fund’s low-income-country assistance more effective.
- The Exogenous Shocks Facility (ESF) was established within the PRGF to help low-income countries deal with temporary balance-of-payments needs that arise from sudden exogenous shocks such as natural disasters or export shocks.4
- As discussed in chapter 3, the IMF participates in the Multilateral Debt Relief Initiative (MDRI), which provides additional debt relief to low-income countries.
- The Policy Support Instrument (PSI) was established to address the needs of low-income countries that may not need Fund financial assistance, but seek the Fund’s advice, monitoring, and endorsement of their economic policies. The PSI will help countries develop policy frameworks focused on consolidating macroeconomic stability and debt sustainability while deepening structural reforms in key areas that constrain growth and poverty reduction.5
Selectivity in MDB Lending
The concessional arms of the MDBs committed themselves to using more transparent and incentive-improving resource allocation systems. The goal of these commitments has been to maximize aid effectiveness and improve policies and institutions in recipient countries. At present the foundation of each of these systems is a formula that calculates the share of the resources that will be allocated to individual countries on the basis of their financial need (proxied by population, with a small correction for per capita income)6 and performance. The performance factors include measures of the quality of policies and institutions, portfolio performance, and—at the African Development Fund (AfDF), Asian Development Fund (AsDF), and IDA—an additional governance factor derived from the same index of quality of policies and institutions. Each MDB combines factors somewhat differently in its performance allocation formula and uses different methods to accommodate exceptional circumstances, such as postconflict problems.7
Global Monitoring Report 2005 examined the policy and poverty selectivity of concessional assistance by MDBs. Although other criteria have been suggested, as noted in chapter 4, these criteria are two of the most widely used by donors to determine where aid will be most effective. Figure 4.3 updates those estimates8 for 2004. It shows that MDBs continue to exhibit higher policy and poverty selectivity than bilateral aid agencies. Poverty selectivity remains at levels similar to those in 2003. Some convergence across MDBs in the extent of policy selectivity is evidenced. Naturally, measures of selectivity would be affected if different criteria (for example, vulnerability) or indicators (for example, nonincome dimensions of poverty) were to be used. Nevertheless, these estimates reflect that, according to the criteria currently recognized as dominant for aid effectiveness, MDBs are being selective.
FIGURE 4.3Policy and poverty selectivity in 2003 and 2004
Source: Dollar and Levin 2004 and World Bank staff estimates.
Among the various measures of quality of policies and institutions considered for allocation of resources, governance currently has a preeminent role. As shown in box 4.1, the allocation of aid resources is very sensitive to changes in the indicators of governance. Part II of this report analyzes both issues of measurement of governance and its influence on the overall quality of policies.
In the past year MDBs have taken significant steps to harmonize their performance-based allocation (PBA) systems. IDA, African Development Bank (AfDB), and Asian Development Bank (ADB) now use nearly identical country performance and institutional assessments (CPIA) questionnaires. Harmonization of methods for calculating and using portfolio performance measures and standards to determine the grant versus loans composition of allocated resources is now under discussion.
BOX 4.1The importance of governance in performance-based allocation formulas
A relatively recent innovation in the performance-based allocation (PBA) formula is the increased weight of governance in MDB allocations. Until 1998 the MDBs accorded governance nearly the same importance as performance factors measuring structural policies, economic management, equity and social inclusiveness, and portfolio performance. Several MDBs now give more importance to governance than to all other factors combined.
|Effective weight of governance in the performance factor|
|African Development Fund||59%|
|Asian Development Fund||53%|
|World Bank, IDA||66% (IDA 14)|
IDA and the African and Asian Development Funds have placed such importance on governance because they understand that its quality has a very positive correlation with the effectiveness of aid. Moreover, it is hoped that PBA systems are generating the proper incentives for client governments to improve the quality of their governance.
A simple sensitivity analysis helps illustrate the importance of the governance factor. Using values for an average IDA-eligible country, a change in governance performance equivalent to half a standard deviation from the median score results in a 25 percent change in the allocation of resources to a country. Similar changes in indicators for economic management or social inclusion policies result in only a 3-f percent change in resources.
Assessing MDB Results Orientation
The contribution of MDBs to the achievement of development results at the country level is heavily dependent on how they conduct their business. As discussed above, one aspect of this business is the size, composition, and selectivity of financial flows. Another aspect is the actions MDBs are taking to improve the results orientation of their own management practices and programs. The importance of adopting a managing for development results (MfDR)9 approach has been recognized among MDBs for several years. Since the first joint progress report for the Marrakech Roundtable on Results, the World Bank’s global monitoring reports have analyzed important initiatives in this area. Global Monitoring Report 2005, for example, reviewed experience with the interim results measurement system for IDA13 and noted the introduction of a refined system for IDA14, which was developed through extensive consultations with IDA donors. The IDA14 Results Measurement System (or RMS) uses a two-tiered approach, in which tier I monitors progress across 14 country-level and MDG-related outcomes and tier II uses existing internal indicators of IDA delivery and effectiveness to gauge IDA’s contribution to those outcomes. The RMS includes a special focus on building statistical capacity, particularly in four sectors related to the tier I outcomes, and is now being implemented for all IDA countries. A Web site monitoring progress against all of the indicators has been launched.
The MDBs and global monitoring reports have not had the benefit of a shared framework for gathering information on initiatives in this area. The MDB Working Group on MfDR identified, and the MDB Heads endorsed, the need for a common source of information on MDB performance and developed the Common Performance Assessment System (COMPAS), which was identified as a key action for MDBs in Global Monitoring Report 2005. The full report from the first COMPAS exercise will be available later in 2006. Given the system’s importance as a baseline to assess the performance orientation of MDBs, initial findings from the exercise are presented below.
The Common Performance Assessment System
The purpose of the COMPAS is to provide a common source of information on the results orientation of the MDBs as manifested by their internal practices and operational relations with country and development partners. The COMPAS is not designed to cover (and certainly not to substitute for) the entire performance system of each MDB. Rather, it focuses on those processes and results that are within the control of each institution and for which they are accountable. The focus is on emerging synergies as a group rather than on individual comparisons among institutions. The expectation is that a joint system will provide a basis for information exchange and lesson learning and that it will respond to MDBs’ international commitments on performance and accountability.
The COMPAS draws on the MDBs’ frameworks and action plans for implementing MfDR (see World Bank 2004; IADB 2004; ADB 2004; AfDB 2004). These frameworks and plans vary slightly among institutions, but in most cases they include three pillars that provide the structure for the COMPAS: actions to build MfDR capacity in developing countries (country pillar), actions to improve the results focus of MDBs’ internal systems and instruments (agency pillar), and actions to improve cooperation among MDBs and with other development partners in attending to the “results agenda” (interagency pillar).
For each category, the COMPAS contemplates both process and results indicators.10 Process indicators are largely descriptive and qualitative and refer to the institutional practices that are needed for an enhanced focus on results. Results indicators seek to provide a quantitative measure of the implementation of a common set of performance actions by all MDBs. The recent introduction of the COMPAS implies that in its initial report there is a preponderance of process indicators. This preponderance reflects the fact that changes in institutional practices need to be introduced and implemented before results indicators become fully available. This initial phase of the COMPAS will also be helpful in identifying difficulties arising from varying definitions and practices across institutions. This information should be instrumental in facilitating further alignment of indicators.
Analysis of data in the first COMPAS report suggests overall progress in the implementation of the MfDR agenda. Awareness of results is increasing, and frameworks, systems, and procedures are being implemented in all the institutions. The degree of institu-tionalization of the MfDR agenda, however, varies across its dimensions and among MDBs. Although the design, approval, and implementation of new procedures and systems at the institutional level is likely to take time, a key challenge for all MDBs will be to go beyond the introduction of such systems and procedures and to establish an institutional culture of using the information on results to inform decision making. Below is a summary of the progress and challenges in each of the three pillars of the COMPAS.
ACTIONS TO SUPPORT COUNTRY CAPACITY TO MFDR (COUNTRY PILLAR)
The COMPAS seeks to measure implementation of actions by MDBs oriented to helping country partners develop the will and institutional capacity needed to manage for results. MDBs have developed and begun to apply various tools and approaches to help countries assess their capacity to manage for results. The Asian Development Bank, for example, recently developed a toolkit entitled “Tool for Conducting a Rapid Assessment of Country-level Capacity for Results-based Management” for that purpose. A source-book on the principles and emerging best practices in MfDR has been produced and discussed in international workshops and will be available online in 2006. All MDBs provide technical assistance to improve capacity for MfDR embedded in projects, although the absence of a systematic monitoring of such activities does not allow measurement of the magnitude or quality of the support offered. The Inter-American Development Bank (IADB) has been particularly active in its efforts to sensitize countries to the need for capacity building for MfDR and, through its PRODEV initiative, has reached most of its client countries that now qualify to receive financial support for capacity-building initiatives.
Weak statistical systems constitute a key bottleneck to the implementation of MfDR approaches. Many countries lack basic demographic data—for example, about 53 percent of the population in Sub-Saharan Africa live in countries that have not undertaken a population census in the last 10 years. Similarly, 62 percent live in countries that have not conducted a poverty survey in the last five years, and no country in Sub-Saharan Africa has a functioning vital registration system. The international development community has agreed on a specific plan to help improve statistical systems and remedy such gaps: the Marrakesh Action Plan for Statistics.11 Estimates of the level of resources needed to close these gaps are relatively modest (on the order of $120 million per year above current levels of official development assistance).
The MDBs have a series of ongoing initiatives to help build statistical capacity in partner countries. The AfDB, for example, has provided support through a $22 million program to build statistical capacity for country-level results and economic statistics reporting. The AfDB is also playing an important role in helping its regional member countries improve the quality of their poverty-related statistics within the framework of the International Comparison Program (ICP) providing financial and technical assistance. The Trust Fund for Statistical Capacity Building (TFSCB)—a multidonor trust fund established at the World Bank in 2000—has invested over $20 million in more than 80 statistical capacity-building projects across the world. The World Bank has also developed an umbrella lending program, the Statistical Capacity Building Program (STATCAP), which promotes a sectorwide approach to statistical capacity building.
Yet experience shows that governments of many developing countries are often reluctant to borrow for statistical and management capacity. A vicious circle appears to be in operation: when policies and programs are developed and implemented without evidence of their effectiveness, decision makers find it difficult to appreciate the value of “buying” tools, including those for producing statistics that would enable them to improve the effectiveness of policies and programs. Breaking this vicious circle will require growing demand (including by citizens) for evidence-based policy making and, consequently, the production and dissemination of better data. A recent review of experience with Poverty Reduction Strategies (PRSs), along with assessment of institutional arrangements for poverty monitoring in 10 countries, reveals the large extent to which this agenda remains unfinished. Neither the donor community nor partner countries have fully internalized the importance of governments’ accountability for results to their own citizens, and not only (or mainly) to donors. Transforming the vicious circle into a virtuous one, whereby citizens demand evidence of results and governments have the systems to provide it, will most likely require a combination of changes at various levels (see box 4.2). As indicated above, additional aid resources will be required to support national efforts: MDBs and bilateral donors also have a critical role to play in providing financial aid to support the statistical capacity without which evidence-based policy is not feasible.
ACTIONS TO IMPROVE THE RESULTS ORIENTATION OF INTERNAL SYSTEMS (AGENCY PILLAR)12
The ability of MDBs to support the development of capacity to manage for results at the country level depends strongly on how well the banks’ own internal systems and instruments are adapted to achieve results. The COMPAS considers many systems, including those for country programming, project design and management, and staff training and incentives.
All MDBs are in the process of strengthening their performance orientation by moving to a more strategic approach to country programming. MDBs have issued guidelines for preparing country assistance strategies (CASs) with a results focus (that is, identifying specific outcomes to be influenced). These new guidelines have been used for most country strategies approved during 2005. Mechanisms are also being implemented to assess the result frameworks of these strategies.
BOX 4.2Conditions for evidence-based policy: lessons from the PRS experience
- High-level leadership. Monitoring systems need to be placed close to the center of government and have strong links to the budget process.
- Demand for information on results. Resource allocation systems must require information on results; otherwise, there is no incentive to collect or use such information. The PRS experience shows the important role of political and legislative bodies in expressing a demand for information that promotes accountability for results in the policy process. Some countries are in the early stages of developing systems for and approaches to increasing transparency and public accessibility of information on results.
- Line ministry capability. Effective use of information requires the availability of sufficient analytic capacities, within government or the local academic community, to ensure that raw data are collated, subjected to quality review, analyzed, and interpreted.
Nevertheless, many challenges and tensions remain. For example, the experience with results-based CASs in the World Bank revealed difficulties in linking the results of individual projects and knowledge services to the CAS results framework; a lack of baselines and targeted performance measures (most often reflecting lack of capacity and systems at the country level) that limit the effectiveness of the results matrix as a monitoring, management, and evaluation tool; the need to customize the results framework to country circumstances; and the need for further behavioral and incentive changes on part of the staff and management.13 Furthermore, although all MDBs have a system of independent ex-post review of country programs, they will need to change the nature of these reviews as they move to results-based country strategies. As the new results-based strategies will take many years to mature, MDBs will face the challenge of evaluating older country strategies, which were not designed to be evaluated, during the transition period. The implication is that even as MDBs move aggressively to base their country strategies on results, they will continue struggling with the transition from traditional, input-oriented country strategies over the next few years.
The information collected under COMPAS offers clear signs that MDBs are making substantial progress in adopting results frameworks for individual projects. For example, information about relevant project-level baseline data is available for all projects in EBRD, for most projects in IADB, and for projects in IDA. Similarly, recognition that speed of project delivery is important (because delays can generate transactions costs for country partners and slow realization of project or program outputs and outcomes) has led MDBs to simplify disbursement procedures and other operational policies.14 There is some evidence that disbursement ratios have increased since introduction of these changes. Similarly, World Bank data on project processing time show an almost 40 percent improvement in fiscal 2005 compared with fiscal 2004. Although there is no evidence so far that faster delivery of financial support to clients has come at the expense of quality, internal reviews at the World Bank find that the results orientation of projects may be negatively affected. For example, some staff members have encountered difficulties in developing satisfactory baselines at the time of project appraisal.
All MDBs now have monitoring systems through regular project supervision of all active projects. These systems are periodically improved. The World Bank, for example, recently upgraded its monitoring system (the Implementation Status and Results, or ISR), which now has a special focus on results frameworks, including the reporting of baseline, target, and result data. All MDBs have or are developing an arms-length review of self-assessed reporting. EBRD’s system, for example, includes an arms-length review at all stages for all of its projects. The World Bank’s independent quality review system covers both projects—at entry and during supervision—and analytical and advisory services (see box 4.3). IADB has a system in place for quality at entry review for projects and is developing one for supervision. AfDB has established a system to review projects at entry and is planning to launch one for supervision.15 The ADB has an established procedure for review of projects at entry, as well as an improved project performance monitoring system.
The challenge of establishing and strengthening information and monitoring systems is compounded by the need to help staff using those systems. MDBs have begun to provide training to their staff in the use of MfDR approaches. But they have not yet established clear and comprehensive training plans that clarify definitions and concepts as well as explain how the many elements of the results agenda fit together and how they relate to the roles of various staff. Furthermore, although MDBs apply MfDR dimensions in annual performance reviews of all staff, they are still creating a results culture in which the incentive and rewards structure is aligned with the results agenda.
BOX 4.3Independent quality review of analytical and advisory activities at the World Bank
A recent report by the World Bank’s Quality Assurance Group (QAG) reviewed the effectiveness of the Bank’s Analytical and Advisory Activities (AAA). It found that although the quality of individual activities is good, a stronger focus on the strategic relevance of the overall program of country AAA is necessary, and that this calls for strengthening its links with the strategic assessments embedded in the Country Assistance Strategies. The report emphasizes the critical importance of stronger outreach and dissemination efforts within countries. Practice in this area, including translation into local languages, is found to be variable across countries. To increase transparency and raise the quality of policy discussion, reports should be accessible, not only to government agencies but also to academics, civil society, and the private sector.Source: Quality Assurance Group, Country AAA Assessment.
Developing learning organizations is a complex process. A key challenge is to establish the systems and work practices that allow learning from practice to improve the quality and relevance of programs. The systems discussed here provide information that MDBs are, increasingly, distilling and making available to staff as “best practice” material. Many of these systems are limited in their capacity to provide, independently of one another, robust evaluations to inform MDBs and their country partners of the effectiveness of alternative approaches to development interventions. Progress in the area of evaluation is reviewed below.
ACTIONS TO IMPROVE INTERAGENCY COOPERATION FOR RESULTS (INTER-AGENCY PILLAR)
The COMPAS seeks to monitor how the MDBs are working with one another and with other donors to support the MfDR agenda. Given the extent of MDBs’ operational presence, these cooperative ventures could heavily influence implementation of the Paris Declaration framework.
Cooperation among MDBs has moved from ad-hoc consultation to cooperation across a broad field of issues, and the MDB heads have articulated and published joint positions on most major global development challenges. Key ongoing collaborative efforts related to the results agenda include development of the Sourcebook on Managing for Results, continuing development and implementation of the COMPAS, support of the new Mutual Learning Initiative (which will identify practical lessons in managing for results in “early mover” countries and share them with countries just beginning to implement the MfDR principles), and fostering of development of regional MfDR communities of practice. The Third Roundtable on Managing for Development Results (see box 4.4) is expected to be the major result of these collaborative endeavors.
The challenge is to translate these activities into concrete implementation at the country level through country action plans, joint strategies, and joint work (operational missions, analytical work, and so on). Progress is being made. Coordinated country strategies and analytical work have now seriously entered MDB cooperation, after a slow start,16 and inter-MDB cofinancing makes up the majority of development cofinancing.17 Continued progress will require proactive measures in each MDB. For example, the latest MDB Roundtable on Harmonization, Alignment, and Results (held at EBRD Headquarters in London in June 2005) identified the need for further changes in the way MDBs attempt to make greater use of joint missions, joint analyses, and common approaches as required for adequate implementation of the Paris Declaration.
BOX 4.4Third roundtable on managing for development results
As part of their mutual commitment to joint work and cooperation on the MfDR agenda, all MDBs participate in the Joint Venture on Managing for Development Results and the MDB Working Group. A key product of these collaborative arrangements is the Third Roundtable on Managing for Development Results. Roundtable participants will
- Assess progress since the Second Roundtable (in 2004) and the Second High-Level Forum (in Paris in 2005).
- Discuss the capacities needed to manage for results and ways to assess these capacities and foster their development, as well as follow up on the recommendations of the Marrakech Action Plan for Statistics and Paris 21 efforts by identifying ways to accelerate investments in statistical capacity at the country level.
- Develop a common capacity assessment tool to be used by both partner countries and development agencies.
- Commit to specific actions for the year ahead with a view to the Third High-Level Forum (in 2008).
Because the Third Roundtable will promote establishment of effective institutional and countrywide systems and processes to manage for results, its expected outcome is a broader, accelerated, and more rigorous implementation of the results agenda in partner countries and development agencies. Achievement of this outcome will be assessed through the Paris Declaration monitoring mechanism (specifically, indicators 1, 5, and 11).
Those changes may prove complex to the extent that they ultimately require strong managerial and staff incentives, as well as work practices in which collaboration is not merely a corporate mandate but an effective device for helping staff achieve results. The adoption of monitoring systems that track the extent of MDB harmonization should help call attention to this challenge and create incentives for implementation of the necessary changes. The World Bank, for example, has already adopted two central Paris indicators as “key performance indicators” of joint economic and sector work and lending using programmatic (“harmonized”) approaches. These indicators will help Bank management track implementation and results.18
Establishment of the COMPAS is a major step in MDBs’ effort to coordinate implementation of the MfDR agenda. Indicators may need to be added or changed over time. As noted above, the initial results are heavily biased toward process indicators. However, the mere existence of this innovative instrument and the lessons learned in its development should, with continued effort, lead to a solid system of collective performance monitoring for MDBs.
Evaluation: A Critical Component of MfDR
Achieving results (improvements in outcomes) at the country level depends on actions by many actors, including service providers, citizens, donors, and different levels of government. Disentangling the contributions of these actors and attributing results to actions of individual actors is often difficult, if not impossible. Figure 4.4 shows a simplified results chain linking actions by external actors that influence government policies and interventions, which in turn affect development outcomes. Establishing the links along this results chain is complex, but understanding the relationship between specific government interventions and policies on the one hand and outcomes on the other is often possible. Although attributing changes in outcomes to specific actions by external actors may remain an elusive goal, evaluating the impact of specific government interventions is not only important but often also feasible from a methodological point of view.
FIGURE 4.4Evaluation and the results chain
Efforts to manage for development results depend crucially on the strength of the understanding of the relationship between interventions (policies and programs) and outcomes (for example, MDGs). In those areas in which that evidence base is strong, a system that focuses on monitoring implementation and tracking outcomes often may be sufficient. By providing empirical evidence on the effectiveness of specific policies and programs to achieve development outcomes, evaluation efforts enhance IFIs’ ability to provide robust evidence-based advice to partner countries and to define the type of interventions or approaches they should support. Global Monitoring Report 2005 reviewed the critical role played in that regard by the evaluation offices at the IFIs. Special studies released during 2005 include evaluations of health and nutrition interventions by ADB’s Operations Evaluation Department, on trade facilitation and delivery mechanisms for small- and medium-size financing by EBRD’s Evaluation Department, and trade and pension reforms by the World Bank’s Independent Evaluation Group (IEG). Since publication of Global Monitoring Report 2005, the IMFs Independent Evaluation Office has produced two reports: Evaluation of the IMF’s Approach to Capital Account Liberalization (May 2005) and Evaluation of IMF Support to Jordan, 1989–2004 (December 2005). The key findings of some of these evaluations are summarized in box 4.5.
Within the multiple evaluation approaches available and effectively used by IFIs, impact evaluations play an important role. Their goal is to assess the specific outcomes attributable to a particular intervention by using a counterfactual that represents the hypothetical state the beneficiaries would have experienced without the intervention. From that perspective, impact evaluations are an important instrument to test the validity of specific approaches to addressing development challenges, such as reducing infant mortality or increasing productivity of poor farmers. They are a powerful instrument for determining what works and what does not work and thus constitute a fundamental means of identifying effective development interventions. At the same time, impact evaluations—particularly when conducted in comparable and consistent ways across countries—can provide the necessary benchmarks for program design and monitoring.
The development community is increasingly recognizing the value of impact evaluations, and MDBs are positioning themselves to play a more active role in this area. The World Bank launched the Development Impact Evaluation (DIME) initiative to promote and coordinate its impact evaluation activities. In its first year, DIME started two dozen evaluations of education interventions (focused on alternative arrangements for service delivery using the conceptual framework described in World Development Report 2004), conditional cash transfers in low-income countries, and slum upgrading initiatives. Regional units, with support from DIME, are building on the growing number of opportunities for useful evaluations in the context of Bank-supported operations. The Bank’s Africa Region Office, for example, has started work on 20 impact evaluations of projects in the areas of early childhood development, education, health, infrastructure, social protection, agriculture and environment, and private sector development.
BOX 4.5Independent evaluation at the IFIs
The shift to a more results-oriented focus in development has led to greater emphasis on evaluation of and learning from experience. Sector/thematic reports by the evaluation offices at the IFIs play an important role in this area. The following are examples of reports produced in 2005:
ADB’s Operations Evaluation Department prepared a study on the Bank’s policy for the health sector. The study found positive trends associated with the 1999 health policy, such as increased attention to supporting governance through health sector reforms and institutional capacity building and more systematic use of economic sustainability analysis in all projects. The study identified, among other challenges, the need for more coordination within ADB to integrate different aspects of the health agenda (for example, communicable and noncommunicable diseases, water, and sanitation) and for a strategy for good governance and prevention of corruption in the health, nutrition, and population sector.
AfDB’s Evaluation Department (OPEV) carried out several sector reviews for its 2005 Country Assistance Evaluations (CAEs) for Ghana, Mali, Mauritania, and Tanzania. These country evaluations focused mainly on the development effectiveness of the Bank Group’s assistance over the period covered by the previous three country strategy papers (1996–2004) and provided lessons for the next round of the papers’ preparation. OPEV found that AfDB’s strategies in all four countries were relevant, focusing on areas that were priorities for the governments. The CAE identified the need for more quantifiable, time-bound, and realistic performance indicators to better track the performance of Bank actions, as well as country-level progress in meeting agreed development goals and targets.
Because of the centrality of country strategies to the evolving work of the IADB, in 2005 the Office of Evaluation and Oversight (OVE) conducted a review of the extent to which country strategies prepared in 2004 and 2005 could be evaluated. The review found evidence of improvements over time in the results focus of country strategies but noted the persistence of several major shortcomings. In particular, country strategies generally provided a weak treatment of risk, paid insufficient attention to issues of institutional quality, and tended to define goals for the program in broad and largely unmeasurable terms.
EBRD’s Evaluation Department conducted a study to synthesize the lessons learned from the Bank’s experience with operations targeting micro, small-, and medium-size enterprises. The study identified the need for more attention to institutional aspects in project design and the importance of selecting intermediaries with sufficient institutional capacity. It found that financing projects should be embedded in a broad framework that addresses legal and supervisory issues.
The IMF’s Independent Evaluation Office (IEO) produced an evaluation of the Fund’s approach to capital account liberalization. IEO concluded that there is a need for more clarity on this approach. It suggested that the IMF could sharpen its advice on liberalization issues on the basis of solid analysis of the particular situation and risks facing specific countries. It also recommended that IMF surveillance should give greater attention to the supply side of international capital flows and to ways of minimizing the volatility of capital movements.
IEG reviewed the World Bank’s experience in supporting a wide variety of pension reforms through lending operations and analytical and advisory activities in 68 countries over the past two decades. The World Bank’s basic approach was to recommend the establishment of a multipillar pension system, provided sound macroeconomic conditions and an adequate financial sector were in place. A major finding of this evaluation is that Bank reforms often contributed to fiscal sustainability but that in many countries with multipillar systems, pension funds are poorly diversified and coverage has not increased. Secondary objectives of funded pillars—to increase savings, develop capital markets, and improve labor flexibility—remain largely unrealized.
There is increasing demand for looking across individual development organizations to evaluate and learn from the results of the development system. For the past decade the major multilateral development banks, recently joined by the IMF, have been working together through the Evaluation Cooperation Group (ECG) to harmonize their evaluation standards, in part to permit such cross-cutting analyses. This work is beginning to pay off. ECG is launching an effort to synthesize lessons on the interactions of infrastructure and environment and on the effect of their cross linkages on overall development and poverty alleviation. This effort is intended to be the first of a series that will address major development issues.Source: Staff from the IFIs.
The OVE at the IADB has also begun a program of ex-post program impact evaluations, typically through the use of nonexperimental methods.19 In 2005 OVE initiated activities to generate ex-post evaluations in three areas: youth training, rural roads, and science and technology. This new exercise will lead to production of individual working papers for each evaluation, less technical ex-post project reports that address process and institutional aspects of a project in addition to the impact evaluation, meta-evaluations (in the three areas mentioned above), and an annual ex-post report.
The success of initiatives such as the ones described above is closely linked to efforts to strengthen developing countries’ statistical and evaluation capacity. The effectiveness of these initiatives would be greatly enhanced through close coordination of MDB and donor efforts and regular sharing of information.
Alignment, Integrity, and Transparency: Progress and Challenges
IFIs are increasingly adapting their policy and operational practices to respond to the challenges of a new aid architecture that requires stronger alignment with country systems in the context of strong fiduciary integrity, as well as expanded transparency concerning their actions. This section reviews progress in these important areas.
Alignment with Country Systems
The Paris Declaration places considerable importance on the use of country systems for aid delivery and management. It calls for increased coordination of donor capacity-building support, support of national system development through increased reliance on national systems, and reduced reliance on parallel structures (such as project implementation units) when national systems have been deemed to have satisfied high standards of quality. Strengthening of fiduciary systems is integral to the development mission of MDBs, which are adopting various approaches to meet these goals. For example, the ADB and the AfDB will apply their technical assistance to enhance country systems and, with the World Bank, will test use of country systems in selected countries. Both these institutions intend to strengthen efforts to fully integrate project implementation units (PIUs) with national executing agencies and to encourage use of joint or common PIUs with other donors. The IADB sees reliance on country systems as a consequence of its work to assist borrowing countries to enhance the effectiveness and transparency of their procurement, public expenditure, and financial management systems. The IADB is working with the World Bank to develop a fiduciary capacity assessment and monitoring tool based on generally accepted practices and on baseline and performance indicators. This tool will be tested in selected countries.
The monitoring process to measure progress toward meeting these and other commitments (described in chapter 3) is at an early stage. The World Bank has identified the use of country systems as one of three key performance indicators to be tracked but has not yet established an internal process for data generation or for quality control of this indicator. Analysis of project appraisal reports for the World Bank indicates that use of public financial management systems (which are associated with development policy operations in IDA countries and with sectorwide approaches in both low-income and middle-income countries) is more frequent than use of national procurement systems. Use of environment management systems is only at the testing stage and therefore cannot be reported on until next year. A good example of alignment under firm fiduciary controls is the IMF’s system of safeguard assessments (see box 4.6).
BOX 4.6Safeguards assessments by the IMF
When the IMF lends to a member country, it does so exclusively for general balance-of-payments support, and the funds are transferred to the central bank. Thus, the Fund does not have the same procurement concerns as the MDBs; its procurement is limited to its administrative expenses. However, in 2000 the IMF introduced safeguards assessments of central banks that borrow from it. These assessments somewhat resemble the MDB practice of ascertaining the capacity of a domestic procurement agency. The safeguards were a reaction to misreporting and allegations of misuse of IMF resources involving the Russian Federation and Ukraine in the late 1990s. The primary purpose of the assessments is to provide assurance to the IMF Board that central banks have adequate financial control systems to manage their resources, including IMF disbursements. An important secondary purpose is to promote international good practices in accounting, auditing, transparency, and governance.
By the end of 2005, the IMF had completed 120 assessments of some 70 central banks, most of them in low-income countries, as reflects the orientation of the Fund’s lending. The assessments, which are not published, contained, on average, seven recommendations for improvements. When the recommendations are serious, their implementation may become a condition for the IMF loan being considered. Central banks generally act on the recommendations; the overall implementation rate is about 80 percent.Source: IMF staff.
There has been limited progress in use of country systems, in spite of the broad consensus on the significant role such use plays in improving aid effectiveness and development impact. This lack of progress is heavily influenced by fiduciary concerns and the consequent risks associated with the use of country systems. Management processes for these risks are evolving, and as a result tensions and concerns remain. Indeed, mounting international attention on fraud and corruption—as reflected in the international legal initiatives and conventions described in chapter 7—is also producing intensified scrutiny of MBDs’ anti-corruption efforts, both in-house and in their country operations. Sharpening of donor requirements (such as those under IDA14), the Sarbanes-Oxley Act of 2002, and the UN Oil-for-Food scandal (see box 7.1) have heightened pressure on MDBs to demonstrate progress in anticorruption efforts.
MDBs’ concerns about corruption range from controlling their own resources and preventing fraud and corruption in MDB-financed projects, to promoting better governance throughout country programs, to helping client countries fight corruption and money laundering (see chapter 7), to supporting international efforts to reduce corruption (such as conventions against bribery or the Extractive Industries Transparency Initiative). Preventing abuse of funds in MDB-financed projects is the focus of many rules and procedures aimed at ensuring high standards of integrity, transparency, and accountability. Concern about governance and corruption are incorporated into normal safeguards on lending—the MDBs’ fiduciary, procurement, and disbursement procedures.
MDBs are taking decisive actions to strengthen their ability to fight corruption in the use of institutional funds:
- AfDB reorganized its Internal Audit Department (AUDT) and created the Anti-corruption and Fraud Investigation Division (ACFD).
- ADB clarified the 1998 Anticorruption Policy, an important extension of the bank’s governance policy.
- EBRD’s office of the Chief Compliance Officer, responsible for integrity matters, will publish in 2006 a report of its anti-corruption activities.
- IADB created the Office of Institutional Integrity (OII) as an independent unit within the Office of the President.
- The World Bank launched its Voluntary Disclosure Program to encourage companies to volunteer information about their involvement with fraud and corruption on Bank-financed projects in exchange for reduced sanctions and assurances of confidentiality.
Despite progress, numerous challenges remain in fully tackling the corruption issue. The risk of corruption can never be entirely eliminated. In some instances there may be a trade-off between tightly containing corruption risks through projects and increasing use of country systems, but since ring fencing of projects can only constitute a temporary substitute for strengthening the overall country environment, this more challenging and lengthy task cannot be avoided. In addition, increasing flexible forms of lending in weaker governance environments may lead to a corresponding increase in allegations of corruption, sharply augmenting risk to MDBs’ reputation. A proactive and systematic management of risks has yet to be developed to avoid ad hoc reactions to instances of corruption.
Although solving the above challenges will remain difficult, improving transparency in MDBs is within the banks’ control and widely recognized as a must. New transparency initiatives are making it easier for MDB partners to scrutinize the policies that affect them and to participate more meaningfully in the development process. These new initiatives include more open policies regarding country performance assessment ratings, records of MDB board meetings, country strategies, policy papers, and project evaluations.
One of the most substantial changes over the last two years has been approval of release of country performance assessments. In the case of IDA, CPIA quintile rankings had been disclosed but not the actual scores for each criterion. In 2006 the Asian Development Bank, the African Development Bank, and the World Bank will join the IADB in disclosing country performance assessment ratings, which largely determine the allocation of those banks’ concessional funds. Additionally, the banks are working to maximize understanding of the country ratings by disseminating knowledge about how scores are determined and by releasing country ratings on the criteria that determine the overall score.
In 2005 the same MDBs introduced policies mandating much broader disclosure of the information on which they base decisions that have major consequences for stakeholders. In April the Asian Development Bank approved a new public communications policy that provides easier access to board minutes and meeting summaries, country strategy documents, economic and sectoral studies, and project evaluation reports throughout the life of the project, from preproject environmental impact studies to postproject impact evaluations. The World Bank and the African Development Bank substantially amended previously approved disclosure policies in 2005. The World Bank is testing disclosure of draft operation policies when they are first circulated to the board. If such disclosure is adopted as a regular practice, the World Bank would be the first MDB to release such documents. The Bank will also now disclose minutes from formal board meetings. The African Development Bank publishes summaries of board discussions. In early 2006 EBRD will introduce new elements to its public information policy, such as disclosure of Board minutes and draft country strategies.
In 2005 the IMF conducted a review of its 2001 transparency policy. The report noted, in particular, that publication rates for country reports had increased markedly since the last review; 77 percent of all Article IV and Use of Fund Resources staff reports are made public (up from 67 percent in the previous review). Regional differences in publication rates also declined sharply. But the review also highlighted a lengthening in the lag between board discussion and publication of country documents (from one month to one and a half months). The review explored various operational aspects of the publication policy, including the use of corrections and deletions, implementation costs, impacts on candor, and the timeliness of publication. The IMF Board agreed to staff recommendations to address those concerns.
A key theme of this chapter is that the contribution of MDBs to development results at the country level depends crucially on how the banks manage themselves. The evidence reviewed clearly indicates that the MDBs have recognized this challenge and are implementing a variety of monitoring and assessment systems—a trend that should continue.
The question, however, is how and whether those systems are being used. This chapter provides some indication that the efforts have not sufficiently permeated the fabric of MDBs. Early assessment suggests the difficulty of switching from input-driven to outcome-focused practices (for example, difficulty in linking individual project results and CAS results frameworks, a strong culture of focusing on deliverables rather than on measurable results, and slow feedback from monitoring and evaluation systems to managerial decisions). Not surprisingly, it is still much easier to systematically report on the number of projects approved or moneys disbursed than on achievements at the country level. Moreover, application of these new results-focused management systems is unequal within each institution.
Time may be part of the answer: some processes will require more time to be fully developed and implemented. But as this chapter suggests, MDBs will need to make further efforts: apply result frameworks at the country and project level, provide staff training and incentives, build statistical capacity, increase harmonization, and so on.
The complexities of the next steps should not be underestimated, not only because they may be technically hard, but because they will, by their own nature, create tensions. Achieving results at the country level will most likely require changes in resource allocation and internal decision making, and even stronger incentives to staff and managers. Furthermore, progress in the implementation of the results agenda depends crucially on efforts to strengthen partner governments’ MfDR capacity. But capacity is at a nascent stage in many countries, and MfDR is likely to be a long-term endeavor, requiring an enhanced MDB focus on actions over a period longer than that traditional in the institutions’ programs. Similarly, harmonization and alignment—two key dimensions of the MfDR agenda—will create tensions—for example, fiduciary concerns associated with use of country systems.
The implication is not that pursuit of the agenda should be abandoned or that “undue realism” should lead to conformism. Instead, the implication is that strong leadership will be needed to continue moving forward. In other words, the message of this chapter is that doing a little bit more of the same thing will not be sufficient. The good news is that all MDBs have begun valuable initiatives—many of which this chapter reviewed. In particular, the COMPAS is a valuable exercise that needs to be improved and deepened. Its usefulness will remain limited unless performance indicators are regularly updated and assessed. Moreover, new dimensions—for example, impact evaluation and transparency—may need to be added. Each MDB is likely to face different bottlenecks in implementing the MfDR agenda. A common assessment system will not, per se, address them. Each institution will need to define the set of instruments (incentives, rules, practices) best suited to its own characteristics. But a systematic, transparent, and clear monitoring of performance indicators should help all MDBs continue moving in the right direction.
|Performance category||Subcategories to be monitored||Process indicators||Results indicators|
|1. Capacity building (CB)||1a Public and private sector CB needs assessment||CB needs assessment tools available||% of countries with needs assessments completed|
% country strategies with analysis of capacity for MfDR
|1b MfDR sensitization and knowledge sharing||MfDR knowledge-sharing mechanisms established (workshops, networks, etc.)||% target audience in developing countries covered|
|1c MfDR CB programs||CB programs launched where relevant||% of target countries with CB programs under way or completed|
|2. Financing||2a Performance-based resource allocation for low-income countries||Country performance indicators in place for eligible countries||% concessional resources allocated to countries with high scores according to each MDB’s PBA formula|
|3. Strategies (country, sector, regional)||3a Outcome focus, evaluability||Revised guidelines issued to enhance results focus of strategies|
Mechanisms in place for reviewing the results focus of strategies
|% of borrowers with MDB strategies based on revised guidelines|
% of country strategies with independently reviewed evaluability
|4. Projects and programs||4a Outcome focus, evaluability||Project results frameworks reviewed and improved where necessary||% with satisfactory results frameworks as defined by each MDB|
% with satisfactory baseline data
% of projects whose evaluability is independently reviewed
|4b Project and program delivery (speed of disbursement and implementation)||Simplification of disbursement procedures|
Simplification of operational procedures
|Actual annual disbursement as % of amount available for disbursement at beginning of the year|
% of projects under implementation whose implementation is delayed beyond their original date of completion
|4c Quality of design and supervision||Increased arms-length scrutiny of project quality at entry|
Quality of supervision reviews in place or under development
|QAE % satisfactory or better|
Quality of supervision satisfactory or better
|4d Management of project risk||Early warning system in place Proactive management of projects at risk||% projects at risk or on alert status|
% projects at risk actively managed
|5. Monitoring and evaluation||5a Monitoring instruments, procedures, practices||Active project monitoring system in place with appropriate levels of information||Monitoring/supervision compliance rates|
|5b Quality assurance systems||Arms-length review of self-assessed reporting||Arms-length review reports|
|5c Independent evaluation||Ex-post evaluation of country programs|
Ex-post evaluations of projects and programs
|% active borrowers that have a country program evaluation|
% projects and programs with satisfactory outcomes
|5d Results reporting system||Adequate resourcing of M&E activities|
Regular and extensive reporting of M&E data
|% admin budget allocated to internal M&E|
|6. Learning and incentives||6a Capturing and using good practice||Good practice materials collected, field tested, and made available online||Management uptake of evaluation recommendations as reported to executive boards|
|6b Staff training and guidance||MfDR training packages developed and field tested|
On-the-job help lines established
|% operational staff participating in MfDR training|
|6c Staff incentives||Formal and transparent use of MfDR dimensions in annual personal performance reviews||% operational staff covered by results-focused incentive system|
|7. Working with other donors on the MfDR agenda||7a Information sharing||Participation in MDB MfDR working group|
DAC JV on MfDR
|MfDR Sourcebook put online and updated regularly|
|7b Harmonization||Areas for MfDR harmonization identified by working group||Number of MfDR products and processes for which common principles are agreed|
|7c Cooperative/joint ventures||Procedures developed for joint country, sectoral, and regional programming|
|Number of joint MfDR activities/programs|
Annual COMPAS reporting