Evaluation of the IMF's Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility
Chapter

Chapter 3 Joint Staff Assessments

Author(s):
David Goldsbrough, Isabelle Mateos y Lago, Martin Kaufman, Daouda Sembene, Tsidi Tsikata, Steve Mugerwa, Alex Segura-Ubiergo, and Jeff Chelsky
Published Date:
September 2004
Share
  • ShareShare
Show Summary Details

Key Messages

On balance, Joint Staff Assessments (JSAs) do not perform adequately the many tasks expected of them.

JSAs generally cover most of the ground suggested by the guidelines, but the clarity, candor, and comprehensiveness of the assessment are uneven, with scope for improvement even in “good practice” cases. This partly reflects shortcomings in the guidelines themselves and a built-in bias to reach a positive conclusion given the “yes-no” character of the underlying judgment on suitability as a basis for concessional financing.

Their value added to the IMF as an internal governance tool is unclear, since decisions on access to IMF concessional financing, on the nature and scope of nonfinancial assistance, and on program design are made in separate contexts without being guided by the JSA.

JSAs of PRSP Progress Reports add little value to the Progress Reports themselves.

The JSA does provide useful feedback to countries about their PRS, but lack of awareness of the JSA among nongovernment stakeholders reduces its effectiveness.

JSAs have not contributed significantly to fostering coordination among development partners, reflecting in part shortcomings in coverage of issues, and in part the limited relevance of JSAs in the eyes of many donors, who would like a greater input into the assessments.

An important part of the “architecture” of the PRS process is the JSA, by the staffs of both the IMF and the World Bank, which evaluates the strengths and weaknesses of a country’s PRSP and considers whether the PRSP provides a sound basis for concessional assistance from the BWIs and for HIPC debt relief. JSAs are submitted to the Executive Boards of the two institutions along with countries’ PRSPs and are always made public. In principle, they can serve three purposes:

  • An internal governance function: providing the Boards with a basis on which to decide whether the country’s strategy deserves concessional support, which may include debt relief.
  • A feedback and outreach function: informing the government and other domestic stakeholders of staff views on the strengths and weaknesses of the PRS.
  • A partnership function: signaling to the donor community whether the BWIs view a particular PRS as worthy of support, what areas need strengthening, and so on.

The jointness of the assessment primarily reflected the perception that it was not sensible to dissociate the assessment of the macroeconomic components of the PRS from that of the developmental components, as they would succeed or fail together. It was expected to improve Bank-Fund collaboration by ensuring that their staff reached a common perspective on the country.

Interestingly, when asked what they thought was the primary value added of the JSAs, IMF staff surveyed emphasized not the internal governance function, but the provision of feedback to the authorities and, to a lesser extent, of an independent expert assessment of the PRSP to interested parties.1 Moreover, the increase in the amount of paperwork expected from IMF staff (Box 2.3) suggests that the JSA did not make any of the preexisting staff assessments redundant. In the remainder of this section, we discuss the contents of JSAs, assess how effective they have been in meeting their different purposes, and review possible factors underlying this performance.

Contents of JSAs

The guidelines for JSAs of PRSPs indicate that the assessments should succinctly answer key questions about each of the four core elements of the PRSP (i.e., a description of the participatory process; poverty diagnosis; targets, indicators, and monitoring systems; and priority public actions), giving greatest weight to the priority public actions and to the arrangements for monitoring and evaluation of PRSP implementation, and paying due regard to the country’s starting point. We assess how the JSAs handle the key questions asked by the guidelines in the areas relevant to the IMF’s mandate.2

Participatory process

While in general the guidelines call for JSAs to assess rather than describe or repeat the contents of PRSPs, in the case of participatory processes the guidelines note that “the Executive Boards have instructed the staffs to describe, but not evaluate, the participatory process.” In practice, however, 22 JSAs out of 23 do assess the participatory process, and 16 of those identify the participatory process as one of the main strengths of the PRSP, while four mention it as one of the weaknesses.3 In about half of the cases, these assessments are not backed by an analysis of the participatory process and its impact on ownership. Several of the key questions asked by the guidelines in relation to the participatory process are frequently not addressed. Furthermore, based on our case studies, our overall judgment is that the JSAs tend to present participation in a more favorable light than warranted by circumstances (except in Cambodia, where the assessment is quite candid and in Tanzania, where the JSA refrained from assessing the participatory process). In particular, JSAs often over-state the extent and significance of civil society participation (e.g., in Mozambique, Nicaragua, and Tajikistan) and the degree to which the participatory exercises were country driven (e.g., Vietnam), while understating (and sometimes entirely overlooking) constraints to effective participation (e.g., in Guinea). A key reason why JSAs tend to present an insufficiently critical assessment of the participatory process is that (in keeping with the guidelines) they emphasize who was consulted and how without giving much information on what was discussed or whether the discussions were meaningful.

Targets and monitoring

On average, JSAs provide a good description of PRSP targets and monitoring arrangements, along with some assessment of their adequacy. However, there are wide variations: the realism of targets and monitoring arrangements and the suitability of the indicators retained are generally assessed. By contrast, the transparency and participatory nature of monitoring arrangements are discussed in just over half of the cases, and the feedback of monitoring into policymaking is discussed only in a minority of cases. This pattern is common to both early and recent JSAs.

Priority action plans

JSAs’ treatment of priority actions plans was examined focusing on three areas that are particularly relevant to the IMF’s mandate: the macroeconomic framework; key fiscal choices; and financing plans. In all three areas, JSAs offer a broadly satisfactory assessment, although with room for improvement.

  • Almost all JSAs discuss the soundness and realism of macroeconomic frameworks. However, only half discuss their robustness to shocks, and only a minority of JSAs assess whether key tradeoffs are discussed in the PRSP.
  • With respect to key fiscal choices, the three dimensions highlighted by the guidelines—internal consistency; quality of costing estimates; and administrative capacity to deliver (including quality of PEM system)—are tackled in most cases.
  • The assessment of financing plans is generally the weakest. A majority of JSAs discuss the realism of financing plans, but their sustainability is rarely addressed, nor is the existence of contingency plans.

The desk review results suggest that some learning over time has taken place with respect to the appraisal of macroeconomic frameworks, but not in the other areas.

Other issues

Surprisingly, JSAs rarely discuss “gaps” in PRSPs, in the sense of critical policy issues that are not addressed. The analysis of key implementation risks is also limited. All JSAs identify at least a couple of such risks but most do not suggest remedial actions. Furthermore, the case studies (e.g., Guinea, Mauritania, and Mozambique) suggest that JSAs are sometimes insufficiently candid in their assessment of the seriousness of the risks to implementation, especially with respect to the implications of weak administrative capacity and of inadequate links between the PRS and normal government processes.

JSA guidelines require the staff to “take into consideration the country’s starting point.” Ideally, JSAs should give a sense of what initial conditions were in key areas and, based on that, of what could realistically be expected from the PRSP process, so that the PRSP would be assessed against explicit countryspecific benchmarks. In practice, JSAs have generally not adopted such an approach. In about one-third of cases, there is a perfunctory discussion of starting conditions. In others, starting conditions are discussed adequately, but the implications for the PRSP are not clearly drawn out.

JSAs of PRSP Progress Reports

According to the guidelines for JSAs of PRSP-PRs, these reports “must make an overall assessment for the Executive Boards as to whether progress in implementation has been satisfactory, and whether or not the strategy presented in a PRSP remains a sound basis for concessional assistance from the Fund and the Bank.” We assessed the JSAs of 12 PRSP-PRs issued through the first quarter of 2004.

Surprisingly, the key questions suggested by the guidelines for staff to consider in drawing their assessment are narrowly focused on the contents of Progress Reports, rather than on actual progress in implementation. Reflecting the guidelines, JSAs focus on the PRSP-PR documents, and in many cases the latter are appraised exclusively on the basis of information reported by the authorities in the document itself.4 A significant proportion of JSAs limit themselves to summarizing the Progress Report and assessing its strengths and weaknesses without attempting to remedy gaps in coverage (e.g., in assessing progress against the targets set in the PRSP itself), by bringing in information from other available sources, or making an independent assessment of progress. Only a small minority attempt to appraise progress in implementation of the strategy against the objectives of the initiative, such as broader ownership, better design of policies and priority setting, and the extent to which the PRSP is assuming a central role in the domestic policymaking process. Among the case studies, the JSAs for Tanzania and Ethiopia are examples of “good practices” in that regard.

Value Added of the JSA

Performance of the JSA as an internal governance tool

The usefulness of the JSA as an internal governance tool is limited by the fact that JSAs are constrained to reach a binary (yes/no) conclusion on whether the strategies presented in PRSPs constitute a sound basis for concessional lending by the BWIs, with the use of standardized language.5 All JSAs to date have provided a “yes” signal and this conclusion has systematically been endorsed by the Boards—even when the PRSP did not in fact provide a satisfactory framework in which to anchor a PRGF-supported program. In these cases, the JSA typically noted serious weaknesses in the PRSP, along with encouragements to amend the PRS in a number of ways, but nevertheless reached a positive conclusion. Furthermore, what constitutes a sound basis for BWI concessional lending (or, more critically, what fails to constitute a sound basis) is not defined anywhere.

JSAs have not contributed in any substantive way to the determination of the amounts of BWI assistance and detailed design of the programs.6 All JSAs identify a (varied) number of strengths and weaknesses in PRSPs, but not in a way that would allow the reader to get a sense of the overall strength of the PRSP, that is, by delivering a graduated assessment on the basis of explicit benchmarks. For example, our review of JSAs shows that they listed more strengths than weaknesses for only 12 of the 23 PRSPs appraised up to end-2002. Sixteen PRSPs were considered “sound bases” even though they had among their main weaknesses at least two of the following—seemingly critical—dimensions: (i) costing, financing, and prioritization; (ii) public expenditure management; and (iii) implementation and monitoring. Five PRSPs were considered deficient in all three respects and yet the JSAs concluded that they provided a suitable basis for concessional lending.

JSAs typically do not discuss what the BWIs plan to do, both to support the country’s strategy and to help remedy any weaknesses (nor are they required to do so explicitly by the guidelines). In particular, they do not underline the implications of perceived weaknesses for the extent to which their own operations may be aligned on the PRSP, nor the priority actions that the BWIs themselves should take. The CAS is intended to fulfill this purpose in the World Bank. However, there is no clear equivalent in the IMF. As a result, the path from the JSA to a PRGF arrangement is often not clear, and many key issues are left to be resolved in the context of PRGF-related negotiations.7

Box 3.1.Key Findings of the Review of JSAs by the World Bank’s OED

As background to its evaluation of the PRSP process, the OED conducted a review of JSAs, assessing how well each JSA identified the strengths and weaknesses of a PRSP and signaled directions for improvements, using a four-point scale where “3” (out of “4”) indicates satisfactory treatment. This assessment was made for 11 issues, 7 of which are not listed as such in the JSA guidelines but were deemed relevant based on the case studies and on the record of discussions of the World Bank’s Executive Board. The average overall rating given by OED to the 28 JSAs reviewed was 2.7, with signs of tentative improvement over time in some respects. The topics most satisfactorily handled in a majority of JSAs were endogenous and exogenous risks, structural and sectoral policies, and poverty diagnoses. The least well covered topics were capacity constraints, private sector participation in PRSP formulation, and donor partnerships issues.

The OED review also concurs with the IEO assessment of limited usefulness of the JSA to local stakeholders and the overly optimistic or incomplete treatment of the value and contribution of the participatory process.

With regard to IMF–World Bank collaboration, the main value added of the JSA reported by IMF staff was to foster the expression of a comprehensive, unified World Bank perspective in its areas of competence. Nevertheless, there is also a view among IMF staff that the need to reach a compromise with Bank staff on issues of shared interest is an important contributing factor to the lack of candor and clarity of the assessments conveyed in the JSA. Some additional findings on the JSA emerging from the OED review are summarized in Box 3.1.

As a result of these various limitations, the main value added of the JSA from the perspective of internal governance seems to be of an informational nature, that is, by summarizing for Board members the main features of the PRSPs prepared by member countries. Even in this role, however, JSAs face shortcomings because of their limited scope to focus on the dimensions of keenest interest to each BWI. For instance, the commentary on participatory processes found in JSAs typically does not discuss whether macroeconomic policies and related issues at the core of IMF concerns were addressed, let alone report on the thrust of any debate that might have taken place.8

Box 3.2.Interactions with Local Stakeholders in the Preparation of JSAs

JSAs are meant to reflect the views of the staffs of the IMF and the World Bank and not those of a broader range of stakeholders (although the latter is not ruled out). Not surprisingly, only a small minority of JSAs (5 out of 23) make explicit references to lateral inputs, and these cases primarily reflect the views of the local donor community, although in two cases the views of civil society organizations on specific issues are also mentioned.

The case studies suggest that practice with respect to consultation with other stakeholders varies widely. For instance, in Cambodia, several key donors, including the AsDB and DFID, provided a joint review of a first draft of the JSA, and some comments were incorporated in the final version. Likewise, in Vietnam, a draft was circulated to donors and NGOs that were members of the Poverty Working Group and they commented on it (approvingly for the most part). On the other hand, the JSA for Tajikistan does not even allude to the existence of alternative views on the macroeconomic framework of the PRSP produced by an NGO umbrella group.1 In Guinea, staff in charge of drafting the JSA had exchanges of views on the PRSP with local donors and selected representatives of civil society, but they did not consult them on the draft JSA itself. Our exchanges with these stakeholders indicated that they thought the JSA misrepresented ground realities and wished they had been given an opportunity to express that view prior to consideration by the Executive Boards.

In most cases, the authorities were given an opportunity to react to a draft of the JSA. In some cases, this way of proceeding appears to have contributed to limiting the candor of the staff’s assessment, either by introducing a bias favorable to the authorities as their comments—unlike those of any other stakeholders—were taken on board (e.g., in Guinea, on the participatory process) or by glossing over disagreements with a view to maintaining a good working relationship with the authorities (e.g., in Nicaragua).

This asymmetry is the source of considerable misgivings among both donors and NGOs, who feel that, at the very least, their views on the matters assessed by the staff should be brought to the attention of the Boards. Others go further by suggesting that, where donors provide most of their assistance in the form of direct budget support, and do so on the basis of a common PAF and a Joint Partner Review (JPR) of PRS implementation—as in Ethiopia, Mozambique, and Tanzania—the JSA and the JPR should be merged or at least closely aligned in timing.

The guidelines on JSAs of PRSP Progress Reports reflect a tentative evolution toward greater lateral consultations, as they explicitly suggest that staff seek inputs from local stakeholders when assessing progress in implementation of the PRS. A desk review of JSAs of PRSP-PRs suggests that this was done in just over half of cases, with some improvement over time.2 However, in the three cases (mentioned above) where the joint donor assessment framework is most advanced, the JSAs make no reference to the JPRs, even though in all three cases such reports had been issued shortly before the finalization of the JSA.

1 These views were provided as inputs into the PRSP formulation process rather than as ex post commentary on the PRSP. Nevertheless, they would have been a relevant reference had the staff sought to reflect the views of a broad range of local stakeholders in the JSA.2 Among JSAs prepared in 2001 and 2002, only 2 out of 5 make references to local stakeholders’ views, whereas among those prepared subsequently, the ratio is 7 out of 12.

Performance of the JSA as a vehicle for feedback to countries

According to the guidelines, the JSA is expected to provide “constructive feedback to the country about how it might improve its strategy over time.” Performance in this dimension is mixed. As pointed out above, most JSAs make a relatively good diagnosis of the weaknesses of PRSPs and risks facing them, but they often fall short in suggesting implementable remedial actions. However, there is some evidence that the feedback has some impact since a comparison of weaknesses listed in the JSAs of I-PRSPs and PRSPs reveals that about half were addressed in the full PRSP.

There are other factors that reduce the effectiveness of JSAs. In particular, the fact that the guidelines effectively invite the staff to focus their assessment on the contents of PRSPs and by-products acts as an incentive for countries to concentrate their efforts in the production of a “good document,” at the expense of efforts to bring about durable improvements in underlying policy formulation and implementation processes.

Our case studies suggest that countries often face significant capacity limitations in responding to identified weaknesses in the PRSP. This problem could perhaps be mitigated if the JSAs flagged clearly the areas where these constraints are particularly binding, so as to encourage the provision of technical support from development partners.

The JSA’s larger role in providing “feedback to the country”—that is, not just the authorities, but rather all local stakeholders in the PRSP process—depends crucially upon availability of the JSA on the ground. All JSAs are in the public domain and are posted on the websites of the IMF and the World Bank when they are issued. No other staff report assessing member countries’ policies has the same degree of automatic transparency, and this practice is welcomed by INGOs and donor representatives alike. However, beyond these constituencies, it is not clear that JSAs are de facto widely accessible. This is in part because the BWIs themselves typically do not make them available on their websites in languages other than English, and in part because active public dissemination on the ground largely rests on the authorities. Our case studies suggest that in most countries civil society stakeholders—along with a number of government representatives—are unaware of the JSA. When appraised of the thrust of the JSA, local stakeholders in several of the case studies (e.g., Ethiopia, Guinea, and Nicaragua) expressed the view that the JSA gave an overly favorable assessment, not in tune with ground realities. This view applied particularly to the treatment in the JSA of the participatory process, but it also reflects dissatisfaction with the JSA’s focus on PRSP documents rather than on underlying realities.

Performance of the JSA in fostering partnership

The JSA was also expected to provide a possible common reference around which the donor community could coordinate its support to countries’ poverty reduction strategies. Acceptance of the JSA in this role among donors generally remains limited. In several country case studies, donors questioned the candor of the JSAs and/or its relevance. More generally, many expressed the view that for the JSA to play a useful role in fostering a partnership approach, it would need to take on board donors’ views explicitly rather than convey exclusively the perspective of the staffs of the BWIs. At present, JSAs typically only do the latter, although there are some exceptions (Box 3.2). While the internal governance function of the JSA requires that it primarily express the assessment of BWI staff, reporting on the views of other stakeholders, in particular donors, could only enhance the credibility and comprehensiveness of the assessment conveyed to the Boards.

The effectiveness of the JSA in fostering partnership is also adversely affected by some of the characteristics discussed earlier, in particular the absence of clear benchmarks and criteria on the basis of which to assess the PRS (i.e., both the contents of the PRSP and progress in its implementation). This effectively means the JSA is not, as presently implemented, an effective vehicle for providing inputs to the broader donor community on aid selectivity. Likewise, the absence of clear assessment in the JSA of the country’s unmet needs (whether financial, analytical, or capacity-related) to implement its PRS and of any outline of the BWIs’ intentions in terms of assistance strategy prevent the use of the JSA as an organizing framework for donors’ support to the PRS.

Thus, in practice, even in countries where a number of donors have moved to budget support and make at least part of their lending decisions on the basis of a joint assessment, that assessment coexists with the JSA.

1Reflecting the ambivalent nature of the JSA, a relatively high percentage of respondents chose the “don’t know” option when answering questions about its value added.
2This analysis is based on a desk review of JSAs issued through June 2003, with complements drawn from the case studies. The desk review involved using a standard assessment matrix to assess each JSA vis-à-vis the criteria outlined in the JSA guidelines. The coding scheme underlying this assessment and average ratings under each criterion are presented in Annex 6.
3We considered that JSAs assessed the participatory processes rather than just described them when a qualitative value judgment was expressed and/or when the participatory process was listed among either the main strengths or the main weaknesses of the PRSP.
4Such exclusive reliance on the document produced by the authorities to assess progress in implementation is bound to give rise to candor issues (see, for instance, the somewhat rosy treatment of progress under the governance pillar in the JSA of Mozambique’s PRSP-PR).
5The use of a standardized conclusion was decided in the early days of the PRSP initiative, at a time when it was tightly linked to the implementation of the HIPC Initiative. In that context, the use of the standard sentence (“The staffs of the World Bank and the IMF consider that this PRSP presents a credible poverty reduction strategy and provides a sound basis for IDA and Fund concessional assistance”) was meant to avoid language that could have given the impression that the BWIs cleared or endorsed countries’ PRSPs, as this would have run counter to the new emphasis on country ownership. While the use of standardized language made sense in the context of the HIPC Initiative, where the amount and schedule of assistance are entirely determined on the basis of objective criteria, the same cannot be said of other BWI lending decisions. Nevertheless, the use of a standardized conclusion was confirmed in the guidelines for JSAs of PRSP Progress Reports.
6The guidelines for JSAs of PRSPs note that these elements are determined through the World Bank’s CAS and arrangements under the PRGF, but that “the JSA contributes to these determinations through its assessment of the strengths and weaknesses of the strategy.” At the World Bank, the determination of lending levels continues to rely primarily on Country Policy and Institutional Assessment (CPIA) ratings (which do not incorporate information on status of the PRS process), while at the IMF access under PRGF arrangements takes into account, in a nontransparent manner, a variety of criteria (such as balance of payments need, strength of adjustment program, and track record with the IMF) that do not include PRS-related factors.
7This is in contrast with the process adopted in selected countries (including Mozambique and Tanzania) by donors who provide general budget support on the basis of a mutually agreed performance assessment framework (PAF) that builds heavily on the PRSP itself and forms the basis for annual Joint Partner Reviews (JPRs). For the IMF, there is no such straightforward mapping between PRGF conditionality, the PRSP policy matrix, and the JSA.
8Interviews with IMF staff suggest that the drafting of the section of the JSA on the participatory process is typically led by the World Bank. This choice, however, should not prevent IMF staff from assessing the coverage of macroeconomic and related issues in the participatory process.

    Other Resources Citing This Publication