Chapter

CHAPTER 4 Macroeconomic Adjustment in IMF-Supported Programs: Projections and Reality1

Author(s):
Alessandro Rebucci, and Ashoka Mody
Published Date:
April 2006
Share
  • ShareShare
Show Summary Details
Author(s)
Ruben Atoyan and Patrick Conway  Marcelo Selowsky and Tsidi Tsikata

This paper examines the accuracy of IMF projections in 175 programs approved in the period 1993–2001, focusing specifically on ratios of the fiscal surplus to GDP and external current account surplus to GDP. Four potential reasons for the divergence of projections from actual values are identified: (a) mismeasured data on initial conditions; (b) differences between the “model” underlying the IMF projections and the “model” suggested by the data on outturns; (c) differences between reforms and measures underlying the projections and those actually undertaken; and (d) random errors in the actual data. Our analysis suggests that while all are important, incomplete information on initial conditions is the largest contributor to projection inaccuracy. We also investigate the role of revisions over time in projection error, and find that they improve projections for fiscal account data, while the current account continues to in dicate a great deal of variability in the revision process.

Introduction

In this paper, we examine the accuracy of IMF projections associated with 175 IMF-supported programs approved in the period 1993–2001. For each program, the IMF staff prepares a projection of the country’s future performance. This projection is based on the country’s initial situation and upon the predicted impact of reforms agreed on in the context of the IMF program.2 We focus on the projections of macroeconomic aggregates—specifically, on the ratios of fiscal surplus to GDP and of current account surplus to GDP—during the years immediately following the approval of the IMF program. We will compare these projections to the actual data for the same years.

Our comparison is statistical. We begin with descriptive statistics for the two macroeconomic aggregates, and demonstrate that the projection deviates substantially from the observed value. We then use a simple vector autoregressive model of the determination of these two aggregates to decompose the deviation into components. We find that the “model” revealed by IMF staff’s projections differs significantly from the model evident in historical data. We also find, however, that a substantial amount of the deviation in projections from historical values is due to the incomplete information on which the IMF staff bases projections. We provide a complete decomposition of these effects. We also investigate the degree to which revisions to the projection eliminate these deviations owing to incomplete information. We find that revisions tend to approximate more closely the historical data, but that substantial differences remain between the revised projections and the historical data.

The data we analyze come from two distinct sources. The projections (also called “envisaged” outcomes) are drawn from the Monitoring of IMF Arrangements (MONA) database.3 The data on historical outcomes are drawn from the World Economic Outlook (WEO) database of the IMF as reported in June 2002. Given the difference in sources, some data manipulation is necessary to ensure comparability.4 The data are redefined in each case to be relative to the initial program year: it is denoted the “year T” of the program.5 We will examine four projection “horizons” in this study. For each projection horizon, we will compare the IMF staff projection with the historical outcome. The year prior to “year T” is denoted T−1. The horizon-T data will be projections of macroeconomic outcomes in period T based on information available in T−1: in other words, a one-year-ahead projection. The horizon-T+1 data are projections of macroeconomic outcomes in period T+1 based on information available in T−1, and are as such two-year-ahead projections. The horizon-T+2 and horizon-T+3 projections are defined analogously. The number of observations available differs for each projection horizon owing to (a) missing projection data, or (b) projection horizons that extend beyond the end of the available historical data. The numbers of observations available for comparisons are as follows for horizons T through T+3, respectively: 175, 147, 115, and 79.

We will focus on two macroeconomic aggregates. The historical fiscal surplus as a share of GDP for country j in year t will be denoted yjt. The historical current account surplus as a share of GDP will be denoted cjt. The projections of these two variables will be denoted ŷjt and ĉjt, respectively. Other variables will be introduced as necessary and defined at that time. It will be useful for exposition to describe projections of these ratios as the change observed in the ratio between period T−1 (just be fore the program began) and the end of the time horizon. We use the notation Δŷjk and Δĉjk to represent the change in the projection ratio between period T−1 and the end of horizon k: for example, ΔĉjT= ĉjTĉjT−1. Historical data from the WEO are differenced analogously.

Each program is treated as an independent observation in what follows. However, it is important to note that the database includes multiple programs for many participating countries. These programs may overlap for a given country, in the sense that the initial year (year T) for one program may coincide with a projection year (e.g., year T+2) for a previous program in that country.

What Does the Record Show?

For an initial pass, we compare the historical outcomes for the countries participating in IMF-supported programs with the outcomes projected by IMF staff when the programs were originally approved.6 When we compare the mean of Δŷjk and Δĉjk for various projection horizons k with the mean of the actual Δyjk and Δcjk, we find that projections differ substantially from those actually observed. Figure 1a illustrates the pattern of mean changes in projected and historical fiscal ratios.7 The two mean changes are nearly coincident for horizon T, while for longer horizons the historical and envisaged changes diverge sharply. The mean projected change in the fiscal ratio rises with the length of the horizon; at horizon T+3, the projected change in the fiscal ratio is 3.5 percentage points. The change actually observed over those time horizons was quite different: 0.68 percentage point for horizon T+1 and up to 1.12 percentage points for horizon T+3.

Figure 1aMean Historical and Projected Changes in Fiscal Ratios

Cumulative change as percentage of GDP

Figure 1b illustrates the pattern for changes in projected and actual current account ratios. The mean projected change in the current account ratio is negative for horizon T and horizon T+1. The change becomes positive and grows for longer projection horizons. The historical change in the current account ratio for participating countries followed a different dynamic: improvement for horizon T, followed by deterioration in longer horizons. Negative changes in mean current account ratio continued three and four years after adoption of the IMF program.

Figure 1bMean Historical and Projected Changes in Current Account Ratio

Cumulative change as percentage of GDP

While these mean differences are suggestive, they cover up the great variability in projections and realizations for both ratios. Figures 2a through 2h illustrate the historical and envisaged changes in each variable for each projection horizon. The diagonal line indicates values at which projected change is just equal to historical change. The dispersion in values around the diagonal lines is quite striking.8 For both fiscal and current account ratios, there is evidence of projected changes exceeding historical changes. This is especially striking for the fiscal ratios over time, as the proportion of observations to the right of the diagonal line rises with the projection horizon. There is also evidence of historical values more extreme than projected, especially for the changes in the current account ratio.

Figures 2a–2hFiscal Ratios and Current Account Ratios

HorizonsTT+1T+2T+3
Fiscal ratio (Δyjk, Δŷjk)0.610.560.310.56
Current account ratio (Δcjk, Δĉjk)0.540.380.320.38

The correlations between projected and historical changes of the two ratios over the various time horizons are as follows:

There is a good, though not perfect, correlation between projected and historical changes for horizon T. For longer projection horizons the correlation is lower. The horizon T+2 correlations exhibit the lowest values, with horizon T+3 correlations rising again to equal those of horizon T+1.

It is not surprising that the projections are inexact at any projection horizon. Nor is it surprising that the shortest horizon exhibits the closest fit to the actual, since longer-horizon projections required predictions on intermediate-year outcomes that almost surely will be inexact. It will be useful, however, to decompose the projection error into parts—can we learn from the record to identify the source of the projected imprecision?

Decomposing Projection Error

Begin with gT, a macroeconomic variable observed at time T. Define sT as the vector of policy forcing variables observed at time T. Denote the projection of ΔgT to be

with XT−1 a matrix representing that information available to the forecaster at time T−1 and ŝT the matrix of projected policy outcomes consistent with the government’s letter of intent.9 The actual evolution of the variable gT can be represented by the expression

with ζT−1 the matrix of forcing variables at time T−1 (including a random error in time T), sT the matrix of observed policy outcomes, and ϕ the true reduced-form model. Projection error can then be represented by the difference (ΔĝT − ΔgT).10

There are four potential sources for this projection error. First, the projection model f(∙) may not be identical with the true model ϕ(∙). Second, the historical policy adjustment (ΔsT) may differ from the projected policy adjustment (ΔŝT). Third, the information set XT−1 available for the projections may not include the same information as the forcing vector ζT−1 for the true process. Finally, there is random error in realizations of the macroeconomic variable.

Consider a simple example. There is a single projection of change in a variable gT. The forcing matrix is simply the lagged variables gT−1 and gT−2.11 The policy matrix is represented by the single instrument sT.

Table 1Projecting the Change in Macroeconomic Aggregates
Horizon T
VariableSimple Statistics
NMeanSth devSumMinimumMaximum
ΔŷjT1751.086512.88011190.14000-7.6000012.50000
ΔyjT1750.877783.25935153.61161-11.3389612.72751
ΔĉjT175-0.221873.47920-38.82699-13.8923611.66200
ΔcjT1750.723404.77454126.59449-17.6898614.49604
Correlations:ΔŷjTΔyjTΔĉjTΔcjT
ΔŷjT1.00000
ΔyjT0.604891.00000
ΔĉjT0.242560.123341.00000
ΔcjT0.199680.303030.534861.00000
Horizon T+1
VariableSimple Statistics
NMeanStd devSumMinimumMaximum
ΔŷjT+11471.624083.22486238.74000-5.6000012.90000
ΔyjT+11470.677223.9329899.55073-19.4293513.69233
ΔĉjT+1147-0.378674.98040-55.66390-22.2318712.01531
ΔcjT+1147-0.032337.07135-4.75294-35.6117625.90529
Correlations:ΔŷjT+1ΔyjT+1ΔĉjT+1ΔcjT+1
ΔŷjT+11.00000
ΔyjT+10.561821.00000
ΔĉjT+10.13572-0.021651.00000
ΔcjT+10.124530.043580.382541.00000
Horizon T+2
VariableSimple Statistics
NMeanStd devSumMinimumMaximum
ΔŷjT+21152.594783.67922298.40000-3.3000015.60000
ΔyjT+21150.818074.8555394.07814-16.7211711.88877
ΔĉjT+21150.647424.6489774.45280-22.0428011.68855
ΔcjT+2115-0.490567.32857-56.41476-38.1474321.78397
Correlations:ΔŷjT+2ΔyjT+2ΔĉjT+2ΔcjT+2
ΔŷjT+21.00000
ΔyjT+20.310461.00000
ΔĉjT+20.11603-0.218401.00000
ΔcjT+2-0.03683-0.116060.323651.00000
Notes: Std dev denotes standard deviation. N denotes the number of observations.
Horizon T+3
VariableSimple Statistics
NMeanSth devSumMinimumMaximum
ΔŷjT+3793.510004.27596277.29000-2.7000019.50000
ΔyjT+3791.119184.8532088.41557-17.4899413.35470
ΔĉjT+3791.281984.91608101.27681-19.8959414.73079
ΔcjT+379-1.3758712.09842-108.69398-81.56932121.75981
Correlations:ΔŷjT+3ΔyjT+3ΔĉjT+3ΔcjT+3
ΔŷjT+31.00000
ΔyjT+30.558901.00000
ΔĉjT+30.14194-0.124991.00000
ΔcjT+3-0.028290.011130.385301.00000
Notes: Std dev denotes standard deviation. N denotes the number of observations.
Notes: Std dev denotes standard deviation. N denotes the number of observations.

Equations (1) and (2) can then be rewritten in the following form:

The coefficients (α1, α2, β1) represent the true model while (a1, a2, b1) are coefficients from the model used for projections. In the projection rule, the forecaster perceives ĝT−1 = (gT−1 + ηT−1) with ηt−1 a random error. This imprecision may occur be cause the information set available to the forecaster is less precise than the set available after later revisions. The variable εT represents the stochastic nature of realizations of the actual variable.

The projection error thus illustrates the four components mentioned previously. First, there is the possibility that the forecaster’s model differs from that evident in the historical data; this will lead to the deviations summarized within the first set of square brackets. Second, there could be a divergence between the projected policy adjustment and the actual policy adjustment. Third, there is the potential that projection error is due to mismeasurement of initial conditions or to past errors in forecasts of variable growth. Fourth, the error may simply be due to the stochastic nature of the variable being projected.

In the sections that follow, we decompose the projection error into these four parts for the fiscal balance/GDP ratio and the current account balance/GDP ratio in countries with IMF-supported programs. First, we create a reduced-form model that represents well the evolution of the actual data. We estimate the model implicit in the projected data, and compare the coefficients from this projection model with those from the actual data. Second, we examine the envisaged and historical data for evidence that revisions in the data led to the discrepancies. Third, we perform a decomposition exercise to determine the percentages of deviations of projection from historical values that can be attributed to differences in models, differences in initial conditions, differences in policy response, or simply random variation in the historical data.

Fiscal and Current Accounts

We begin with the macro identity

holding for all countries j and time periods t. yjt is the fiscal surplus as a share of GDP, cjt is the current account surplus as a share of GDP, and pjt is private savings as a share of GDP.

We posit as well that there is a “normal” level of private saving specific to each country and to each time period. This normal level pjtn can be represented by a country-specific component, a component that is common to all countries for a given time period, and a positive relationship between foreign saving opportunities and private saving.

Combining (4) and (5), and defining ejt=(pjtpjtn) as the excess private saving in any period yields

The variables yjt and cjt can be represented by a vector autoregression. With appropriate substitution, this vector autoregression can be rewritten in error-correction form as12

There is, in general, no way to assign contemporaneous causality in (7a) and (7b). If it were possible to assert that the current account ratio is exogenously determined, for example, then the contemporaneous change Δcjt could be a separate regressor in the Δyjt equation to account for that contemporaneous correlation.

The econometric effects modeled here can be divided into three groups. The first group, represented by the terms in Δcjt−1 and yjt−1, capture the autoregressive structure of the system. The second group, represented by the terms in yjt−1 and cjt−1, capture the adjustment of these variables in response to deviations from the “normal” relationship described in (6). The third group represents random errors. Al though the direction of contemporaneous causality cannot be verified, there is a version of dynamic causality that can be checked. The coefficients of yjt−1 and cjt−1 represent the degree to which the cur rent account and fiscal ratios respond to deviations from the norm.

The system of equations in (7) will hold for all t, and thus should be in evidence at time T when the IMF-supported program is introduced. The system has excluded policy interventions from the deriva tion for simplicity, but it is straightforward, though messy, to introduce them. One way to do so will be through definition of a policy response function, by which ΔsjT is itself a function of cjT−1 and yjT−1. The second will be to incorporate the policy variables as exogenous forcing variables. The approach we use will incorporate parts of each.

Estimation Using Historical Data

The results of the coefficient estimates from equations (7) for all programs in the sample at horizon T using historical data are summarized in Table 2a. Specification testing revealed that lagged first-difference terms with lag length greater than two did not contribute significantly to the regression.13 The contemporaneous causality imposed on the model is that changes in the fiscal account are caused by changes in the current account, and not vice versa.14 The error-correction term (ejT−1) was derived from the regression in levels (i.e., not first differenced) reported in Annex I.

Table 2aRegression Results, Historical Current and Fiscal Account Ratios, Horizon T
ΔyjTΔcjTΔyjTΔcjT
CoefficientS.ECoefficientS.E.CoefficientS.E.CoefficientS.E.
ΔcjT0.28**(0.06)0.25**(0.05)
ΔyjT−10.25**(0.10)-0.08(0.19)0.23**(0.10)-0.04(0.20)
ΔcjT−1-0.05(0.05)-0.04(0.10)-0.02(0.05)-0.23**(0.09)
ΔyjT−20.16**(0.08)-0.01(0.16)0.14*(0.08)0.13(0.16)
ΔcjT−2-0.07*(0.04)-0.02(0.08)-0.05(0.04)-0.17**(0.07)
yjT−1-0.82**(0.11)-0.09(0.21)
cjT−10.16**(0.07)-0.40**(0.12)
ejT−1-0.81**(0.11)-0.17(0.22)
N176176176176
R20.780.560.780.50
Notes: Full sample, Horizon T. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent level of confidence.** Indicates significance at the 95 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Full sample, Horizon T. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent level of confidence.** Indicates significance at the 95 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

For the ratio of fiscal balance to GDP, the estimation results suggest the following insights (see the first two columns of Table 2a):

  • There is significant positive contemporaneous correlation between the two variables, and the normalization chosen here assigns causation to ΔcjT. For a 1 percent increase in the current account ratio, there is a 0.28 percent increase in the fiscal ratio.
  • The current first-difference responds positively and significantly to shocks in the own ratio in previous periods. For a unit shock to ΔyjT−1, there are other things equal a 0.25 increase in ΔyjT. For a unit shock to ΔyjT−2, the transmitted shock is positive and significant at 0.16. Past positive current account shocks have small negative effects on ΔyjT with the two-period lagged effect significant at the 90 percent level of confidence.
  • The coefficient on yjT−1 is significantly different from 0, but not from −1. It implies that for an average country, a deviation from its “normal” fiscal account ratio will lead to an adjustment in the next period that erases 82 percent of that deviation.

For the ratio of current account to GDP, the estimation explains a lower percentage of the variation (as indicated by the R2 statistic of 0.56). The second set of columns reports coefficients and standard errors for that specification, and indicates the following:

  • The lagged first-difference terms have no significant effect on the current first difference.
  • The coefficient on cjT−1 of −0.40 is significantly different from both 0 and −1. It indicates that 40 percent of any deviations of the current ac count ratio from its normal value is made up in the following period.

The last four columns of Table 2a report the results of error-correction regressions in which yjT−1 and cjT−1 are replaced by ejT−1 from equation (6), as implied by a cointegrating relationship between the two variables. As is evident in comparing the first set and third set of results, the cointegrating relationship captures nearly all of the explanatory power in the ΔyjT regression. The cointegrating relationship is less effective in the ΔcjT equation, how ever, as indicated by the R2 statistic.15

These results are specific to the data for horizon T. Results for horizon T+1 are presented in Table 2b. The construction of these data differs somewhat, in that the endogenous variable is a two-period forecast; we chose to use two-period lags on the right-hand side of the equation for comparability. For horizon T+1, the contemporaneous effect of the current account ratio on the fiscal ratio is halved—this is perhaps due to the doubling of the length of the time horizon. The autoregressive structure of the fiscal ratio, significant in horizon T, is no longer significant for horizon T+1. By contrast, the lagged “level” effects have larger coefficients. This effect in the current account ratio equation is significantly larger, as well, with the coefficient (−0.833) more than double the comparable term for horizon T (−0.40).

Table 2bRegression Results, Historical Current and Fiscal Account Ratios, Horizon T+1
ΔyjT+1ΔcjT+1ΔyjT+1ΔcjT+1
CoefficientS.ECoefficientS.E.CoefficientS.E.CoefficientS.E.
ΔcjT+10.14**(0.06)0.12**(0.05)
ΔyyjT−10.09(0.11)-0.15(0.24)0.08(0.10)0.01(0.26)
ΔcjT−1-0.04(0.07)-0.12(0.16)-0.01(0.06)-0.55***(0.12)
yjT−1-1.13***(0.17)0.47(0.37)
cjT−10.23**(0.10)-0.83***(0.21)
ejT−1-1.10***(0.16)0.27(0.41)
N147147147147
R20.830.720.830.65
Notes: Variable definition (this table only for all variables gj):

  • ΔgjT+1= gjT+1gjT−1
  • ΔgjT−1 = gjT−1gjT−3

Full sample, Horizon T+1. Standard errors (S.E.) appear in parentheses.** Indicates significance at the 95 percent confidence level.*** Indicates significance at the 99 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Variable definition (this table only for all variables gj):

  • ΔgjT+1= gjT+1gjT−1
  • ΔgjT−1 = gjT−1gjT−3

Full sample, Horizon T+1. Standard errors (S.E.) appear in parentheses.** Indicates significance at the 95 percent confidence level.*** Indicates significance at the 99 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

Estimation Using Projected Data

If we interpret the estimated model of the preceding section to be the “true” model (2), we posit that the model used in forming projections for IMF programs should have a similar form. We can use similar econometric techniques to those of the previous section to derive the economic model implied by the projections. We report the results of this estimation exercise in Table 3a for projection horizon T.

Table 3aRegression Results, Envisaged Current and Fiscal Account Ratios, Horizon T
ΔŷjTΔĉjTΔŷjTΔĉjT
CoefficientS.ECoefficientS.E.CoefficientS.E.CoefficientS.E.
ΔĉjT0.15*(0.09)0.20*(0.08)
ΔŷjT−1-0.15*(0.09)-0.15(0.12)-0.14(0.09)-0.15(0.13)
ΔĉjT−1-0.09(0.06)-0.0004(0.08)-0.09(0.06)-0.06(0.09)
ΔŷjT−2-0.03(0.08)0.16*(0.10)-0.05(0.08)0.11(0.11)
ΔĉjT−2-0.06(0.04)0.04(0.06)-0.08*(0.04)-0.04(0.06)
ŷjT−1-0.44**(0.09)-0.02(0.11)
ĉjT−10.08(0.07)-0.33**(0.08)
emjT−1-0.44**(0.09)-0.04(0.12)
N165165165165
R20.850.760.850.69
Notes: Full sample, Horizon T. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Full sample, Horizon T. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

The results from estimating the projection model for the fiscal ratio suggest the following (see the first set of columns in Table 3a):

  • There is significant contemporaneous correlation between the projected fiscal and current account ratios. For a 1 percentage point increase in the current account ratio, there is evidence of a 0.15 percentage point increase in the fiscal ratio. This is roughly half of the response found in the actual data. By implication, the IMF staff model will project a 0.85 percentage point increase in the ratio of private net savings to GDP in response to such a current account shock, while the historical data indicate a 0.72 percentage point increase in the private saving ratio in response to such a shock.
  • A 1 percentage point increase in last period’s fiscal ratio will trigger an 0.15 percentage point decrease in this period’s ratio. This suggests that the projection is relying on fiscal policy correction to overcome any inertia in fiscal stance over time and to offset past excesses with current austerity.16 This response also is less than was observed in the historical data.
  • There is evidence of an error-correction effect in the data. The coefficients on the lagged ratios have the correct signs, and that associated with ŷjT−1 is significantly different from zero. The coefficient −0.44 indicates that the projection is designed to make up 44 percent of any deviation of fiscal ratio from the country’s “nor mal” ratio within a single year. This adjustment is also roughly half of the adjustment observed in the historical data.

The results from estimating the projection model for the current account ratio are reported in the second set of columns in Table 3a:

  • There is no significant evidence of an autore-gressive structure in ΔĉjT, just as was true in the historical analysis.
  • Past shocks to the fiscal ratio have a significant lagged effect on the current account ratio, a feature unobserved in the actual data.
  • There is a significant error-correction effect as evidenced by the coefficient on ΔĉjT−1. The coefficient −0.33 indicates that the projection is constructed to make up about ⅓ of any deviation of the current account ratio from its nor mal value within a single year. The coefficient on ŷjT−1 is insignificantly different from zero. These features are quite similar to those ob served in the historical data.

When the envisaged data are examined with the cointegrating relationship imposed, the evidence is once again stronger for the fiscal ratio. In that regression (reported in the third set of columns), the cointegrating variable (emjT−1) has explanatory power nearly equal to the lagged ĉjt−1 and ŷjt−1 re-ported in the first set of columns. In the equation for the current account ratio, the results are much weaker.

For time horizon T+1 (Table 3b), the projection “model” is quite similar to that of horizon T. The contemporaneous and lagged “level” effects are almost identical for the fiscal ratio, as is the lagged “level” effect for the current account ratio. The autoregressive terms differ somewhat, but the differences are not statistically significant. The similarity of error-correction effects is quite striking, as it suggests that the projected adjustment from imbalance occurs totally in horizon T—there is no further adjustment in horizon T+1. This is quite different from the historical record, where adjustment continues in fairly equal increments from horizon T to horizon T+1.

Table 3bRegression Results, Envisaged Current and Fiscal Account Ratios, Horizon T+1
ΔŷjT+1ΔĉjT+1ΔŷjT+1ΔĉjT+1
CoefficientS.ECoefficientS.E.CoefficientS.E.CoefficientS.E.
ΔĉjT+10.158*(0.08)0.179**(0.08)
ΔŷjT−1-0.175*(0.10)0.020(0.16)-0.165*(0.09)0.104(0.16)
ΔĉjT−10.049(0.08)-0.208*(0.12)0.034(0.07)-0.364***(0.12)
ŷjT −1-0.462***(0.11)-0.029(0.18)
ĉjT−10.048(0.09)-0.370***(0.14)
emjT−1-0.474***(0.11)-0.128(0.19)
N129129129129
R20.860.720.860.68
Notes: Variable definitions (this table only for all variablesgj):

Full sample, Horizon T+1. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.*** Indicates significance at the 99 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Variable definitions (this table only for all variablesgj):

Full sample, Horizon T+1. Standard errors (S.E.) appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.*** Indicates significance at the 99 percent confidence level.A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

Divergence Between Projected and Actual Policy

We note from the preceding discussion that there is substantial evidence of difference between the coefficients in Tables 2a and 3a, and between Tables 2b and 3b. We interpret these differences as evidence that the “model” used in IMF projections and the “model” generating the historical data are significantly different. However, as the earlier discussion demonstrated, model differences are only one source of projection errors. In this subsection, we use the framework of equation (3e) to decompose the observed projection error for horizon T into components.

As the earlier discussion indicated, the projection error can conceptually be decomposed into four parts: differences in models, differences in policy response, mismeasurement of initial conditions at time of projection, and random errors.17 Projection error is measured directly as the projection of the variable for horizon T minus the realization of the variable. Errors in initial conditions are measured as the difference between projected and historical observations of the level of the variable in period T−1. Two policy variables are considered indicators of the importance of policy reform conditions in the error: the difference between projected and historical depreciation of the real exchange rate (ΔêjT − ΔejT), and the difference between projected and historical change in government consumption expenditures as a share of GDP (ΔŵjT− ΔwjT).18 We hypothesize that the former should have a significant effect on the current account, while the latter should be a significant component of the fiscal surplus.

Estimation of (3e) using the error-correction framework presented in equations (8) is complicated by the simultaneity of the macroeconomic balances and the policy variables over which conditions are defined. As (3e) indicates, (ΔêjT−ΔejT), (ΔŵjT −ΔwjT), ΔejT and ΔwjT will all be included as regressors in the estimation framework, but all of these are potentially simultaneously determined with the macro balances. We address this by estimating the equations with both ordinary least squares (OLS) and two-stage least squares (2SLS), with the 2SLS results presumed to be free of simultaneity bias.19 For each equation, as implied by (6), year-specific dummy variables are included to control for year-to-year differences in capital availability on world markets; we also include significant country-specific dummy variables to control for abnormally large cross-country differences in macro balances. Those results are reported in Table 4. The top panel reports the results of regressions in the cur rent account ratio and the fiscal ratio. There are two columns: the first with OLS estimates, on a slightly larger sample, and the second with 2SLS estimates on a consistent-size sample of 162 observations across all variables. The bottom panel reports the regressions that served as the “first stage” of the 2SLS. The first column reports OLS results over the largest sample for which data were available for that regression, while the second column reports OLS results over the consistent 2SLS sample of 162 observations.

Table 4Estimation of Projection Error Equations
ΔĉjT−ΔcjTΔŷjT−ΔyT
OLS2SLSOLS2SLS
cjT−10.010.04cjT−1-0.11**-0.10**
ĉjT−1cjT−1-0.41**-0.31**ĉjT−1cjT−1-0.06-0.07
yjT−1-0.10-0.02yjT−10.010.04
ŷjT−1yjT−1-0.19-0.25*ŷjT−1yjT−1-0.34**-0.30**
ΔejT-0.0003-0.002ΔejT0.01**0.01
ΔêjT −ΔejT-0.005-0.001ΔêjT−ΔejT0.0070.009
ΔwjT0.10**0.02
ΔŵjT−ΔwjT-0.47**-0.47**
ΔcjT−10.030.03
ΔyjT−1-0.08*-0.08
N172162167162
R20.590.590.740.71
ΔêjT−ΔejTΔŵjT−ΔwjT
OLSOLSOLSOLS
ΔêjT−1 −ΔejT−10.14**-0.03ŵjT−1wjT−1-0.02-0.03
ΔejT−1-0.03**-0.05**wjT−10.15**0.14**
ΔwjT−1-0.06-0.06
ŵjT−1 − ΔwjT−1-0.28**-0.23**
N166162166162
R20.680.680.520.53
ΔeTjTΔwT
OLSOLSOLSOLS
ΔejT−10.01**0.03**wjT−1-0.39**-0.38**
cjT−1-0.100.04ΔwjT−1-0.13**-0.16**
yjT−10.370.28yjT−10.27**0.29**
N174162173162
R20.650.740.610.60
Notes: The two-stage-least-squares (2SLS) procedure used the estimating equations in the two lower panels for ΔeT, ΔêT−ΔeT, ΔwT, and ΔŵT−ΔwT, and estimated those equations simultaneously with the two reported in the upper panel. The equations in the two lower panels are all ordinary least squares (OLS), since they did not include endogenous regressors. The coefficients differ because of the number of observations included: those with 165 were estimated in the simultaneous equation system, while those with other numbers of observations were estimated as single equations.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.Standard errors and other regression statistics are available from the authors on request.
Notes: The two-stage-least-squares (2SLS) procedure used the estimating equations in the two lower panels for ΔeT, ΔêT−ΔeT, ΔwT, and ΔŵT−ΔwT, and estimated those equations simultaneously with the two reported in the upper panel. The equations in the two lower panels are all ordinary least squares (OLS), since they did not include endogenous regressors. The coefficients differ because of the number of observations included: those with 165 were estimated in the simultaneous equation system, while those with other numbers of observations were estimated as single equations.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.Standard errors and other regression statistics are available from the authors on request.

We interpret the results as follows. Take as example the coefficient on cjT−1 in the two regressions. Given our derivation in equation (3e), this coefficient should represent the difference between the projection coefficient and the actual coefficient. When we compare the results of Tables 2 and 3, we find this to be the case. Consider the 2SLS results. In the fiscal ratio regression of Table 4, the coefficient of −0.11 is quite similar to the difference (0.08−0.16) of the coefficients reported in Tables 2 and 3. For the current account ratio, the coefficient of 0.04 is also very similar to the difference (−0.33−(−0.40)) of the coefficients reported in Tables 2a and 3a. A positive coefficient in this regression indicates that the projection incorporated a more positive response to that variable than was found in the actual data.

We separate the discussion into the various types of errors.

Differences in modeling. If the projections used a different model from that evident in the actual data, we expect to find significant coefficients on the variables cjT −1, yjT−1, ΔcjT −1, ΔyjT−1, ΔejT, and ΔwjT in the top panel. Our discussion of Tables 2a and 3a indicated that we anticipated greater evidence of differing models in the fiscal projections than in the current account projections. This point is partially supported by results reported in Table 4. Consider the OLS results. In the fiscal ratio estimation, there are significant coefficients on cjT−1 (−0.11), ΔejT (0.01), ΔyjT−1 (−0.08), and ΔwjT (0.10). If we consider the last case for illustration: a positive ΔwjT should reduce the fiscal balance. The coefficient (0.10) indicates that the IMF projections incorporated less pass through of increased government ex penditures into the reduced fiscal ratio than was actually observed, leaving a positive projection error. However, the 2SLS results suggest that differences in modeling are less apparent than is suggested by the OLS estimates, since only the coefficient on cjT−1 (−0.10) remains significantly different from zero.

For the current account ratio, there is no significant evidence of differences in modeling. All coefficients on these variables are both small and insignificantly different from zero.

Mismeasurement of initial conditions. Another source of projection error will be the difference between the initial conditions known to IMF staff and the actual initial conditions available in historical data. For these differences to be a significant source of projection error, the coefficients on the variables (ĉjT−1cjT−1) and (ŵjT−1yjT−1) must be significantly different from zero.

In the fiscal ratio regression, the difference in initial fiscal ratios (ŷjT−1yjT−1) is a significant contributor to projection error. The 2SLS coefficient (−0.30) indicates that when IMF staff had access to artificially high estimates of the previous period’s fis cal ratio, they adjusted downward the projected necessary policy adjustment necessary. This response was a rational one, given the error-correction nature of the fiscal ratio, but was based on an incorrect starting point.

In the current account ratio regressions, the differences in initial conditions are the only significant determinants of projection error. With coefficients (−0.31) for (cjT−1cjT−1) and (−0.25) of (ŷjT−1yjT−1) in the 2SLS version, the regressions suggest that the projections were in error largely because of incomplete information about the true value of the cur rent account ratio in the preceding period.

Differences in policy response. If the projections included a policy response at variance with that actually observed, the coefficients on (ΔŵjT− ΔwjT) and (ΔêjT − ΔejT) will be significant in the two regressions. In both the 2SLS and OLS results, there is little evidence of this. In the fiscal 2SLS regression, there is a significant coefficient (−0.47) on (ΔŵjT − ΔwjT). This indicates that when the IMF projected smaller expenditure increases than actually occurred, the projection error on the fiscal ratio was, on average, positive—as expected.

The regressions in the bottom panel hold some clues as to why the projections differed from historical values. As is evident in the (ΔŵjT − ΔwjT) regression, previous forecast errors were significant determinants of this policy projection error, as was a bias toward more positive projections as the previous period’s fiscal ratio rose. The policy projection errors in the real exchange rate depreciation (ΔêjT−ΔejT) had no significant contribution to either regression in either specification.

Random errors. As the R2 statistics indicate for the two regressions, the preceding three sources of projection error explain only 59 percent (for the current account ratio) and 71 percent (for the fiscal ratio) of total projection error. The remainder should be considered random shocks.

An Empirical Decomposition of Projection Error

In previous subsections, we identified several potential sources of projection errors. The magnitude and significance of the regression coefficients reported in Table 4 shed some light on the relative importance of each of the sources. To investigate this issue in more detail and to get a better insight into the relative contributions of each of the sources to the resulting projection errors, we implement the following exercise. Setting variables used in the 2SLS regressions of Table 4 to their mean values and using estimated coefficients, we compute the contribution of each of the model variables to the projection errors for current account and fiscal balance ratios, (ΔĉjT − ΔcjT) and (ΔŷjT − ΔyjT), respectively. Using means of projection errors as anchors, we can draw some conclusions about the relative contributions of differences in modeling, differences in initial conditions, and differences in policy response to the projection errors. Tables 5a and 5b summarize the results of the described experiment for current account and fiscal balance projection errors, respectively.

Table 5aForecast Error Components: Current Account Ratios
VariableCoeff.MeanEffectPercent EffectTotal Percent Effect by Type
cjT−10.04-6.64-0.2525.46
yjT−1-0.02-4.390.08-8.23Differences in modeling: 16.63 percent
ΔejT0.00−3.140.01−0.60
(ĉjT−1cjT−1)-0.311.26-0.4040.38Mismeasurement of initial conditions: 44.55 percent
(ŷjT−1yjT −1)−0.250.16−0.044.17
êjT −ΔejT)0.008.46−0.010.85Differences in policy response: 0.85 percent
t930.970.100.10-10.43
t940.370.150.06-5.78
t951.140.140.16-16.54
t961.990.110.22-22.60
t971.840.110.20-20.83Year-specific: −125.53 percent
t983.350.100.35-35.93
t991.310.110.15-14.90
t000.500.100.05-5.37
t01-1.210.06-0.076.85
Country dummiesCountry-specific: 163.50 percent
ΔcjT −ΔcjT−0.98
Total:−0.98100100 percent
Number of observations162
Table 5bForecast Error Components: Fiscal Balance Ratios
VariableCoeff.MeanEffectPercent EffectTotal Percent Effect by Type
cjT−1-0.10-6.640.68265.67
ΔcjT−10.03-0.57-0.01-5.63
yjT−10.04-4.39-0.19-74.68Differences in modeling: 166.17 percent
ΔyjT−1-0.080.10-0.01-3.03
ΔejT0.01-3.14-0.04-14.43
ΔwjT0.02−0.280.00−1.73
(ĉjT −1cjT−1)-0.071.26-0.09-33.98Mismeasurement of initial conditions: −52.78 percent
(ŷjT−1yjT −1)-0.300.16-0.05-18.80
ΔêjT − ΔejT0.018.460.0728.22Differences in policy response: 15.99 percent
ΔŵjT − ΔwjT-0.470.07-0.03-12.24
t93-0.120.10-0.01-4.77
t940.420.150.0625.34
t951.190.140.1766.65
t96-0.110.11-0.01-4.96Year-specific variables: 182.08 percent
t970.080.110.013.38
t980.350.100.0414.53
t991.300.110.1456.64
t000.420.100.0417.40
t010.360.060.027.87
Country dummies-1.360.01-0.02-6.60Country-specific variables: −211.46 percent
ΔŷjT −ΔyT0.25
Total:0.25100100 percent
Number of observations:162

In the case of current account ratios, the most significant source of projection error comes from the measurement of the initial conditions. This component is responsible for 44.55 percent of the total projection error, while differences in modeling and differences in policy response generate forecast errors with magnitudes of only 16.63 percent and 0.85 percent, respectively. The positive signs of percentage contributions of all three sources suggest that these sources of errors tend to bias the current account mean projection error toward negative values.

However, when the components of the forecasting error for fiscal-balance ratios are considered, the two major sources of the errors are differences in modeling (166.17 percent) and mismeasurement of initial conditions (−52.78 percent). It appears that the model used in projections tends to make the projection error more positive while the measurement error in the initial conditions pulls the projection error in the negative direction, as occurred in the case of the current account projection errors. Differences in policy response are responsible for approximately 16 percent of the total mean projection error.

Projection errors of both variables are greatly influenced by the year and country-specific factors captured by the corresponding dummy variables.

It is evident, in examining the data, that there is substantial mismeasurement in the fiscal and current account ratios when the initial values in the two databases are compared. Simple statistics for the actual and projection ratios are as follows (based on 175 observations):

HorizonsMeanStandard DeviationMinimumMaximum
cjT−1-6.247.89-39.8211.08
ĉjT−1-5.095.99-39.9210.41
ŷjT−1-4.364.11-20.485.61
ŷjT−1-4.224.42-22.604.00

The difference in mean between historical and pro jected data for the current account ratio is quite striking. The value of ĉjT−1 should be known (i.e., historical) at the time of the projection. Differences of this magnitude are an indication that there has been substantial revision in the macroeconomic aggregates over time.20 The difference in mean for the fiscal ratio is not so pronounced. The standard deviations are large, and these differ substantially between actual and projection databases. There is more variability in the actual current account ratios than in those projected; by contrast, there is more variability in the projected fiscal ratios than there is in the actual ratios.

Figures 3 and 4 present the scatter-plots of actual and projected ratios. The 45 degree line represents those combinations for which projected coincides with actual. As the figures show, there is tremendous measurement error even in these initial conditions. There is also a strong positive correlation of projection with actual: for (cT −1, ĉT−1) it is 0.84, while for (yT−1, ŷT−1) it is 0.86. There is not the perfect match that would exist in theory, but the match is quite strong.

Figure 3Initial Conditions for Fiscal Account Ratio

Figure 4Initial Conditions for Current Account Ratio

Examining the Role of Revisions

New information is made available to IMF staff on a continuous basis throughout the duration of the IMF program. The staff periodically revisits its initial projections in the context of a program review, and updates them to reflect the information more recently received. We should then observe that the IMF projections converge to the actual performance as revisions are made over the duration of a multiperiod IMF program: imprecision in initial conditions will be eliminated, projected policy reform can be revised in light of observed behavior, and inaccuracy in the forecasting model can be reduced. Moreover, one can expect that for multiperiod programs, the IMF staff’s major efforts in the design of the original programs would be concentrated on the improvement of short-horizon projections while less emphasis is placed on long-horizon projections since initial projections can be fine-tuned in the context of later reviews.

Assuming that the new information is efficiently incorporated, we expect to observe that the IMF projections converge to the historical performance as revisions are considered. Therefore, any assessment of the quality of the IMF projections will be incomplete without examining the evolution of projections. We address this issue by comparing the projections of the original programs (OPs) with those reported in the first reviews (FRs), which take place during the first program year.21 Some basic qualitative information on the evolution of the out come projections can be illustrated by Figures 5a and 5b, where we compare historical and envisaged mean changes in the fiscal and current account ratios. These plots are based on the data summarized in Table 6.22 An obvious observation is that for the vast majority of projection horizons, the first-review projections, Δy^jkFR and Δc^jkFR, are closer to the actual outcomes, Δ ŷjk and Δĉjk, than the original program projections, Δy^jkOP and Δc^jkOP. The only exception is the change in the fiscal ratio for the horizon T.

Figure 5aMeans of Projected and Historical Changes in Fiscal Ratios

Figure 5bMeans of Projected and Historical Changes in Current Account Ratios

Table 6Projecting the Change in Macroeconomic Aggregates(Original Program (OP) Versus First Review (FR))
VariableNMeanSth DevSumMinimumMaximum
Horizon T
Δc^jTFR1200.186243.6748222.34863-11.2596114.09314
Δc^jTOP120-0.263923.25906-31.66988-13.892369.14219
ΔcjT1200.641134.3366176.93582-16.6370814.29500
Δy^jTFR1201.212502.68248145.50000-5.500009.40000
Δy^jTOP1200.927002.82961111.24000-7.6000011.00000
Δyjt1200.974912.95418116.98875-6.2702712.72751
Horizon T+1
Δc^jT+1FR95-0.745835.81160-70.85403-26.1749411.90127
Δc^jT+1OP95-0.813425.31906-77.27518-22.2318711.90477
ΔcjT+195-0.399087.35763-37.91262-35.6117617.19095
Δy^jT+1FR951.271893.35196120.83000-5.6000013.30000
Δy^jT+1OP951.402533.18396133.24000-5.6000012.90000
ΔyjT+1951.058133.75009100.52207-6.7670413.69233
Horizon T+2
Δc^jT+2FR74-0.311515.18034-23.05211-17.3163511.10499
Δc^jT+2OP74-0.225704.91495-16.70172-22.0428010.81110
ΔcjT+274-0.374568.12439-27.71773-38.1474321.78397
Δy^jT+2FR741.987843.49077147.10000-4.7000013.50000
Δy^jT+2OP742.278383.15904168.60000-3.3000013.20000
ΔyjT+2741.510004.07910111.73996-14.8807011.88877
Horizon T+3
Δc^jT+3FR500.244625.1613712.23123-14.7373712.91443
Δc^jT+3OP500.249604.8925412.48019-19.895947.35647
ΔcjT+350-2.0773914.24848-103.86963-81.5693221.40533
Δy^jT+3FR502.964004.02060148.20000-3.9000013.10000
Δy^jT+3OP503.115803.52157155.79000-1.9000013.00000
ΔyjT+3501.486444.2149774.32213-12.3386211.64537
Notes: N denotes number of observations; Std Dev denotes standard deviation.
Notes: N denotes number of observations; Std Dev denotes standard deviation.

Assessment of only mean changes might be misleading, since, as we showed in the previous sections, there is a great deal of variability in envisaged and historical data. Some additional insights on the evolution of the projections can be obtained by examining developments in correlations between envisaged and actual changes. Table 7 reports those correlations for fiscal and current account ratios for both first reviews and original programs. As we found in the regressions of the previous section, en visaged changes in fiscal ratios exhibit higher correlation with the actual changes than do comparable changes in envisaged and actual current account ratios. This observation is true for projections drawn from the original program as well as those from the first reviews. Inaccuracy of the current account projections seems to worsen significantly with the length of the projection horizon. Also, there is a strong pattern showing that the projection performance of the first reviews, measured by the correlation coefficient, improves relative to the projections of the original programs for all variables and all projection horizons. The gain in forecasting power is particularly noticeable over short horizons—and decreases as the length of the projection horizon extends.

Table 7Correlations Between Projected and Actual Outcomes for Changes in Macroeconomic Aggregates(Original Program (OP) Versus First Review (FR))
HorizonsTT+1T+2T+3
Fiscal ratio (Δyjk, Δy^jkFR)0.696350.761570.706240.60535
Fiscal ratio (Δyjk, Δy^jkOP)0.607420.690370.657610.57322
Correlation improvement (ρyFRρyOP)0.088930.07120.048630.03213
Current account ratio (Δcjk, Δc^jkFR)0.691750.463450.339550.35714
Current account ratio (Δcjk, Δc^jkOP)0.503900.341930.307470.35449
Correlation improvement (ρcFRρcOP)0.187850.121520.032080.00265

Bias, Efficiency, and Accuracy of Revisions

Musso and Phillips (2001) suggested an interesting approach to evaluating projections. They analyze projections on the basis of three major characteris tics: bias, efficiency, and accuracy. In this paper, we follow their approach and document some of the facts along these three dimensions to compare the relative performance of the projections of the original programs and their first revisions.

Bias. By bias, we refer to the divergence of the distribution of projection errors from the zero-mean normal distribution. Table 8 presents statistics characterizing the distribution of (ΔĉT −ΔcT) and (Δŷt −ΔyT) for the original programs as well as for their first revisions.23 Several observations can be made from the information presented in Table 8:

  • The null hypothesis of the true mean of the distribution being zero is rejected more frequently for the original program projection errors than for those from the first reviews. It is especially noticeable for the fiscal balance ratios.
  • Standard deviations are considerably smaller for the first-review projection errors than those for the original programs. The difference is greater for short horizons and becomes very small or even reverses for longer horizons.
  • For the horizon T, positive skew of the distribution of the projection errors for both variables suggests that projection errors are more likely to be far above the mean than they are to be far below the mean. This result can be observed for both groups of projections. However, for longer horizons, the skew tends to be negative, reflecting the opposite trend.
  • For both variables and for most of the horizon lengths, the distribution of errors has more mass in the tails than a Gaussian distribution with the same variance. The only exceptions are projection error distributions for horizon T changes in the current account (FR), and horizon T+1 changes in the fiscal balance (both OP and FR). • For the OP projection errors, most of the tests find statistically significant evidence that the distributions exhibit lack of normality. The only exception is the T+1 horizon for fiscal balances. For the FR projection errors, the results are mixed. Some of the goodness-of-fit tests for nor mal distribution cannot reject the null hypothesis of normality.24
Table 8Program Projection Errors
HorizonTT+1T+2T+3
Projection Errors in Changes of Fiscal Balance Ratios to GDP
MeanOP0.048-0.492**-1.334*-1.809*
FR-0.238-0.252-0.543-1.478*
MedianOP-0.014-0.492-0.786-1.541
FR-0.232-0.341-0.548-1.728
Standard deviationOP2.5652.8524.0703.673
FR2.2112.4712.9523.663
SkewnessOP0.028-0.009-2.406-1.582
FR0.8270.312-1.731-1.591
KurtosisOP4.321.63610.1004.747
FR6.1421.0579.8126.467
Normality testOPRejectedMixed (3/4)RejectedRejected
FRRejectedMixed (3/4)Mixed (1/4)Mixed (2/4)
Projection Errors in Changes of Current Account Ratios to GDP
MeanOP0.905*0.258-0.595-2.176
FR0.4550.262-0.222-2.220
MedianOP0.5830.669-0.614-0.666
FR0.2810.410-0.228-0.994
Standard deviationOP3.8977.2607.76612.492
FR3.2046.8247.83212.726
SkewnessOP1.222-2.630-2.555-4.612
FR0.241-3.405-3.072-4.702
KurtosisOP5.44217.96117.03229.405
FR1.82524.07918.61829.359
Normality testOPRejectedRejectedRejectedRejected
FRRejectedMixed (1/4)RejectedRejected
Notes: * Significantly different from zero at the 90 percent confidence level (based on student’s t test). ** Significantly different from zero at the 95 percent confidence level (based on student’s t test). Mixed (X/4): X out of four tests cannot reject normality of the error terms at the 95 percent confidence level. OP denotes original program; FR denotes first review.
Notes: * Significantly different from zero at the 90 percent confidence level (based on student’s t test). ** Significantly different from zero at the 95 percent confidence level (based on student’s t test). Mixed (X/4): X out of four tests cannot reject normality of the error terms at the 95 percent confidence level. OP denotes original program; FR denotes first review.

Efficiency. We test the efficiency of the FR and OP projections by regressing the value of the historical change on a constant term and the value of projected change as illustrated in equations (8a) and (8b) for macroeconomic variable gT, with vT and uT as random errors. We perform the estimation for changes in variables as well as for the levels.

This type of efficiency test is referred to as the weak criterion since it uses a limited information set (Musso and Phillips, 2001). We would conclude that the projection was an efficient estimate of the historical datum if the intercept were insignificantly different from zero and the slope were insignificantly different from unity. Tables 9 and 10 report results of the estimation in changes and in levels, respectively.

Table 9Test of “Weak” Efficiency(in changes)
Original ProgramFirst Review
Coeff.S.E.t–statisticCoeff.S.E.t–statistic
Fiscal balance ratios
Horizon: T
Intercept (H0: Intercept=0)0.387*0.2271.7090.0450.2140.210
Slope (H0: Slope =1)0.634***0.076-4.7920.767***0.073-3.192
R20.3690.485
Horizon: T+1
Intercept (H0: Intercept=0)-0.0840.296-0.284-0.0560.265-0.211
Slope (H0: Slope =1)0.731***0.082-3.2810.848**0.075-2.027
R20.4390.577
Horizon: T+2
Intercept (H0: Intercept=0)-0.1680.498-0.337-0.1820.379-0.480
Slope (H0: Slope =1)0.542***0.114-4.0180.814**0.096-1.938
R20.2110.492
Horizon: T+3
Intercept (H0: Intercept=0)-0.8040.613-1.312-0.3950.598-0.661
Slope (H0: Slope =1)0.704**0.12-2.4670.635***0.12-3.042
R20.3770.366
Current account ratios
Horizon: T
Intercept (H0: Intercept=0)0.818**0.3452.3710.489*0.2881.698
Slope (H0: Slope =1)0.671***0.106-3.1040.816**0.079-2.329
R20.2540.479
Horizon: T+1
Intercept (H0: Intercept=0)-0.0960.671-0.143-0.0060.650-0.009
Slope (H0: Slope =1)0.467***0.130-4.1000.583***0.113-3.690
R20.1140.215
Horizon: T+2
Intercept (H0: Intercept=0)-0.5680.803-0.707-0.3650.850-0.429
Slope (H0: Slope =1)0.507***0.169-2.9170.550***0.169-2.663
R20.0970.120
Horizon: T+3
Intercept (H0: Intercept=0)-2.1831.649-1.324-2.2211.734-1.281
Slope (H0: Slope =1)1.0160.3530.0451.0040.3500.011
R20.1270.134
Notes: * The null hypothesis (H0) can be rejected at the 90 percent confidence level. ** The null hypothesis can be rejected at the 95 percent confidence level. *** The null hypothesis can be rejected at the 99 percent confidence level. Coeff. denotes coefficient; S.E. denotes standard error.
Notes: * The null hypothesis (H0) can be rejected at the 90 percent confidence level. ** The null hypothesis can be rejected at the 95 percent confidence level. *** The null hypothesis can be rejected at the 99 percent confidence level. Coeff. denotes coefficient; S.E. denotes standard error.
Table 10Test of “Weak” Efficiency(in levels)
Original ProgramFirst Review
Coeff.S.E.t–statisticCoeff.S.E.t–statistic
Fiscal balance ratios
Year: T
Intercept (H0: Intercept=0)-1.870***0.345-5.420-1.483***0.297-4.993
Slope (H0: Slope =1)0.493***0.076-6.6710.685***0.071-4.437
R20.2630.437
Current account ratios
Year: T
Intercept (H0: Intercept=0)-1.529**0.694-2.203-1.121**0.529-2.119
Slope (H0: Slope =1)0.706***0.101-2.9110.9120.081-1.086
R20.2890.512
Notes: ** The null hypothesis (H0) can be rejected at the 95 percent confidence level. *** The null hypothesis can be rejected at the 99 percent confidence level. Coeff. denotes coefficient; S.E. denotes standard error.
Notes: ** The null hypothesis (H0) can be rejected at the 95 percent confidence level. *** The null hypothesis can be rejected at the 99 percent confidence level. Coeff. denotes coefficient; S.E. denotes standard error.

There is a striking relationship between historical and projected changes found in the data: in each case, for the original program (except horizon T+3 changes in current account ratios), the hypothesis of weak efficiency is strongly rejected by the data. The rejection is in all cases based on an estimate of c1 or d1 that is significantly less than unity. When the FR results are examined, weak efficiency is once again rejected. However, when compared with the OP results, the coefficient estimates of c1 and d1 are closer to the hypothesized value of unity.25

Accuracy. We test relative accuracy of the OP and FR projections by comparing them with a random-walk benchmark projection. That is, we investigate whether the IMF projections of the year-T values of the variables do better than if the projections were simply set equal to the T–1 values. We draw our conclusions from Theil’s U statistic and report results in Table 11.26 Larger values of the U statistic indicate a poor projection performance. The bench mark random-walk projections for OP are based on the T–1 value of the variable as it is documented in OP, while the benchmark random-walk projection for FR uses the initial conditions from the revised data of FR.

Table 11Test of Accuracy(in levels)
Projection ModelTheil’s U Statistic
Number of

Observations
Fiscal

balance ratios
Current

account ratios
Original program1210.6950.696
Benchmark for OP (random walk)1210.7880.639
First review1200.5710.568
Benchmark for FR (random walk)1200.7600.635
Notes: OP denotes original program; FR denotes first review.
Notes: OP denotes original program; FR denotes first review.

For the fiscal balance, both OP and FR projec tions perform better than the random walk. How ever, only the FR projection outperforms the ran dom walk for the current account; the OP projection for this variable is slightly worse than that of its random-walk counterpart. Overall, the FR projections exhibit lower values of the U statis tic, reflecting their more accurate projections.27

An Empirical Decomposition

The preceding results suggest that the IMF staff modifies its projections to incorporate new informa tion, and that the revised projections have better forecasting power when compared with the projec tions of the original programs. It is possible to de compose the difference in the OP and FR projec tions using a methodology similar to that of equation (3e) and Table 4. The details of this analy sis are reported in Annexes I–III. The salient findings for our purpose are as follows:

  • There is a substantial difference in initial condi tions used in the two projections, and these dif ferences contribute significantly to the im provement of FR over OP.
  • There is also evidence that the model used in the FR projections differs significantly from that used in the OP projections.

There is evidently “learning by doing” in these projections at the modeling stage as well as at the stage of data collection.

Conclusions

Envisaged and historical observations on the fiscal and current account ratios in countries participating in 175 IMF programs between 1993 and 2001 devi ated strongly from one another. Our statistical analysis suggests that the causes can be separated into four components.

First, the IMF staff was apparently working with quite different information about the initial condi tions of the program countries than is currently ac cepted as historical. This difference leads to substan tial divergence even if the IMF staff used the model revealed by the historical data. This result is consis tent with the conclusions of Orphanides (2001); and Callan, Ghysels, and Swanson (2002) on the making of U.S. monetary policy.

Second, the IMF staff did appear to have a differ ent model in mind when making its projections. Its model was characterized by gradual fiscal account adjustment, both in response to contemporaneous current account shocks and to long-run imbalances, while the model revealed by historical data was characterized by more rapid adjustment to both types of imbalances. Further, its envisaged response was concentrated in horizon T, while the historical response to shocks was roughly equally proportioned across horizons T and T+1.

Third, there is a difference between projected and historical implementation of policy adjustment. Given the level of aggregation of the policy variables investigated (total government consumption expenditures, real exchange rate depreciation) we cannot conclude that the difference is due to a failure to meet the conditions of the program; the dif ferences could also be due to shocks that worsened the performance of these aggregates even when con ditions were fulfilled. This is a question that can, and should, be investigated further.

Fourth, there is ample evidence that, like other macroeconomic projections, IMF projections are quite inaccurate. The evidence on “accuracy” reported here is instructive—while the projections outperform a random walk most of the time, they are not much better. The Meese and Rogoff (1983) results remind us of the difficulty in projecting ex change rates in time series. The project described here indicates the inaccuracy of simple models in a panel (i.e., time-series and cross-section of countries) format.

Our results on revisions indicate that the IMF staff learns from past projection errors—and from new information. However, even that learning leaves large gaps to fill. The largest margin for im provement may well be in “just-in-time” data col lection, so that the errors owing to incomplete in formation, especially from initial conditions, can be eliminated.

Annex I. Creating the Error−Correction Residuals

In the following tables (AI.1AI.3), we use the IMF’s World Economic Outlook (WEO) dataset covering those programs with time horizon T. There are 175 observations in general, although somewhat more when considered in levels.

Creating the error correction residual ejt

Dependent variable: yjt (WEO)

Table AI.1Analysis of Variance
SourceDFSum of SquaresMean SquareF ValuePr > F
Model865,518.1456664.164486.58<.0001
Error96935.617059.74601
Uncorrected total1826,453.76271
Root MSE3.12186R–square0.8550
Dependent mean-4.33059Adjusted R–square0.7252
Coefficient of variance-72.08859
Parameter Estimates
VariableDFParameter EstimateStandard Errort ValuePr > ǀtǀ
Cjt10.099960.062031.610.1103
t931-7.417511.72365-4.30<.0001
t941-4.838511.91288-2.530.0131
t951-6.315861.84898-3.420.0009
t961-5.374861.92894-2.790.0064
t971-3.980821.88383-2.110.0372
t981-3.636221.95216-1.860.0656
t991-4.645331.95383-2.380.0194
t001-5.266441.97374-2.670.0090
t011-5.929371.83106-3.240.0017
Notes: DF denotes degrees of freedom; MSE denotes mean square error.
Notes: DF denotes degrees of freedom; MSE denotes mean square error.

This is the formulation used to create the error correction variable (eT−1= yt – predicted value) for WEO data. A complete set of country dummies was used as well, but is suppressed here.

The following regression results report the coefficients used in creating the error-correction variable for envisaged data:

Table AI.2Analysis of Variance

Dependent variable: yjt (envisaged)

SourceDFSum of SquaresMean SquareF ValuePr > F
Model956,449.9418767.894125.29<0.0001
Error971,244.2662312.82749
Uncorrected total1927,694.20810
Root MSE3.58155R-square0.8383
Dependent mean-4.47401Adj. R-square0.6799
Coefficient of variation-80.05230
Notes: DF denotes degrees of freedom; MSE denotes mean square error.
Notes: DF denotes degrees of freedom; MSE denotes mean square error.
Table AI.3Parameter Estimates
VariableDFParameter EstimateStandard Errort ValuePr > ǀtǀ
Cjt10.316640.078614.030.0001
t931-6.843321.81926-3.760.0003
t941-4.688061.95701-2.400.0185
t951-5.698611.90593-2.990.0035
t961-3.856021.93132-2.000.0487
t971-3.342521.93759-1.730.0877
t981-2.741181.85499-1.480.1427
t991-4.387182.07410-2.120.0370
t001-3.959662.08914-1.900.0610
t011-5.053672.00406-2.520.0133
Notes: DF denotes degrees of freedom. A complete set of country dummies was used as well, but it has been suppressed here for brevity.
Notes: DF denotes degrees of freedom. A complete set of country dummies was used as well, but it has been suppressed here for brevity.
Annex II. Does the Timing of Approval of IMF-Supported Programs Matter to These Results?

Projection errors, especially for the initial program year (year T), may reasonably be hypothesized to de pend on the point in time during the year when a program was approved. We investigated this hy pothesis in two ways. First, we calculated Pearson correlations of the approval month with the size of the projection error for horizons T and T+1 (Table AII.1). Second, we regressed the projection error on dummy variables indicating the quarter of year T in which approval occurred (Table AII.2).

Table AII.1Pearson Correlations for Projection Errors
Fiscal Balance:

Original Program

(approval month in T)
Fiscal Balance:

First Review

(approval month in T)
Current Account:

Original Program

(approval month in T)
Current Account:

First Review

(approval month in T)
Horizon T0.01905-0.026770.006870.06711
0.83640.77160.94060.4665
120120120120
Horizon T+1-0.05439-0.157500.142890.14916
0.58530.12340.14990.1406
1039710399
Notes: Each cell in this table includes three statistics: the top entry is the Pearson correlation coefficient; the middle entry is Prob > ǀ r ǀ under the null hypothesis of zero correlation; and the bottom entry is the is the number of observations.
Notes: Each cell in this table includes three statistics: the top entry is the Pearson correlation coefficient; the middle entry is Prob > ǀ r ǀ under the null hypothesis of zero correlation; and the bottom entry is the is the number of observations.
Table AII.2Regressions on Quarterly Dummies (Horizon T) for Projection Errors
Fiscal balance (OP)Fiscal balance (FR)Current account (OP)Current account (FR)
Quarter 10.06-0.110.44-0.15
(0.46)(0.40)(0.69)(0.57)
Quarter 2-0.27-0.471.08*0.94*
(0.39)(0.34)(0.60)(0.49)
Quarter 30.26-0.161.76**0.26
(0.51)(0.44)(0.77)(0.63)
Quarter 40.44-0.040.130.63
(0.59)(0.51)(0.90)(0.74)
R20.010.020.070.04
Number of observations120120120120
Notes: OP denotes original program; FR denotes first review. Standard errors appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.
Notes: OP denotes original program; FR denotes first review. Standard errors appear in parentheses.* Indicates significance at the 90 percent confidence level.** Indicates significance at the 95 percent confidence level.

The Pearson correlations provide no evidence of a significant approval-time effect in either variable. For the fiscal ratio, there is no evidence of a significant approval-time effect for either OP or FR pro jection errors. For the current account ratio, a num ber of coefficients are positive and significant. However, they do not grow uniformly over the sample; the largest deviations from the mean occur for programs approved in the second and third quarters of “year T.”

We did the same exercise for the deviation in ini tial conditions (Table AII.3); in that case, the hy pothesis is that programs approved later in year T will have more accurate information on the initial conditions, so that deviations will be lessened. There is no evidence of a significant effect in the Pearson correlations. There is some evidence of this in the regression results, however (Table AII.4). For both OP and FR versions of the fiscal ratio and the OP version of the current account ratio, the devia tion in initial conditions is significantly larger, on average, for programs approved in the first quarter of year T than for those approved later in year T. There is thus a downward bias in the fiscal ratios used as initial conditions in projections created in the first quarter of year T relative to the historical data, most likely because the IMF staff did not have access to the later revisions when creating its projections.

Table AII.3Pearson Correlations for Discrepancies in Initial Conditions(Actual – Projection)
Fiscal Balance:

Original Program

(approval month in T)
Fiscal Balance:

First Review

(approval month in T)
Current Account:

Original Program

(approval month in T)
Current Account:

First Review

(approval month in T)
All horizons0.137440.106280.09385-0.11611
0.13280.24400.30760.2028
121122120122
Notes: Each cell in this table includes three statistics: the top entry is the Pearson correlation coefficient; the middle entry is Prob > |r| under the null hypothesis of zero correlation; and the bottom entry is the is the number of observations.
Notes: Each cell in this table includes three statistics: the top entry is the Pearson correlation coefficient; the middle entry is Prob > |r| under the null hypothesis of zero correlation; and the bottom entry is the is the number of observations.
Table AII.4Regressions on Quarterly Dummies for Discrepancies in Initial Conditions
Fiscal Balance (OP)Fiscal Balance (FR)Current Account (OP)Current Account (FR)
Quarter 1-0.80**-0.73**-1.75**1.56
(0.39)(0.35)(0.73)(1.11)
Quarter 2-0.37-0.40-0.95-0.82
(0.34)(0.31)(0.63)(0.96)
Quarter 30.270.05-0.750.62
(0.44)(0.39)(0.82)(1.23)
Quarter 4-0.18-0.21-0.34-2.17
(0.51)(0.46)(0.95)(1.44)
R20.050.050.070.04
Number of observations120120120120
Notes: OP denotes original program; FR denotes first review. Standard errors appear in parentheses.** Indicates significance at the 95 percent confidence level.
Notes: OP denotes original program; FR denotes first review. Standard errors appear in parentheses.** Indicates significance at the 95 percent confidence level.

If there is a value to this information, it should also be evident in the initial conditions as reported in FR relative to OP for each program. In Table AII.5, we compare the initial conditions, with deviations mea sured as FR values minus OP values. A similar regres sion on approval times within year T yields little evi dence of a systematic bias, with only the current account ratio showing any deviation of significance. The estimated coefficients are suggestive, though, rising from negative values for Q1 approval to ever-increasing values for subsequent quarters.

Table AII.5Regressions on Quarterly Dummies (Horizon T) for Differences in Initial Conditions Between First Review (FR) and Original Program (OP)
Fiscal Balance (FR – OP)Current Account (FR – OP)
Quarter 1-0.072-0.176
(0.171)(0.367)
Quarter 20.0230.074
(0.148)(0.317)
Quarter 30.2150.323
(0.190)(0.408)
Quarter 40.0320.904*
(0.222)(0.477)
R20.0130.039
Number of observations120120
Notes: * Indicates significance at the 90 percent confidence level. Standard errors appear in parentheses.
Notes: * Indicates significance at the 90 percent confidence level. Standard errors appear in parentheses.
Annex III. What New Information Do Revisions Incorporate?

The results in the text suggest that IMF staff modifies its projections to incorporate new information and that the revised projections have better forecasting power relative to the original program. However, it is not yet clear whether this is a reflection of adjusting projections for new values of the initial conditions that contain less measurement error, or a sign of using new information to modify the entire scope of the model used in projection. We choose to address this issue by estimating regressions of the following general form:

The form is the same as that advanced in the previous section. The difference in projections can be stated in somewhat different form in equation (A3c). When we subtract (A3a) from (A3b), we note four different reasons why the two projections will not be the same: an updating of information on past events (within the first set of square brackets in (A3c)), in creased information on policy implementation, a change in the “model” used in projection (within the second set of square brackets in (A3c)), and projection errors.

Here, we regress projected changes in the macroeconomic variable as projected in the first review of the program (Δg^jTFR) on the projected change of the variable as it was originally planned at the outset of the program (Δg^jTOP) and on the terms reflecting improvement of the information on the initial conditions (Δg^jT1FRΔg^jT1OP) and (Δg^jT1FRg^jT1OP). We also in corporate a term representing differences in projected changes in the policy variable, (Δs^jTFRs^jTOP), to cap ture changes in the implementation of conditions as sociated with the programs. Finally, all the terms within the second set of square brackets are included to study whether the forecasting model has changed.

We predict that the value of the coefficient on Δg^jTOP will be unity, as would be the case, for example, if the first review simply caused a mean−preserving contraction in the distribution of random errors. Values of ã 1 and ã2 differing significantly from zero will indicate that the revision observed in FR reflects the improved information about the initial condi tions governing the economic success of the program. Figure AIII.1 illustrates the interpretation of this model. With ã1 and ã2 significantly different from zero and the coefficient on Δg^jTOP being unity, the revision should trigger the “old model, new ini tial conditions” scenario pictured there. However, if the new information available during implementa tion of the program called for correction of the entire projection model, the coefficients on Δg^jT1OP,g^jT1OP, and Δs^jTOP would be significantly different from zero and the estimates would follow the “new model, new initial conditions” scenario in Figure AIII.1.

Figure A.III.1Incorporation of New Information in Projections

Table AIII.1 summarizes the results of the model estimation for the ratio of fiscal balances to GDP for all programs in the sample at horizon T. Changes in fiscal ratios as they are projected in the first reviews of the programs are regressed not only on terms rep resenting the error-correction structure of fiscal ra tios but also on the similar terms corresponding to the current account ratios. A complete set of time and country dummy variables was also included in the regressions. The following insights can be ob tained from the first column of Table AIII.1:

  • The value of the coefficient on Δy^jTOP is 0.986, which is not statistically different from unity. This could be interpreted as if the correction of the projection reported in the first review of the program is just a modification of the projection owing to the more accurate initial conditions. The updated information set available at the moment of revision is incorporated into the same projection model that was used to create OP projections. This result is consistent with the fact that none of the terms included to cap ture projection model modification is significantly different from zero.
  • The coefficient on (Δy^jT1FRΔy^jT1OP) is negative and significant. One of the potential explanations of this fact can be outlined as follows. Sup pose that reduction of the measurement error results in an improvement in the fiscal balance in the years preceding the program relative to what it had been originally thought to be when the program was designed. That would mean that (Δy^jT1FRΔy^jT1OP) is a positive number. Given our finding, this would result in a reduction of the projected change in the fiscal ratio projected in the first review of the program. Moreover, the value of the coefficient, −0.973, is not sig nificantly different from −1, which suggests that this is a one-to-one relationship. This finding makes intuitive economic sense, because if the government’s budget deficit is not as bad as was originally thought, the required correction of the fiscal balance is also less demanding.
  • Specification testing reveals that changes in lagged first-difference terms with lag length greater than 1 do not contribute significantly to the regression. At the same time, none of the current account ratio terms is significantly dif ferent from 0, which suggests that improvement in the data quality of the current account has lit tle effect on the projections of the fiscal ratios.
  • The coefficient on the difference in the policy variable, (Δs^jTFRΔs^jTOP), is negative and significant, implying that differences in policy between OP and FR are also responsible for the amendments of the original projections. More over, the negative sign of this coefficient sug gests that a greater observed real depreciation results in less positive forecasts of changes in fis cal balance ratios.28
  • Finally, testing jointly that both lagged level terms are not significantly different from zero allows us to conclude that revisions to initial conditions do not contribute systematically to the changes observed in FR relative to OP.
Table AIII.1Regression Results, Fiscal Account Ratios(first review versus original program)
Δy^jTFRΔy^jTFR
CoefficientStandard errorCoefficientStandard error
Δy^jTOP0.986***0.1410.970***0.130
(Δc^jTFRΔc^jTOP)0.0400.1370.0470.124
(Δy^jT1FRΔy^jT1OP)-0.973***0.357-0.756***0.271
(Δc^jT1FRΔc^jT1OP)0.3140.2370.2100.224
(Δy^jT2FRΔy^jT2OP)-0.3500.406-0.2030.316
(Δc^jT2FRΔc^jT2OP)-0.2460.221-0.2060.192
(y^jT1FRy^jT1OP)0.2110.553
(c^jT1FRc^jT1OP)-0.3080.209
Δy^jT1OP0.0750.0870.0350.076
Δy^jT1OP-0.0430.133-0.0660.114
e^jTOP-0.0090.008-0.012*0.007
(Δe^jTFRΔe^jTOP)-0.045***0.016-0.041***0.016
Number of observations9191
R20.9880.986
Adjusted R20.9260.925
Notes: Full sample, Horizon T.* Significantly different from zero at the 90 percent confidence level.*** Significantly different from zero at the 99 percent confidence level. A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Full sample, Horizon T.* Significantly different from zero at the 90 percent confidence level.*** Significantly different from zero at the 99 percent confidence level. A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

The second column in Table AIII.2 reports results of the estimation of a similar model when the lagged level terms are excluded from the regression.

  • The coefficient on Δy^jTOP is still insignificantly different from unity and the hypothesis that the IMF staff does not modify the projection model as the new information arrives is strongly sup ported by the data.
  • At the same time, the coefficient on Δe^jTOP is significantly different from zero at the 90 per cent confidence level, providing some support for the hypothesis that the scope of the projecting model was amended.
  • The coefficient on (Δy^jT1FRΔy^jT1OP) is still negative, although much smaller in absolute value.
  • The policy variable coefficient is still significantly different from zero.
Table AIII.2Regression Results, Current Account Ratios(first review versus original program)
Δc^jTFRΔc^jTFR
CoefficientStandard errorCoefficientStandard error
Δc^jTOP0.411*,+++0.2340.439***,+++0.207
(Δy^jT1FRΔy^jT1OP)-0.1941.037-0.2110.666
(Δc^jT1FRΔc^jT1OP)-0.2970.691-0.1710.547
(Δy^jT2FRΔy^jT2OP)-1.0990.939-1.279*0.728
(Δc^jT2FRΔc^jT2OP)0.0370.4740.0840.428
(y^jT1FRy^jT1OP)0.1851.250
(c^jT1FRc^jT1OP)0.1780.586
Δc^jT1OP-0.0850.201-0.0660.180
c^jT1OP-0.366***0.142-0.382***0.126
e^jTOP0.0130.0120.0130.011
(Δe^jTFRΔe^jTOP)-0.0180.043-0.0250.036
Number of observations9191
R20.9390.938
Adjusted R20.6510.688
Notes: Full sample, Horizon T.* Significantly different from zero at the 90 percent confidence level.*** Significantly different from zero at the 99 percent confidence level.

Significantly different from unity at the 99 percent confidence level.

A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.
Notes: Full sample, Horizon T.* Significantly different from zero at the 90 percent confidence level.*** Significantly different from zero at the 99 percent confidence level.

Significantly different from unity at the 99 percent confidence level.

A complete set of time and country dummies was included in the regressions, but their coefficients have been suppressed for brevity.

Similarly, Table AIII.2 presents outcomes of the model estimation for the current account ratios for all programs in the sample at horizon T. Once again, we regress changes in current account ratios from the first reviews of the programs on terms representing the error-correction structure of current account ratios and on the similar terms corresponding to the fiscal ratios, as well as on the policy variable and the set of time and country dummies. The first column of the table represents the case in which the error-correction terms are included in the regression:

  • The value of the coefficient on the originally projected change in the current account, Δc^jTOP, is 0.411 and the null hypothesis of the true value of this coefficient being unity is rejected at the 99 percent confidence level. Unlike our result for the case of fiscal ratio projections, the projected change in the current account ratio in the revision of the program appears to be de rived under a different model relative to the change in the current account projected at the beginning of the program.
  • Modification of the projection model is also strongly supported by the fact that the coefficient on Δc^jT1OP is significant at the 99 percent confidence level.

Excluding the lagged level terms from the regression gives us a slightly better understanding of the rela tionship between the considered variables.

  • The coefficient on the originally projected change in the current account, Δc^jTOP, is still significantly different from unity at the 99 percent confidence level and takes the value of 0.439. Thus, we still find strong support for distin guishing between the original program and first-review projection models.
  • However, one of the terms representing changes in the initial conditions for fiscal ratio, (Δy^jT2FRΔy^jT2OP), is significantly different from zero at the 90 percent confidence level, with a coefficient value of −1.279. This suggests that the projection of the current account ratio is sig nificantly affected by the changes in the initial conditions of the fiscal balance ratios. Moreover, the sign of the estimated coefficients indicates that an improvement in the initial conditions of the fiscal balance relative to what it was originally assumed to be when the program was de signed induces a reduction in the projected change in the current account for some given values of the other variables. This result is sup ported by our previous finding that the coefficient on (Δy^jT1FRΔy^jT1OP) in the regression of fis cal balances reported in Table AIII.1 is negative. To illustrate this, suppose that reduction of the measurement error results in the improvement of the fiscal balance initial conditions relative to what it had been originally thought to be when the program was designed, implying that (Δy^jT1FRΔy^jT1OP) is positive. Since the coefficient on this term is negative, this would result in the reduction of the projected change in the fiscal ratio projected in the first review of the program, Δy^jTFR. Then the macro identity written in the first−difference form, Δy^jTFR=Δc^jTFRΔp^jTFR, suggests that for any given value of private saving, Δp^jTFR, the projected change in the current account, Δc^jTFR, also reduces. This decrease in the current account ratio as a result of improvement in the initial conditions for fiscal ratios is captured in our model by the negative sign of the coefficient on the corresponding terms.

Our analysis shows that the correction in the initial conditions has a strong influence on the magni tude of the projections for both fiscal and current account ratios. Therefore, it appears to be logical to look at the magnitude of those corrections and their distribution. Figures AIII.2 and AIII.3 illustrate the distribution of the corrections in the levels of fiscal balance ratio to GDP and the distribution of the corrections in the levels of current account ratio to GDP, respectively, for the year T−1. These correc tions are large, varying between −5.3 percent and 4.6 percent of GDP for fiscal ratios and between −9.3 percent and 8.1 percent of GDP for current account ratios. The mass of the distributions is concentrated around 0. The negative skew in both cases shows that the corrections of the initial condi tions are more likely to be far below the mean than they are to be far above the mean. Also, both distri butions have kurtosis that exceeds 3, which implies that they have more mass in the tails than a Gaussian distribution with the same variance. Table AIII.3 reports results of the goodness-of-fit tests for the normal distribution. All of the tests strongly reject the null hypothesis of the initial con dition corrections having a Gaussian distribution.

Figure A.III.2Distribution of Differences in T−1 Levels Between FR and OP for Fiscal Ratios

Notes: FR denotes first review; OP denotes original program.

Figure A.III.3Distribution of Differences in T–1 Levels Between FR and OP for Current Account Ratio

Notes: FR denotes first review; OP denotes original program.

Table AIII.3Goodness-of-Fit Tests for Normal Distribution for Corrections in Initial Conditions
VariableTestStatisticP Value (H0: normal)
(yjt1FRyjt1OP)Kolmogorov-Smirnov0.3179< 0.010
Cramer-von Mises3.9281< 0.005
Anderson-Darling18.6995< 0.005
chi-square17,877.8796< 0.001
(cjt1FRcjt1OP)Kolmogorov-Smirnov0.24224< 0.010
Cramer-von Mises2.73701< 0.005
Anderson-Darling13.70785< 0.005
chi-square9,843.90054< 0.001
Note: H0 denotes the null hypothesis.
Note: H0 denotes the null hypothesis.
References

    CallanMylesEricGhysels and Norman R.Swanson2002“Monetary Policy Rules with Model and Data Uncertainty,”Southern Economic JournalVol. 69 pp. 23965.

    • Search Google Scholar
    • Export Citation

    ClementsMichael P. and DavidHendry1995“Forecasting in Cointegrated Systems,”Journal of Applied EconometricsVol. 10 pp. 12746.

    • Search Google Scholar
    • Export Citation

    HamiltonJames1994Time Series Analysis (Princeton, New Jersey: Princeton University Press).

    HendryDavid1997“The Econometrics of Macroeconomic Forecasting,”Economic JournalVol. 107 pp. 133057.

    HowreyE.P.1984“Data Revision, Reconstruction and Prediction: An Application to Inventory Investment,”Review of Economics and StatisticsVol. 66No. 3 pp. 38693.

    • Search Google Scholar
    • Export Citation

    MeeseRichard and KennethRogoff1983“Empirical Exchange Rate Models of the Seventies: Do They Fit Out of Sample?”Journal of International EconomicsVol. 14 pp. 324.

    • Search Google Scholar
    • Export Citation

    MussoAlberto and StevenPhillips2001“Comparing Projections and Outcomes of IMF-Supported Programs,”IMF Working Paper 01/45 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation

    OrphanidesAthanasios2001“Monetary Policy Rules, Macroeconomic Stability and Inflation: A View from the Trenches,”Working Paper (Washington: Board of Governors of the Federal Reserve System).

    • Search Google Scholar
    • Export Citation
1A version of this paper was presented at the Conference on the Role of the World Bank and the IMF in the Global Economy, held at Yale University during April 25–27, 2003. An abbreviated version is to be published by Routledge in a volume of conference proceedings.
2We will hold to a specific definition of “projections” in this paper. We do not consider projections to be identical to “forecasts.” We define a forecast to be the best prediction possible of what is to occur at a given time in the future. A projection in this context is a prediction based on the participating country under taking and completing all structural and policy reforms agreed to in the letter of intent approved between the participating government and the IMF. The two could diverge if the best prediction includes only partial implementation of policy and structural reform.
3When an IMF program is approved, the IMF staff uses the best statistics available at that time for current and past macroeconomic data to create projections for the evolution of those variables over the following years. These projections represent the “original program” projections for that IMF program. Program performance is reviewed periodically over time, and at each review the IMF staff creates a new set of projections for the macroeconomic data reflecting the best available information of that time. We will use the “first review” projections for each program in a later section.
4For example, the projections are reported on an annual basis but the year is not invariably a calendar year. For some programs, the fiscal year was used as the basis of data collection and for projections. In those instances, the historical data are converted into fiscal-year equivalents through weighted-average conversion of the calendar-year data.
5The “year T” of each program is defined by IMF staff to be that fiscal year (as defined by the country) in which the program is approved. Programs are typically not approved at the beginning of year T, but rather at some point within the year.
6In this section, we use the projections from the “original program.”
7The data on which Figures 1 and 2 are based are reported in Table 1.
8Table 1 reports the standard deviations of the projected and actual changes in the database; these are in all cases and for all projection horizons at least as large as the mean values.
9By contrast, we consider the forecast of ΔgT to be defined as ΔgeT = f(χt−1; ΔseT), with ΔseT representing the forecaster’s best prediction as of period T−1 of the policy vector to be observed in period T.
10Hendry (1997) provides an excellent summary of the possible sources of projection (in his case forecasting) error when the projection model is potentially different from the actual model. This example can be thought of as a special case of his formulation.
11gT−2 enters the expression through the term ΔgT−1.
12We will refer to the “error-correction form” as one that includes both lagged differences and lagged levels of the two variables as explanatory variables for the current differenced variables. This can be derived from a general AR specification of the two variables; the AR(2) specification is used here for ease of illustration. The form presented in the text can be derived from the following AR(2) set of equations:
Specification tests are used to choose the lag length appropriate to the empirical work. In a world in which yjt and cjt are non-stationary but are cointegrated on a country-by-country basis, further simplification is possible. If yjt and cjt are nonstationary in the current dataset, equation (6) represents a cointegrating relationship. The “error-correction” variable ejt can then be inserted in the equations (7) in place of the terms in yjt−1 and cjt−1 and will have the coefficient associated with yjt−1 in (7). It is impossible to verify a nonstationary relationship in this dataset, given that we have only scattered observations from each country’s time series. We do investigate that possibility in the second and fourth columns of Tables 2 and 3, with support for that interpretation of the error-correction term in the Δyjt equation. Hamilton (1994, Ch. 19) provides a clear derivation of this error-correction form from the underlying autoregression.
13Statistical confidence in this paper will be measured at the 90 percent, 95 percent, and 99 percent levels. In the text, statistical significance will indicate a degree of confidence greater than 95 percent unless otherwise indicated.
14This assumption will be justified, for example, if the participating country is constrained in its international borrowing, so that the ratio of current account surplus to GDP is set by foreign lenders.
15While imposition of the cointegration condition through the error-correction variable is effective for the fiscal ratio, our comparison of projections with historical data will be based on the system without this condition imposed. As Clements and Hendry (1995) demonstrate, the imposition of the cointegration condition in estimation when cointegration exists improves forecast accuracy, most notably for small (i.e., N= 50) samples. For larger samples, the improvements in forecast accuracy are small.
16We would observe this negative coefficient, for example, if we had a model that required the government to balance its bud get over each two-year period. There could be excess spending in odd years, but it would be offset by spending cuts in even years.
17For now we treat each program as if it were approved at the beginning of year T, so that the projected effects of the program on macroeconomic adjustment have a full year to take hold. In fact, programs are approved at different times within year T. Thus, the timing of approval within the year may explain part of the projection error. We explore this in Annex II.
18The variable for government consumption expenditures is available in consistent format in both historical and envisaged data. The variable on real depreciation is constructed in both cases as nominal depreciation minus CPI inflation for the horizon in question. These variables are explicit in the historical data. In the envisaged data, the nominal exchange rate is derived as the ratio between GDP in home currency and GDP in U.S. dollars.
19Both sets of results are reported because the systems approach to estimation reduces the number of observations usable in estima tion. The OLS results thus provide a more comprehensive analysis, although one potentially tainted by simultaneity bias.
20We have been careful in constructing the dataset, but we must admit as well the possibility that the definition of current account used in the historical data may differ in some instances from the definition used in the envisaged data. While we see no reason for this difference to be systematic, it may well represent some of the observed difference in mean values.
21Actual timing and number of reviews vary from program to program. In general, Stand-By Arrangements and extended arrangements (SBAs and EFFs) have more frequent reviews than arrangements under the structural adjustment facilities (SAF, ESAF, and PRGF). Further, the completion of reviews is often held up by difficulties in complying with conditionality. For these reasons, we plan in future research to address the relation ship between review timing and projection error.
22Figures 5a and 5b include similar information to that of Figures 1a and 1b. They differ, however, in the number of observations used in creating the mean values. For example, Figure 1a uses 175 observations for horizon T to calculate the mean historical and envisaged change in the fiscal ratio, while Figure 5a uses 120 observations for which both original program and first review observations of fiscal ratio are available.
23Projection errors are calculated as the differences between envisaged values and actual realizations.
24We used Kolmogorov-Smirnov, Cramer-von Mises, Anderson-Darling, and chi-square tests to check normality. The results are reported in Table AIII.3.
25There is a difficulty in this type of estimation that is not addressed by Musso and Phillips (2001). Since the right-hand-side variable is only an estimate of the true OP projection, the regression may be characterized by error-in-variables. This will cause the slope coefficient to be biased downward. We investigated this possibility using an instrumental-variable technique. The resulting slope coefficients were, in most cases, closer to unity and insignificantly different from unity for the fiscal ratio, thus exhibiting weak efficiency. They were farther from unity for the current account ratio, thus sustaining the conclusion of inefficiency for that variable.
26The Theil’s U Statistic:
27The pattern of errors in OP and FR projections are similar to those observed by Howrey (1984) in his study of inventory investment. He found in that case that there was evidence of substantial revision to initial data in inventory investment over the period 1954–80. He also found, however, that knowledge of the revision reduced only marginally the variance of projection error.
28Although it would be more reasonable to use total government expenditure as a policy variable in the regression for fiscal balances, the number of observations available for the first reviews limits the use of this variable as a proxy for a policy variable.

    Other Resources Citing This Publication