howard white provides a critique of this approach
Results-based management has become a fact of life for development agencies. They might hope to learn from the experience of the US Agency for International Development (USAID) which has already gone down this road. It is indeed instructive that USAID has come back up the road again saying 'there’s nothing down there'. But development agencies have been rightly criticised in the past for paying too little attention to the final impact of their activities, so we would like to support a results-based approach. But we should not do so blindly when it suffers from the severe limitations outlined below. Serious attempts to link agency performance to developmental outcomes must rely upon the log-frame. The log-frame is not a universal panacea but, used properly, can force agencies into a critical examination of the nature of their programmes and projects, and the results they achieve.
what is it?
Results-based management (RBM) is the creation of an incentive structure, including the allocation of budgetary resources, reflecting achievements as measured by outcomes. It may seem that all incentives should operate in this way but, in fact, management information systems have often focused on inputs and activities. Hence, for example, educational resources have traditionally been allocated based on student numbers. The philosophy of results-based management has led to changes in countries such as the Netherlands and UK whereby a part of funding for tertiary education is now allocated according to research performance. In the UK, league tables are published of school exam results, implying that resources will follow pupils to the better schools. More recently, the possibility has been mooted of mortality rate tables for hospitals and even individual doctors.
Monitoring and evaluation (M&E) for development activities has shared this input-orientation, with the success of a project being judged by the speed at which it gets through its budget. Even end-of-project evaluations have often shied away from the tricky issue of development impact. During the 1990s, governments in developed countries embraced RBM, an embrace which took in aid agencies.
results-based management in aid agencies
The longest track record with RBM amongst aid agencies is that of USAID, following the 1993 Government Performance and Results Act (GPRA) which required all government agencies to have outcome-based targets against which their performance could be measured. The Agency’s experience is particularly instructive. In 1997, USAID laid out six strategic development goals, such as 'broad-based economic growth and agricultural development encouraged', and for each of these defined a set of outcome indicators at both country and global levels (for example 'average annual growth rates in real per capita income above 1%'). With respect to the growth goal, the FY 2000 performance report states that 'nearly 70% of USAID-assisted countries were growing at positive rates in the second half of the 1990s, compared with 45% in the early part of the decade'. However, that same performance report noted that 'one cannot reasonably attribute overall country progress to USAID programmes.'
Commenting on the USAID performance report from 1999, the General Accounting Office (GAO) had similarly observed that the goals were 'so broad and progress affected by many factors other than USAID programmes, [that] the indicators cannot realistically serve as measures of the agency’s specific efforts'. In response to these criticisms, the FY 2000 performance report announced that the indicators related to the strategic goals will no longer be used to measure USAID’s performance (but they will be reported as being of interest in their own right, being referred to as 'Development Performance Benchmarks'). Rather performance will be measured against the strategic objectives of the individual operating units (such as country programmes).
On the other hand, the British Department for International Development (DFID) is just starting to get to grips with these issues. All UK agencies are required to have a Public Service Agreement (PSA) and Service Delivery Agreement (SDA), the former containing outcome targets against which an agency’s performance is to be judged and its funding determined. DFID’s PSA measures are based on International Development Targets (IDTs), albeit with a much shorter time frame. For health, the performance targets are stated as improvements in child, maternal and reproductive health in the 'top ten' recipients of DFID health care assistance demonstrated by
- a decrease in the average under-5 mortality rate from 132 per 1,000 live births in 1997 to 103 on the basis of data available in 2004;
- an increase in the proportion of births assisted by skilled attendants from a baseline established in 2000 of 43% to 50% on the basis of data available in 2004; and
- improved access to reproductive health care.
Given the widespread commitment to the IDTs, which is particularly strong in the UK, are such targets likely to lead to improved agency performance? This question is considered by looking at the desirable properties of performance indicators.
desirable properties of performance measures
Performance measures have a number of desirable properties. Most of these can be dispensed with fairly quickly to focus on three key problems:
relevance and balance
Performance measures such as those embodied in the IDTs are undoubtedly relevant. There are, however, two problems of balance. First is that the measures may not span an agency’s portfolio. For example, the health targets in DFID’s PSA exclude health activities outside of the top ten recipients. And the targets as a whole do not apply to activities in middle-income countries. The second problem of balance is that, for reasons discussed below, it is better to have a set of indicators spanning the range of the log-frame from inputs to outcomes.
measures known, understood and trusted
Performance measures used in development are usually clearly defined and well understood, at least by those who work directly with them. There has been some tension between top-down setting of targets and a more participatory approach. This has also been a factor behind the promotion of the Millennium Development Goals (MDGs) as rivals to the IDTs. Data quality can be more problematic. The IDTs include maternal mortality, for which data are notoriously unreliable. The measure was included in DFID’s first PSA, but when it came to reporting DFID told the Treasury that data were unavailable. Country coverage for most indicators is patchy. But the more serious problem is the timeliness of the data, discussed below.
affected and attributable
Changes in performance measures should be affected by the activities of the organisation and the extent of the effect measurable (attributable). However, attribution becomes harder to establish as we move through the log-frame from inputs to outcomes. As we saw above, USAID has abandoned claiming that changes in economic outcomes can be attributed to its work.
achievable
Targets should be achievable, but not too easily. Most analyses suggest that the IDTs will not be met, although there is regional variation in performance. But this is not the same as saying that they could not be met if aid and government resources were better focused toward meeting them.
linked to existing management systems
Organisations will already have in place management information systems. The rise of results-based measures was in part a response to the fact that existing systems focused on inputs and internal activities, such as spending and staffing, rather than achieving outcomes. But new results-oriented systems should not be separate or parallel to these existing systems. There are two further problems here: aggregation (can the various measures be added up to summarise performance for regions, sectors and the agency as a whole?) and alignment (do the data tell us about impact on the outcomes of interest?)
three key problems
The above discussion identifies three problem areas. These problems are less severe (though present) when using RBM at the project level, becoming more intense as we attempt to measure results at country level or for the whole portfolio.
data availability
Data become available only with a time lag, this being particularly so for developmental outcome measures like infant mortality and income-poverty. DFID’s PSA covering 2001-2004 includes amongst its targets 'a decrease in the average under-5 mortality rate from 132 per 1,000 live births in 1997 to 103 on the basis of data available in 2004'. Implicit here is the recognition that data available in 2004 will probably be for 2001, namely the start of the period of the current PSA. How can results of current strategies be judged on the basis of results achieved before the strategy was put in place?
attribution
It is very difficult to attribute changes in developmental outcomes to the activities of an individual agency. This problem is encountered where agencies work together, as in sector programmes or providing budget support, or for agency performance at the country level. One agency’s support to public sector reform in an African country (providing the funds to pay for retrenchment packages) had as the outcome indicator: maternal mortality. It is difficult to imagine what sensible use could have been made of maternal mortality data (even with no time lag) to judge the performance of this programme.
integration with existing management information systems
All donor agencies have some sort of monitoring and evaluation system at the project and programme level which should provide a basis for both feedback at the project level and 'feed-up' to management. But how do these systems tackle the problems of aggregation and alignment?
The World Bank’s rating system is an example of a system yielding agency-wide results. These ratings include an overall rating if an activity has been satisfactory. Hence overall portfolio performance can be judged by the percentage of projects deemed satisfactory and 'problem areas' identified. But this information does not allow us to say anything about the contribution of the World Bank to achieving development outcomes as measured by, say, literacy or premature death. The information collected is simply not the right sort to provide that information.
There are two possible responses to this problem of misalignment. The first is to say that it is inevitable. The problem of attribution is not going to be solved for routine monitoring purposes, so no attempt should be made to link agency performance as measured by bottom up systems with agency performance as measured by achieving developmental results. That answer does not seem satisfactory for agencies, like DFID, which have pinned their performance to such results.
The second response to the misalignment problem is to resort to the log-frame. The bridge must be made between observing satisfactory activities and presumed impact on development outcomes.
the case for the log-frame
The US GRPA suggested that performance measures should span inputs, outputs and outcomes. However, in practice such a logical framework (log-frame) based approach has been applied in few GRPA plans. The absence of indicators reflects the fact that there is too little conscious analysis as to how plans and programmes will affect outcomes. The attraction of the log-frame is partly that it should force the agency to examine programmes to see if it really will achieve the desired outcomes. Used in this way, the log-frame helps tackle the attribution problem.
Attribution becomes harder as we move along the causal chain. It is easy to attribute responsibility for delivering inputs, and usually for carrying out activities, although external factors may play a part. These activities should lead to desired outputs which deliver the target outcomes, again subject to external factors. If the underlying model is correct then indicators should capture if the organisation is doing what it needs to do to achieve the outcomes - which may well be the case even if the targets are not met. The various links in the log-frame should, in principle, be scientifically-established, though in practice we rely upon 'plausible association'.
Howard White is a Fellow at the Institute of Development Studies, University of Sussex, UK.
