paul engel, charlotte carlsson and arin van zee propose three perspectives on learning from evaluation
Development agencies are consistently required to improve their performance, in terms not only of project outcomes, but also of the quality of their programming and their institutional capacity. In a very practical sense, evaluations are now perceived as learning opportunities. It only seems logical, therefore, to try and improve the internalisation of evaluation results at different levels. But how can this be done? What dilemmas do we face and what have we learned so far?
Evaluation has always been about learning, about how to be accountable, how to be transparent, how to learn from experience. The issue today, therefore, is not whether it is desirable to learn. Instead, the question is: who should learn, why should they learn and how should they set about it? Quality issues also come into play if learning is to be widely shared.
internalising evidence from evaluations: three perspectives on learning
perspective 1: development policy and programming
Probably the most straightforward way of enhancing learning is to look at its internalisation in development policy and programming. However, many of the existing feedback mechanisms are still mainly one-directional, drawing on the logic of information dissemination to selected target groups rather than communication around evidence as a reiterative learning process.
The 2001 workshop Evaluation feedback for effective learning and accountability, organised by the OECD-DAC Working Party on Evaluation now known as the OECD-DAC Network on Aid Evaluation, reviewed some of the current experiences. Most reports have called for the establishment of what may be called policy innovation networks, mobilising Southern partners and stakeholders to engage in existing learning and feedback routes. The International Fund for Agricultural Development recognises the need to 'shift the fulcrum of evaluation feedback to the South' to encourage more direct links between feedback of findings and the planning and monitoring of country programmes. Many development agencies emphasise results-based planning and management as a way of improving the practical use of evaluation results. The European Commission's Fiche Contradictoire and the UK Department for International Development's (DFID) Public Service Agreements are examples of this. The review regards the communication of lessons through the mass media as increasingly important. Finally, the review calls on development agencies to do more to share their experiences with each other.
perspective 2: within organisations and among partners
A second approach focuses on organisational learning, recognising that development processes result from actions and interactions by a set of diverse stakeholders. Active participation, capacity-building and learning by all these actors is a fundamental rather than an instrumental condition. The locus for change is the facilitation of collective rather than individual learning. As a result, policy-makers and/or donors become one among many, rather than the only intended learner.
An organisational learning approach to evaluation not only fundamentally changes the way social actors relate to each other, it also requires a radical shift in the role of the evaluator. All actors, including the evaluator, have to recognise they are part of a joint learning effort. In such an 'epistemic community', the evaluator becomes a facilitator in a joint inquiry rather than an expert wielding an 'objective' measuring stick. Yet as Rebecca Sutton argues in her 1999 Overview of the policy process, such communities run the risk of 'clique-building', reducing the diversity of opinion if the discourse is captured by the most vocal actors in the group. Critical self-reflection must be maintained in order to build in 'reality checks' and thus avoid too narrow a discourse among closed circles.
perspective 3: society at large
A third perspective focuses on a type of learning that leads to change in society at large. When the sharing and interpretation of evidence extend beyond those directly involved in the evaluation process, conflicts of interest are common and consensus becomes the exception rather than the rule. The question then is whether and how interested parties can exert pressure for change and whose interpretation of the findings is the dominant one. The co-existence of multiple truths requires a more transparent analysis of findings and the creation of 'sense-making fora' for stakeholders to interpret and validate the evidence. Some commentators, including Norman Uphoff and Jerry Combs in their paper Some things can't be true but are: rice, rickets and what else? Unlearning conventional wisdoms to remove paradigm blockages, stress that such broadening of the interpretation of evidence to a wider audience and different interest groups can help to avoid 'paradigm traps' among scientists and policy-makers that limit their views on development options.
The perspective on societal learning from evaluations, i.e. learning that goes beyond a small number of directly involved stakeholders, could have far-reaching consequences for our thinking on development cooperation. Traditional evaluation as we know it may gradually fade into the background, to be replaced by multiple forms of evidence-gathering and sharing among diverse groups of stakeholders, the adaptive management of resources and multiple communication and negotiation processes at various levels. There will be a greater need for good governance to create enabling conditions for such processes and for conflict resolution between stakeholders. As a result, governments and donors will become crucial players who may either enable or prevent society from learning. As both development policy-makers and field practitioners alike have urged, this perspective links evaluation with governance issues.
dilemmas at the evaluation-learning interface
Various dilemmas arise in trying to turn the current evaluation practice into a learning process.
why strengthen the learning function? what other purposes should evaluation serve?
Evaluation serves a number of different purposes. A growing emphasis on learning from evaluation means a shift in intentions. Traditionally, control has been an important purpose. From this perspective, the aim of evaluation is to enhance transparency and accountability, particularly from the donors' point of view. Another vital purpose is assessment, i.e. deciding whether the agreed objectives have in fact been achieved. With learning in order to improve performance now becoming an increasingly important purpose, evaluations could eventually become geared towards adaptive management. Institutional learning and the development of the institutions responsible for managing development would be prerequisites for this. So would pursuing one purpose imply a trade-off with another? Might these aims be mutually exclusive in an evaluation context? Or would a shift in intentions reflect a growing complexity of evaluation functions rather than a shift from one to the other?
what do we mean by learning?
Learning is a buzz-word, often used but not often clearly defined. As a result, one of the challenges defined by the 2001 workshop Evaluation feedback for effective learning and accountability is the need to unpack the learning concept. After all, there are all sorts of different approaches. According to Ruud van der Veen in Learning natural resource management, different types of learning in adult education are linked to distinct cognitive and motivational processes and fulfil complementary roles. In the their chapter in the book Wheelbarrows full of frogs, Christine King and Janice Jiggins utilize organisational learning theory to distinguish between single-, double- and triple-loop learning, reflecting the degree to which underlying rules, values, norms and behaviour are truly affected. In the same book, Niels Röling focuses on cognition, the process by which an organism deals with changes in context. He stipulates two fundamental drivers for social learning: the coherence sought among values/emotions/perceptions on the one hand and theory/interpretations/actions on the other; and the need for correspondence between the above elements and the prevailing context.
whose learning are we talking about?
At the three levels of learning outlined above, the development policy and programming focus mainly emphasises learning by donors and policy-makers. Hence, there is a risk that the participation of local actors is simply instrumental, i.e. more intent upon extractive actor consultation or 'rubber stamping' policies than effective participation in decision-making. The organisational learning approach recognises the complementary role of stakeholders and a multi-layered institutionalisation of findings. It emphasises the collective understanding of relevant development issues and finding the way forward. Finally, the societal learning approach recognises the existence of potentially insurmountable differences in perspective among interested parties, yet at the same time acknowledges the need to negotiate sustainable answers to common challenges. In line with its focus on adaptive management and aspects of governance, it focuses on institutional development for improving societal learning and decision-making.
widespread learning versus the quality of the learning process
In their 2001 book Participation: the new tyranny, Bill Cooke and Uma Kothari argue that unless evaluation instruments and evidence-gathering tools go hand in hand with an outspoken commitment to decentralising decision-making, engagement with local communities may increase the risk of co-opting them into existing development paradigms, thereby reinforcing rather than counteracting existing inequalities. At the same time, quality could be weakened by a lack of uniformity in the methods used, jeopardising the scientific 'rigour' of the evaluations. The counter-argument is that quality is enhanced by a greater richness of opinion. Unfortunately, to date, very few studies have paid much attention to assessing learning and its concrete outcomes.
power relationships and learning
Skewed power relationships, that are inherent to development practice due to the unequal access to resources, affect the process and outcomes of learning even when local ownership and mutual learning are intended. In their analysis of institutional learning with respect to local forest management in the 2001 book Social organization of innovation, Paul Engel and Monique Salomon point to the need for the empowerment of local stakeholders and conclude that, even then, the national policy context may make or break the process at a local level. This leads us directly to the question of the management of learning: Who sets the rules? Who enforces them? Who evaluates the evaluators?
learning and the role of the evaluators
Where evaluation is used for learning and adaptive management, this implies a fundamental change in the role of the evaluator. From being a remote, research-oriented person trying to systematise the known and unearth the hidden, he or she becomes a process facilitator. His or her role is to help design and organise others' inquiry and learning effectively. Stakeholder analysis and communication skills, as well as the ability to manage group dynamics, become prime assets. Non-partisanship is a must, yet at times even objectivity may be challenged, e.g. when the empowerment of disadvantaged groups is felt to be a prerequisite for their full participation in joint learning. Enhancing learning will eventually mean a fundamental restructuring of the training, methodological baggage, professional skills and outlook of evaluators.
can learning help to mainstream the internalisation of evaluation results?
At first sight, pushing evaluation into a learning mode looks like an easy business. However, it leads more or less automatically to the next level: adaptive management, i.e. a systematic and continuous effort to adjust individual and organisational behaviour in accordance with the lessons of past experience. Consequently, agencies that are prone to promote learning by others will be drawn irresistibly into asking questions about their own in-house learning capability. Hence, capacity development and the institutionalisation of internal learning processes emerge as the logical complements to systematic efforts to improve the internalisation of evaluation results.
how can we create the conditions for effective learning?
Different approaches emphasise different enabling conditions for learning. The latter generally include the following:
- the absence of threats to openness and the sharing of opinions;
- curiosity and motivation on the part of the participants;
- the availability of intellectual and/or practical challenges, incentives and institutional support; and
- opportunities for practical follow-up.
In addition, Irene Guijt, Jim Woodhill and colleagues in their paper Learning through e-networks and related M&E issues underscore the need for a systematic and explicit learning process and awareness of the minimum conditions that need to be met to make joint learning effective.
some lessons
These conclusions may not be new, but they do underline the need to strengthen the effective uptake of evaluation results by means of learning.
evaluation policy
Improving the internalisation of evaluation results implies a shift in perspective with respect to the way in which development processes, and indeed development institutions, are managed. Adaptive management, i.e. flexibility and responsiveness to experience, must become a key institutional feature. As a consequence, institutional development is a necessary complement to learning by development agencies.
The balance between accountability and transparency on the one hand, and learning and adaptive management on the other, needs further scrutiny. At the same time, it is important to be aware of any power imbalances between donors and aid recipients. We shall need to develop ways and means of safeguarding the credibility of evaluators and development professionals engaged in learning through evaluation.
evaluation practice
In terms of methodologies, what is needed is a pragmatic, process-oriented approach rather than a dogmatic, content-oriented one. There is no single set of tools that guarantees learning. Indeed, many different ones can be used if the principles governing successful learning are taken into account. These include mutual respect, inclusive thinking, a readiness to understand other people’s perspectives, and a willingness to take criticism seriously and to revise institutional cultures, rules and procedures.
Complexity, unpredictability and uncertainty are regular features of multi-actor learning processes. In her chapter in the book Learning from change, Irene Guijt argues that methodological issues 'extend far beyond simply which method works best, as these are just a small part of the extensive communication processes that lie at the heart of monitoring and evaluation.'
evaluation professionals
The need to rethink the professional profile of evaluators is paramount if they are to facilitate learning. Apart from having excellent research and communication skills, the evaluators of the future will also need to possess process facilitation, conflict resolution and negotiating skills. Analytical skills will have to include tools for situation and stakeholder analysis, as well as for understanding conflict and group dynamics.
evaluation research
Far more emphasis needs to be placed on developing effective conceptual and practical tools. So far, research has hardly touched upon the systematic assessment of the learning effects of different monitoring and evaluation approaches, tools and techniques.
south-north and south-south exchange
The flow of lessons on practices and tools from the South to the North, and between Southern actors, is a field of inquiry in which much can be learned from systematic collaboration among evaluation researchers and practitioners.
This Briefing is an abridged version of Making evaluation results count: Internalising evidence by learning, ECDPM Policy Management Brief no. 16. It draws substantially on a workshop entitled 'How can we learn more from what we do? Evaluation and evidence-based communications for development'. The workshop was organised jointly by the ECDPM and Exchange, and was held in Maastricht on 13-14 December 2002. Its findings were also presented to the DAC Working Party on Aid Evaluation Workshop in Paris in March 2003.
Paul Engel is Director, Charlotte Carlsson Programme Officer, and Arin van Zee is Programme Assistant at ECDPM
back to the main page
