New Issues in Evaluation

INTRODUCTION
Evaluation is a formalisation of the kinds of questions that arise for nearly everyone who spends money: Do they get value for it? And what is value for money, anyway? Did the purchase meet expectations? Does it meet current requirements? And there are other questions that the more careful consumer may ask: will it still meet requirements in the future? Has a better and/or cheaper alternative emerged in the meantime?Public bodies, such as governments, public institutions, and international organisations are particularly interested in evaluation. This is principally because it is the only way that they can measure their performance on a continuous basis. Private enterprises have many measures easily available to them, such as return on capital, market share, and returns to shareholders. But public bodies lack such composite metrics: the only way that they can see how they are succeeding in their work is to review performance under the headings of individual projects or programmes. And that review often has to be qualitative, because the goals of their activities cannot always be quantified.
There are exceptions to this general statement. Capital investment projects, in particular, will have included consideration of the returns to capital when the project was being justified, and the social costs and benefits may have been included also, especially when the project was part of development assistance activity. This quantification will provide a set of targets against which the outturn can be assessed. However, if it is a development capital project, the funders may have wider questions for which no indicators are readily available.
The importance of evaluation for public bodies is obvious, because their expenditures are so significant in their economies. Government consumption is typically around 20 per cent of GDP, whether it is a rich or a poor country. And much of government expenditure has a wide purpose, even if at first sight it seems narrow. Even the purchase of supplies by a government department, such as paper for the office, may have a broader agenda: the government may have a policy of favouring indigenous suppliers, or small businesses, or businesses that also fulfil some other social criterion, with the government’s objective being therefore to make progress towards some goal in this regard.
For development assistance, there is a target for developed countries to reach 0.7 per cent of GDP. A few have already done so, and even the worst performer, the United States, still attains 0.25 per cent of GDP. While these are small percentages compared to total government expenditure, they represent financial commitments from wealthy countries. So the absolute values are high, of great importance to the recipients but also of interest to the taxpayers of the donor countries. This means that evaluation is of wide concern. Quite apart from this, it is of pressing concern to the policy-makers in the donor countries, because it may provide answers to troubling questions: are we getting value for money? Are we doing the right thing?
These concerns have led to considerable activity over many years directed towards improving evaluation itself. The principles that should underlie evaluations have been articulated. The methodologies to be followed have been elaborated in guidelines and manuals. There is broad international cooperation in this activity. The United Nations Evaluation Group (UNEG) which brings together most of the relevant UN agencies has recently revised its general principles for carrying out evaluations, first set out in 2005, and has also prepared a handbook on how to evaluate the kind of work that the UN system does in developing new treaties, regulations and other agreements, including the efforts to translate these into action at national level. The Development Assistance Committee of the OECD (DAC) plays a major role in evaluation also: work has been developed on determining the kinds of resources needed for good evaluations, what the staff requirements are and what training they should have. Information exchange on evaluations, including the development of repositories of completed evaluations of wider interest has also been undertaken. The activity also includes the drawing up of key reference documents on the topic and the extensive work of the DAC Network on Development Evaluation. The OECD principles appear to be generally followed by its members. For instance, Austria bases its own manual for project and programme evaluation on them.
JUSTIFICATION
In spite of this activity, there are a number of things happening in the world that may not have been taken fully into account. There is accelerating change in the following areas
• Climate change: under this broad heading is a broad range of alterations in the environment, including air and water temperatures, sea levels, cultivable land areas, and living species adaptation which will in turn lead to major population migrations, new patterns of health risks, and associated political instability.
• Technological change: invention leading to innovation will continue, with new products and services emerging, affecting both productivity, markets and consumer behaviour. Information and communications technologies (ICTs), biotechnology, and new materials are the main lines of development but the need to combat climate change will continue to stimulate energy-related technologies including renewable energy and storage technology.
• Change in the structure of the world economy: Consumption structures are converging, government. consumption is a stable share of the total, and we are heading towards a single production system that is global. Regional integration is also on the rise, and it goes beyond trade to include investment, STI, infrastructure and technology cooperation and so on. How can this be taken into account?
These point to the need for three special emphases for evaluations in the years to come:
1. A longer view: the future is getting nearer. The changes taking place in technology and in economic structure globally are accelerating, and those associated with climate change may in the worst case be catastrophic and in the medium term likely to be enormous in any case. Evaluation should have a perspective on how the world will look at the time the project is completed but also whether the project is contributing or otherwise to possible solutions to more than the immediate problems. In turn this points to the need for reference scenarios for foresight, to extend the framework for evaluations. Scenarios present snapshots of possible futures. They are tools for focusing thought, developing shared visions, and determining policy. They do not predict the future, but help us to decide what to do now in order to shape it. One way to construct scenarios is to begin with a long list of factors of interest and then identify the key determinants of change, those that are expected to have the most impact but are outside the control of the policy maker in question. These could in the future take extreme values that will lead to major difficulties in economic, social and security terms. Different combinations of extreme values of these key variables can then be considered in order to give a sense of the full range of possibilities for the future. The scenarios can be constructed through analysis of trends in technologies, the environment, health, demographics, and major natural and human changes that are altering the economic landscape and economic geography.

2. A broader view: the connections may be important. Because of the growing complexity of the world economy, individual aspects tend to be more linked together than they used to be, and this linkage will increasingly be international. So a factory, a set of small businesses, or a group of farms may have some relations with other parts of the world, in that they may be dependent on it for markets or for essential supplies. But there may also be linkages that are not explicit: the economic unit may find its markets disappearing as a result of changes in markets or technological innovation in other parts of the world. Less likely, but not impossible, is the converse: a new trend in some markets may give an unexpected boost to a traditional crop for instance, as is the case with maca production in Peru, but even here the benefit may be temporary. Better understanding of complex systems is needed. The Sustainable Development Goals (SDGs) can be used to provide a first checklist in considering the interactions between different sectors of activity. They may also provide a framework for scenario work as discussed above. Work in other areas is also relevant, such as the Bellagio STAMP principles of the International Institute for Sustainable Development (IISD) for assessing sustainability.
3. More speed in evaluation: the results are time-critical. Because of the rapid pace of change in the world, an evaluation has to be done quickly if it is not to be irrelevant. Also, its results have to be disseminated (and promoted to policy-makers) if its findings are to be acted on in time. Access to other relevant evaluations will be very helpful in this, but to achieve it there is a strong requirement for better reference evaluation database systems to support the use of other evaluations as references. This will call for increased cooperation internationally, including involvement of the new agencies appearing on the landscape, such as the Asian Infrastructure Investment Bank (AIIB) and the International Renewable Energy Agency (IRENA), as well as existing bodies likely to increase in importance in the years to come, such as regional economic groupings and the Arctic Council.

The significance of these new directions for evaluation is summarised in the table below. It suggest the priorities to be given to the long view, the broad view and the speed of evaluation for different types of evaluations.

Table: Priorities with respect to new issues

Priorities for evaluation