Keynote speech at the Seminar on Implementing Climate Change Policy Evaluation
National Institute of Ecology and Climate Change (INECC), Mexico, May 25, 2016
Progress and challenges to evaluate environmental and climate change policies
- Dr. Juha I. Uitto, Director, Independent Evaluation Office of the Global Environment Facility
Ladies and gentlemen,
It is a great honor to have been invited as keynote speaker to this impressive gathering on Implementing Climate Change Policy Evaluation. Needless to say, this topic is very close to my heart. It is also a pleasure to attend this conference in the fascinating and bustling Ciudad de Mexico.
Climate change is one of the most complex contemporary challenges facing humankind. Its complexity stems from the fact that it is linked with virtually all sectors and areas of human activity. Addressing climate change is therefore not only a matter of technological solutions or dealing with greenhouse gas emissions. Climate change calls for integrated solutions that encompass the social, economic and political alongside the environmental and technical.
The 2030 Agenda for Sustainable Development and the associated Sustainable Development Goals adopted by the UN member states in September 2015 recognize this . The SDGs all take a holistic approach incorporating the social, economic and environmental pillars of sustainable development. The Global Environment Facility has mapped its work on climate change against the SDGs. While the SDG 13 is explicitly concerned with climate action, our work on climate change is also relevant to several other goals, which have direct linkages to societal factors, including goal 5 (Gender Equality), 7 (Affordable and Clean Energy), 8 (Decent Work and Economic Growth), 9 (Industry, Innovation and Infrastructure) and 12 (Responsible Consumption and Production) .
Unlike their predecessor Millennium Development Goals, the SDGs are universally applicable to all countries. Also unlike the MDGs, evaluation has been built into the SDGs, which goes beyond monitoring and indicators. Evaluation can help countries and the international community to understand why progress towards the achievement of the goals is taking place or what are the hindrances along the way. There is a need to mobilize national data systems and to set up M&E that enable reporting on the SDGs and that at the same time serve national needs and perspectives.
Mexico is a large country with a great variability of situations when it comes to geography and natural conditions, as well as levels of economic development. Mexico’s development has been rapid and it is now an advanced industrialized country. The UN analyses place Mexico in the high human development category . As everywhere, such rapid industrialization and economic development puts strains on the environment. Mexico is in the top dozen countries in the world when it comes to greenhouse gas emissions. Population and income growth, as well as urban sprawl and reliance on private automobiles all contribute to the growth in emissions. OECD recognizes the efforts made in the recent years in giving high political priority to battling climate change and protecting the environment . This event is proof of that and focuses on the important topic of the role of evaluation in ensuring that policies tackling climate change are effective.
Evaluation of climate change responses faces specific challenges. Some are generic to the environmental arena , while others pertain to the very nature of climate change. Basic issues pertaining to evaluating environmental actions in general include the fact that environmental phenomena tend to involve long time horizons. The projects and programs we set up are usually time-bound. They may set in motion processes that we hope will lead to environmental outcomes at a later stage, but this often happens only long after the project is gone. Similarly, environmental phenomena often have different geographical scales than the human systems within which we operate. Watersheds cross jurisdictional boundaries – often even international borders; water both above and below ground flows across human set administrative units; fish and wildlife do not care about jurisdictional boundaries; pollution is carried by rivers and, especially, by air from one place to another often across long distances. Climate is typically a global common . It is influenced by anthropogenic factors, which are localized, but the effects are global. Economic activities – energy use, transport, industry, agriculture, deforestation – that cause climate change bring profits to those engaging in them, at least in the short term. Sadly, the brunt of the costs of climate change accrue primarily to the poor countries and poor people who contributed little to the problem in the first place.
These factors – long time horizons, differing geographical scales, impacts occurring in places other than the sources – all complicate evaluation of environmental phenomena. Climate change has added further complications to these issues because of the uncertainties associated with it. Even the best climate models are unable to accurately predict what will happen and the lower one gets on geographical scale the larger the uncertainties. There are discontinuities and tipping points, but we cannot be sure where they are. The Paris Agreement that has just recently been ratified by countries set the limit of warming at 2oC beyond which scientists believe the consequences will be uncertain.
The frequently used shorthand, global warming, is misleading, as climate change appears to lead to increased variability in weather and different effects in different geographical areas. About half of Mexico’s territory is either desert or semi-desert with very limited freshwater resources. The Intergovernmental Panel on Climate Change warns that warming in the high mountain areas could lead to less snow and ice, which could affect river flows and subsequently further reduce water availability . IPCC further predicts, with high confidence, that increased temperatures will reduce crop yields, including those of maize and wheat, by shortening the crop cycle. These are significant risks.
Therefore, climate change policies and programs – as well as their evaluation – must deal explicitly with risk and uncertainty . These challenges are further exacerbated by data and information gaps.
As multiple factors contribute to climate change, it is also difficult to isolate the effects of particular interventions. We may try to curb emissions from transport or industry, or stop deforestation and transformation of forests into pastures for cattle, all of which will make contributions towards the fight against climate change. But can we really attribute any observed changes in climate to these programs? This is definitely hard, especially given that the actions to mitigate climate change are dwarfed by the environmentally destructive practices. The 5th Overall Performance Study of the GEF brought this into focus . While the total funding to global environmental issues is estimated at US$10 billion per year (of which the GEF stands for about US$1 billion), the needs would be in the range of US$100 billion. On the other hand, just subsidies to environmentally destructive practices – including fossil fuels and industrial agriculture – amount to some US$1 trillion annually.
This does not mean that attempts to control emissions through policies, programs and projects is futile. On the contrary, we need to do it and continue intensifying our efforts. What it means, though, is that we also must strengthen our M&E systems and our evaluation approaches so that we know better whether our efforts are effective and that we are able to learn from the past and focus our work on those approaches that have greatest potential.
My office, the Independent Evaluation Office of the GEF, conducted an impact evaluation of climate mitigation projects funded by the GEF a couple of years ago . Mexico was one of the four countries covered in the evaluation because the major emerging market economies are particularly important for the climate change mitigation potential. The evaluation found that the projects that were particularly successful in demonstrating progress towards impact were those, which had adopted comprehensive approaches to address market barriers and specifically targeted supportive policy frameworks.
So how can evaluators address such daunting challenges? I believe there are major opportunities and solutions but we have to strengthen the approaches and methodologies we use. Theory-based approaches to evaluation will still be useful, but they must be sensitive to the fact that linear models of causality may not be valid. We need to start by understanding the system boundaries, the components of the system, their relationships and the emergent properties when we design our evaluations . How do we do that? In the first place, it is important to be aware of what science tells us, so a good evaluation should start by a review of pertinent scientific literature on the topic. Based on this understanding, we can construct the theory of change that will serve as a hypothesis of how causality works in the particular context of the evaluand at hand. This hypothesis is then to be tested and the theory of change updated as needed in light of data and as our understanding of the evaluand increases.
We also must use our data collection and analysis methods based on the questions that we pose and the situation in which we evaluate. The form must follow function. Some evaluators and researchers believe that only experimental and quasi-experimental methods, such as randomized control trials, yield reliable evaluative evidence. This is a fallacy. Such quantitative methods have their place in evaluation, but their applicability is limited. They can best be used in evaluating narrow interventions where counterfactuals and control groups can be identified in a clear manner. Even in medicine, RCTs are used only as one step in a rigorous and extensive research and testing process. Similarly, in sustainable development evaluation they can only be part of the solution and a relatively small at that. We need to employ a variety of methods, both quantitative and qualitative, to address the complex issues related to climate change.
And I come back to the need for integrated, holistic perspectives. It is not possible to evaluate climate change in isolation focusing only on mitigation of greenhouse gas emissions. We must deal with the drivers of climate change and environmental degradation. Issues such as urbanization and population growth, land use change, deforestation and carbon sequestration are key factors driving climate change and we must therefore focus on them. These in turn have close linkages to societal factors, such as economic and social development, poverty, inequality, politics and power relations. Climate change is inherently political and economic. All interventions, whether policies, programs or projects, must operate in the political and economic arena. Similarly, evaluations must be cognizant about and deal with these issues as well.
Most often, the interventions operate at the level of addressing societal issues, developing policies, creating incentives or disincentives that we hope will lead to more sustainable behaviors, influencing consumption and production patterns. Evaluation must thus focus on these intermediate outcomes to help understand the intended and unintended consequences of our actions. We must use evaluation to learn from past efforts what works, through which mechanisms, why, and under what circumstances. At the same time, we must keep our eye on the ball, never to forget that our ultimate goal is to reduce greenhouse gas emissions to the atmosphere and to stabilize climate change. Too many policies, programs and projects have done what they set out to do, reached their objectives, but barely made a dent in the big picture. Evaluators have the responsibility to look beyond outputs of individual interventions to determine whether they are making a difference, contributing to our ultimate goal.
As I mentioned earlier, climate change impacts vary by geography. But they also affect different groups of people in different ways. It is a fair statement to say that poor people are generally more vulnerable to climate change impacts than the wealthy people. Their choices and ability to cope with climate impacts are limited. They have a harder time to bounce back after a major storm destroys their property or a prolonged drought causes a crop failure. There are pockets of highly vulnerable groups in Mexico, notably those living in isolated rural communities depending on traditional agriculture in places like Chiapas, but also in central Mexico not too far from here . As mentioned earlier, climate change is affecting crop productivity and water availability and, thus, directly their livelihoods and wellbeing. Furthermore, coastal areas in places as varied as Quintana Roo, Veracruz, Tabasco and Guerrero are at risk from coastal inundation, saltwater intrusion and tropical cyclones.
We must not deal only with climate change mitigation, but importantly with adaptation. This was recognized in the Climate Change COP and the Paris Agreement for the first time gives equal weight to the need to adapt to climate change impacts. In practice, this means reducing the vulnerability and increasing resilience of people and infrastructure to climate change, including more frequent natural disasters, such as intensified coastal storms and increased weather variability. Evaluation approaches in adaptation are still being developed and our knowledge of this aspect is somewhat behind that of evaluating climate change mitigation. It is, however, important to recognize that evaluating adaptation places us firmly in the realm of human systems and the social, economic and political spheres.
As we evaluate actions that are intended to reduce vulnerability and increase resilience, we must again take an integrated and holistic view that encompasses both natural and human systems. We have choices in the ways we seek to enhance resilience. For example, we may decide to build a seawall to protect a community against storm surges and sea level rise. Or we may decide to restore the coastal mangroves that provide a natural protection against the sea, while also acting as spawning grounds for fish and crustaceans, and providing ecosystem services for example by way of water purification. These choices have consequences for the coastal communities and their livelihoods. Evaluators should be able to assess the benefits and risks and the long term sustainability of such choices and to inform policy.
The focus of this seminar is on evaluation of climate change policy. Policy evaluation always entails uncertainty and I would argue that this uncertainty is increased further by climate change. In all cases, policy effects are long term and complex. They depend on multiple intervening factors, as no policy takes place in a vacuum. Many of the intervening factors are beyond the control of policymakers. They include economic factors that may depend on developments in the global economy. Technological and scientific advances can make policies obsolete or irrelevant. And natural factors, such as a catastrophic storm or weather-related event may overwhelm the policy-responses. Again, climate change adds risk and uncertainty to the policy.
Policies are also only as good as how well and consistently they are implemented. We need regulatory frameworks and legislation that is enforced for policy to be implemented. Another factor is the predictability of policy. In democratic systems where the political pendulum may swing significantly following an election, the new government may attempt to reverse policy.
These factors again highlight the need for climate change evaluators to look at the phenomena in an integrated and holistic manner focusing on the effectiveness of policy implementation and the longer term consistency. Are policies relevant and effective in addressing the drivers of climate change? Are they relevant and effective for reducing vulnerability and building resilience?
In order for us to be able to do this, we need to develop robust M&E systems and collect systematic data. But we also need a broad range of rigorous evaluation methodologies and solid theories of change to understand why progress is or isn’t made, and how we can improve future policies and performance. Evaluative evidence of this kind is essential for Mexico and all other countries to make informed choices in the fight against climate change. It also serves the dual purpose of being able to assess and report back on the achievement of the SDGs.
Thank you.
