Evaluation in the time of pandemic

On a gondola ride up to a biodiversity rich mountain area in Sichuan, China (photo by the author).

The year 2020 has been defined by the COVID-19 pandemic that has disrupted lives and livelihoods everywhere around the world. The way we work has been interrupted and altered. This is true for those working to advance and manage international environmental projects and programs – it is also true for the professionals working to evaluate for effectiveness and impact of those initiatives. At the GEF Independent Evaluation Office (IEO), the body I lead, we have had to innovate in our data collection and analysis to counter the travel and other limitations posed by the situation. The fact that the coronavirus causing the pandemic is zoonotic, and thus directly linked to how humanity exploits and abuses the natural environment, places it at the center of the Global Environment Facility’s work. For independent evaluators, equally, understanding the connections between the human and natural systems, between environmental health and human health has become essential.

The pandemic struck at a critical time for the IEO, as we are in the midst of the Seventh Comprehensive Evaluation of the GEF, known as OPS7. These periodic evaluations are an important part of the GEF’s four-year replenishment cycle, providing evidence of the multilateral funding body’s impact and performance and informing the preparations of new policies and programs. In the IEO, we rely on solid data and information from multiple sources as the basis of our evaluations, using both quantitative and qualitative methods for analysis.

While analysis of portfolio data and the use of project-level evaluation reports are building blocks of our evaluations, being able to collect information from the field in the countries where GEF-supported projects and programs take place is usually essential to our work. After all, while the GEF’s purpose is to tackle pressing global environmental problems, and support global environmental benefits, its projects and programs are also intended to benefit the countries and the people in those countries where they operate. It is important for our evaluators to be able to observe what happens on the ground in all the places and all the areas where the GEF operates. We need the perspectives of the government and civil society representatives, as well as the agencies that implement and execute the projects. Crucially, we must understand the needs and concerns of the people whose lives are affected by GEF-supported interventions. Therefore, data collection in the field is a regular part of our evaluations. When the seriousness of the pandemic hit home in early March, IEO colleagues were conducting field visits in far-flung places, from Samoa to Ecuador, and had to be called home on short notice.

Our pre-pandemic strategy of expanding the use of national consultants for country expertise and broader country coverage over time helped us in this time of crisis, and we were quickly able to leverage experts in the field around the world to continue our work. In the evaluations that have continued to progress this year, we were able to engage local in-country consultants to collect data and information about GEF programs and projects, based on agreed protocols. These included country studies in Mozambique and Costa Rica for the Evaluation of the Role of Medium Size Projects in the GEF Partnership, and project cases in Peru and the Philippines for the Evaluation of GEF Interventions in the Artisanal and Small-scale Gold Mining Sector. There still were some limitations, as we did not want to expose our consultants or stakeholders to any health risk, but the national consultants were in a much better position to interact with and hear from local actors than we would have otherwise been able to glean.

Several evaluations contributing directly to OPS7 were already well underway before COVID-19, with field work completed before the pandemic made travel impossible. But there were also earlier studies that could be mined for the purposes of current evaluations. The datasets and information collected had been utilized for other purposes but could be used also to dig deeper into present evaluation questions.

It is worth noting that the IEO is not facing these challenges alone. The networks in which we participate, including the United Nations Evaluation Group and the Evaluation Cooperation Group of the international financial institutions, have been actively finding solutions in similar circumstances. We have been able to work together with our partners, such as the independent evaluation bodies of the World Bank, UNDP, and IFAD, in coordinating and honing robust approaches to data collection under these extraordinary circumstances. In some areas, we have been recognized as leaders. One such area pertains to the use of geospatial tools, including remotely sensed data. Those familiar with IEO’s work know that we have pioneered such techniques since OPS5, and our work has become increasingly sophisticated in recent years. We have expanded our analysis to factors beyond land cover change and vegetation productivity to also now include value-for-money and socioeconomic analyses. In a recent Uganda case study, the IEO team overlaid data on GEF-supported sustainable forest management projects with World Bank socioeconomic household survey data, which also was geocoded, and was able to demonstrate a positive correlation between the GEF interventions and household wellbeing.

Our role as independent evaluators is not only to verify whether each project or program has achieved the goals set for it. Evaluation goes far beyond performance auditing in that respect. To be truly useful, evaluation must not only look at what was achieved in the past but also take a perspective towards the future. Such a perspective must be based on an analysis of what has worked, under what circumstances, and why. We must also look for missed opportunities and unintended consequences. To be able to do this, evaluation must tap into cutting-edge knowledge on the topic being evaluated, especially in areas that are novel in the context of the GEF. In the evaluations of GEF Support in Fragile and Conflict-affected Situations and gold mining, the IEO worked closely with leading external experts who could bring to the table state-of-the-art thinking that would help the GEF move forward in these critical areas.

The ongoing pandemic has forced us to think creatively about evaluating the GEF. In light of the above, I am personally confident that the IEO will be able to continue delivering quality evaluations that our partners have come to expect from us. Equally, the Seventh Comprehensive Evaluation of the GEF will provide timely and reliable insights into the next replenishment process, along with lessons learned from this novel experience that will feed into future evaluations processes as well.

[Originally published at the Global Environment Facility website at https://www.thegef.org/blog/evaluation-time-pandemic]

Towards evaluation for a sustainable and just future

Transporting logs on Rio Tapajos in Amazonia (photo by author).

Over the past eight months, the novel coronavirus pandemic has infected some 20 million people and killed more than 700,000, sparing virtually no country. The economic and social consequences have been devastating. The virus SARS-CoV-2 that caused COVID-19 crossed over from its non-human host, probably a bat, directly or more likely through an intermediate host like a pangolin, to a human in or around the city of Wuhan in China in late 2019. The exact transmission mechanism is still not known but the root causes are clear. The spill-over of zoonotic viruses like SARS-CoV-2 is becoming more common as we come into ever closer contacts with animals, both domesticated and wild.  As human activities extend deeper into undisturbed ecosystems, undiscovered pathogens are released. The destruction is driven by the expansion of agriculture and cattle ranching, logging and deforestation, road construction, mining, new settlements and urban sprawl, making space for the growing human population and its ever increasing demands for raw materials, food stuffs and consumer goods.

Although COVID-19 in itself was not known, the coming pandemic was widely predicted by scientists and there were even government taskforces set out to prepare us for its eventuality. There were precedents—SARS, MERS, H1N1, Zika, Ebola and others—although their impacts were much more modest. COVID-19 spread like a wildfire in a globalized world—there were 3 billion airline trips taken in 2019—due to its characteristics of being airborne and contagious before infected persons become symptomatic.

What does any of this have to do with evaluation, you might ask. In my view, everything. And if not, what is the relevance of evaluation to the real problems of the world? The pandemic is an illustration of the kind of challenges we face today, how interconnected the world is, and how events in one place have global consequences. It also shows how economic development and environmental degradation are intimately intertwined. As we cut down trees, not only do we come into contact with lethal pathogens, but we also undermine the forest’s ability to sequester carbon thereby speeding up global warming. As people get richer, their diets tend to become more meat-based. There are now half a billion cows and 23 billion chicken on the planet. There is a patch the size of Denmark in the Amazon, which has been cleared to grow soy beans to feed pigs in Denmark. Another consequence of the increased meat consumption is higher rates of obesity, diabetes and cardiovascular diseases even in countries that previously didn’t experience them. A recent study by Harvard University provided strongest evidence yet linking air pollution directly to higher mortality. Human health and ecosystem health are inseparable.

The pandemic has affected different groups and communities differently. In the USA, Black, Indigenous and other People of Colour (BIPOC) have been disproportionately hit because they are more likely to be employed in essential jobs that cannot be done remotely, and their living conditions are more cramped. They may also have more pre-existing medical conditions rendering them more vulnerable to the virus. Climate change affects the poor and vulnerable communities hardest, whether it is those living on the low-lying coast of Bangladesh pummelled by more frequent cyclones and sea-level rise or small farmers in African drylands suffering during prolonged droughts.

Many evaluators write about these global challenges, using terms like ‘complex’ and ‘wicked,’ but I am not sure that the practice of evaluation has kept up with the theory. Evaluation as a profession has its roots in social inquiry, where we test the effectiveness of interventions on a well-defined treatment population against a control group. We may use experimental or quasi-experimental tools, or we may lean more towards more participatory and qualitative approaches, but either way the focus is on a single intervention and its effects. Our evaluations test the effectiveness of the intervention in terms of its pre-determined objectives. The desire is to be able to attribute any changes in the outcome to the intervention—or, recognizing the complexity and presence of multiple actors, at least the specific contributions of the intervention.

Apart from being narrowly project-focused, evaluations are still driven by donor concerns for accountability and ‘value for money.’ This treats the central question as a matter of simple accounting instead of a choice between types of intervention or organization that can, say, lift the largest number of people out of poverty with the least amount of money. To make things worse, the accounting in development cooperation is for the purposes of the donors and their priorities, not for the benefit of the claim-holders that the project is intended to benefit. This accounting mentality in evaluation tends to miss the big picture and may end up doing more harm than good.

Seldom do evaluations look at the big picture: Are we actually doing the right thing? Is the intervention that we are promoting meaningful in the larger whole? Is it something that the intended beneficiaries want and need? Is it fixing one part of the problem but creating others elsewhere? Is it having unintended consequences for the environment, for disadvantaged groups, for indigenous peoples, for power relations, etc.?

We must incorporate the environment into our evaluations. Sustainable development lies on social, economic and environmental foundations, yet evaluation—like national accounting—is almost exclusively concerned with the economic and, to a lesser degree, social capital, while natural capital and its depreciation are considered external to the system. According to the World Bank, low-income countries get 47% of their wealth from natural capital. This figure certainly underestimates the value of ecosystem services, in terms of clean water and air, health benefits, recreation, protection against natural hazards, etc. Evaluators must learn how to operate at the nexus of environment and development, which means understanding the interplay between human and natural systems.

Some of these lessons come out clearly in the evaluation of the GEF Global Wildlife Program, which is directly relevant to warding off pandemics such as the current one. The evaluation revealed the need to address the root causes of illegal wildlife trade on multiple fronts while also protecting endangered species in situ. Working with local communities to provide sustainable livelihoods is important, but not sufficient. It is essential to address political will, corruption and demand for wildlife products in the market countries of Asia, Europe and North America. Such interventions—and evaluating them—require holistic perspectives and a broad understanding of the dynamic systems.

For evaluation to remain relevant, it must rise above its project mentality and start looking beyond the internal logic of the interventions that are evaluated. It must systematically search for unintended consequences that may lie outside of the immediate scope of the evaluation. It must expand its vision to encompass the coupled human and natural systems and how they interact. And it must resist focusing on accountability for donors and instead make sure that it contributes to learning, for the wellbeing of the beneficiaries and nature in an equitable manner. If we achieve this, evaluation will be better positioned to contribute to more sustainable and just development in an interconnected world.

[Originally published on the website of the European Evaluation Society (https://europeanevaluation.org/2020/08/17/towards-evaluation-for-a-sustainable-and-just-future/)]