Evaluation in the time of pandemic

On a gondola ride up to a biodiversity rich mountain area in Sichuan, China (photo by the author).

The year 2020 has been defined by the COVID-19 pandemic that has disrupted lives and livelihoods everywhere around the world. The way we work has been interrupted and altered. This is true for those working to advance and manage international environmental projects and programs – it is also true for the professionals working to evaluate for effectiveness and impact of those initiatives. At the GEF Independent Evaluation Office (IEO), the body I lead, we have had to innovate in our data collection and analysis to counter the travel and other limitations posed by the situation. The fact that the coronavirus causing the pandemic is zoonotic, and thus directly linked to how humanity exploits and abuses the natural environment, places it at the center of the Global Environment Facility’s work. For independent evaluators, equally, understanding the connections between the human and natural systems, between environmental health and human health has become essential.

The pandemic struck at a critical time for the IEO, as we are in the midst of the Seventh Comprehensive Evaluation of the GEF, known as OPS7. These periodic evaluations are an important part of the GEF’s four-year replenishment cycle, providing evidence of the multilateral funding body’s impact and performance and informing the preparations of new policies and programs. In the IEO, we rely on solid data and information from multiple sources as the basis of our evaluations, using both quantitative and qualitative methods for analysis.

While analysis of portfolio data and the use of project-level evaluation reports are building blocks of our evaluations, being able to collect information from the field in the countries where GEF-supported projects and programs take place is usually essential to our work. After all, while the GEF’s purpose is to tackle pressing global environmental problems, and support global environmental benefits, its projects and programs are also intended to benefit the countries and the people in those countries where they operate. It is important for our evaluators to be able to observe what happens on the ground in all the places and all the areas where the GEF operates. We need the perspectives of the government and civil society representatives, as well as the agencies that implement and execute the projects. Crucially, we must understand the needs and concerns of the people whose lives are affected by GEF-supported interventions. Therefore, data collection in the field is a regular part of our evaluations. When the seriousness of the pandemic hit home in early March, IEO colleagues were conducting field visits in far-flung places, from Samoa to Ecuador, and had to be called home on short notice.

Our pre-pandemic strategy of expanding the use of national consultants for country expertise and broader country coverage over time helped us in this time of crisis, and we were quickly able to leverage experts in the field around the world to continue our work. In the evaluations that have continued to progress this year, we were able to engage local in-country consultants to collect data and information about GEF programs and projects, based on agreed protocols. These included country studies in Mozambique and Costa Rica for the Evaluation of the Role of Medium Size Projects in the GEF Partnership, and project cases in Peru and the Philippines for the Evaluation of GEF Interventions in the Artisanal and Small-scale Gold Mining Sector. There still were some limitations, as we did not want to expose our consultants or stakeholders to any health risk, but the national consultants were in a much better position to interact with and hear from local actors than we would have otherwise been able to glean.

Several evaluations contributing directly to OPS7 were already well underway before COVID-19, with field work completed before the pandemic made travel impossible. But there were also earlier studies that could be mined for the purposes of current evaluations. The datasets and information collected had been utilized for other purposes but could be used also to dig deeper into present evaluation questions.

It is worth noting that the IEO is not facing these challenges alone. The networks in which we participate, including the United Nations Evaluation Group and the Evaluation Cooperation Group of the international financial institutions, have been actively finding solutions in similar circumstances. We have been able to work together with our partners, such as the independent evaluation bodies of the World Bank, UNDP, and IFAD, in coordinating and honing robust approaches to data collection under these extraordinary circumstances. In some areas, we have been recognized as leaders. One such area pertains to the use of geospatial tools, including remotely sensed data. Those familiar with IEO’s work know that we have pioneered such techniques since OPS5, and our work has become increasingly sophisticated in recent years. We have expanded our analysis to factors beyond land cover change and vegetation productivity to also now include value-for-money and socioeconomic analyses. In a recent Uganda case study, the IEO team overlaid data on GEF-supported sustainable forest management projects with World Bank socioeconomic household survey data, which also was geocoded, and was able to demonstrate a positive correlation between the GEF interventions and household wellbeing.

Our role as independent evaluators is not only to verify whether each project or program has achieved the goals set for it. Evaluation goes far beyond performance auditing in that respect. To be truly useful, evaluation must not only look at what was achieved in the past but also take a perspective towards the future. Such a perspective must be based on an analysis of what has worked, under what circumstances, and why. We must also look for missed opportunities and unintended consequences. To be able to do this, evaluation must tap into cutting-edge knowledge on the topic being evaluated, especially in areas that are novel in the context of the GEF. In the evaluations of GEF Support in Fragile and Conflict-affected Situations and gold mining, the IEO worked closely with leading external experts who could bring to the table state-of-the-art thinking that would help the GEF move forward in these critical areas.

The ongoing pandemic has forced us to think creatively about evaluating the GEF. In light of the above, I am personally confident that the IEO will be able to continue delivering quality evaluations that our partners have come to expect from us. Equally, the Seventh Comprehensive Evaluation of the GEF will provide timely and reliable insights into the next replenishment process, along with lessons learned from this novel experience that will feed into future evaluations processes as well.

[Originally published at the Global Environment Facility website at https://www.thegef.org/blog/evaluation-time-pandemic]