Tips for young and emerging evaluators

This blog was originally posted at www.zendaofir.com.

Unprecedented wildfires, deadly heatwaves, extraordinary floods, storms and droughts from California to Sweden, from Greece to Japan. The northern summer of 2018 has given us a preview of what the greenhouse world will look like, as countries and governments have become increasingly divided and inward looking, struggling to find solutions to sustainable development. Scientists are now more confident than ever linking the likelihood of extreme weather events to global climate change. Already now, climate change affects all spheres of human endeavor, including economics, health, population movement, food security, and politics. It will have huge implications on development around the world and, especially, on the most vulnerable people.

Evaluation can play a major role in analyzing policies, strategies, programs and projects in light of what works, under what circumstances, and how our actions can lead to positive changes for the people and the planet. It is up to the next generation of young and emerging evaluators to rise to this challenge.

Top Tip 1. Think beyond individual interventions and their objectives. For evaluation to remain relevant, the profession must broaden its horizons beyond checking whether individual interventions are doing what they were set out to do. It must verify whether the interventions are having an impact on the problem they are addressing and whether the impact is lasting. Evaluation is not just about monitoring and indicators, nor is it about performance audit. It is about understanding and explanation of how change happens and how we can more effectively enhance positive change and minimize negative consequences.

Top Tip 2. Understand, deal with and assess choices and trade-offs made or that should have been made. What we know clearly from experiences at the Global Environment Facility (GEF) is that all interventions take place in a broader system, which is dynamic and complex. It is safe to assume that everything we do influences things beyond the immediate effects of an intervention. And virtually all interventions will have implications for the environment, either positive or negative, intended or unintended. Sometimes we face trade-offs and must make choices between maximizing certain benefits at the cost of others.

In a recent GEF evaluation on multiple benefits, we identified such cases where maximization of both environmental and economic benefits was not possible, or where there were possible conflicts between environmental benefits, e.g., in terms of reforestation and maintaining biological diversity. The evaluation brought these factors out to the open for an informed policy and strategy discussion. It is no longer possible for evaluation to focus narrowly on the internal logic of an individual intervention without paying attention to the broader context in which it is situated.

Top Tip 3. Methods should not drive evaluations. While solid methodologies for data collection and analysis are essential, evaluators should not let methods drive the scope and question setting of evaluations. It must be the other way around: choose the approaches and methods based on what questions you need answers to. In most cases, mixed methods in the context of a solid theory of change is the way to go. Quantification of impacts is an attractive goal, but there are significant limitations to experimental and quasi-experimental tools with regard to their explanatory power and external validity. They seldom allow us to understand why something happened, what motivated people, what were the unintended consequences and so forth. For this, we need subtler, often qualitative tools.

At the GEF, we normally start our evaluations with a literature review, as there often is plenty of scientific evidence around the issues that we are tackling. Such a review allows us to refine our theory of change, avoid false assumptions, and also to save time and effort. An adequate understanding of the natural system, as well as the human system, is needed to be able to identify the environmental impacts of the intervention. An individual evaluator can of course not be an expert in all fields, so it can be very useful to team up with colleagues with diverse backgrounds.

Top Tip 4. Think about our interconnected world, and implore others to do the same. These approaches go beyond how evaluations are often conducted and can be challenging. It is however necessary to broaden our vistas to make a meaningful contribution to solving the challenges for a more sustainable, inclusive and environmentally sound future. As evaluators it is incumbent upon us to also advise the users and commissioners of evaluation that they need to allow evaluation to explore the broader connections of interventions in this complex world.

After all, we all want to make a difference for the better, and done right, evaluation can be a powerful tool to inform policy and decision making for sustainable development in this rapidly changing world.