Blue Marble Evaluation: Premises and Principles, by Michael Quinn Patton

Blue Marble Evaluation is a call to action for all evaluators (and others) concerned with the state of planetary affairs in the new geological era of Anthropocene in which we live. What defines Anthropocene is that human impact on the Earth is so pervasive as to be the dominant force in modifying biosphere and atmosphere. Patton brings a palpable sense of urgency to the task of transforming evaluation to deal with transformative change. As an evaluator, Patton weighs the global challenges we are facing—including climate change and related phenomena, growing concentration of wealth and inequality, virulent infectious diseases and evolving super-viruses (yes, the book was written before the emergence of COVID-19), pollution, terrorism, refugee crises, etc.—against the positive trends (such as reductions in poverty and illiteracy, hunger and violence).  Striving toward a balanced assessment, he concludes that the “evidence points overwhelmingly … to a severe and growing crisis” (p. 19). 

Blue Marble Evaluation takes its name from the iconic photograph taken by the Apollo 17 astronauts on 7 December 1972, the first photograph of our home planet taken from space. This signifies the view of the whole Earth, which is fundamental to the Blue Marble Evaluation vision. Patton, a veteran evaluator and one of the most respected thinkers in the field, elaborates on the premises and principles of his vision for a renewed evaluation profession that is better able to respond to the global challenges. In the process he provides a biting critique of the narrow project mindset that dominates the evaluation practice today.

The book is organized in three parts. The first, The Blue Marble Perspective, presents four overarching blue marble principles: (1) Global thinking principle, (2) Anthropocene as context principle, (3) Transformative engagement principle, and (4) Integration principle. “The Blue Marble worldview constitutes a paradigm,” Patton writes (p. 6). Taking a holistic perspective and understanding global patterns and their implications is essential, as is thinking about all aspects of systems change at all levels from local to global. The integration principle emphasizes that transformation requires multiple interventions and actions on many fronts by multiple actors. Blue Marble principles should not only be applied to evaluation but also to design and implementation. Bringing the four initial principles together, Patton takes apart traditional project and program evaluation, as well as performance measurement and monitoring, as insufficient to addressing systems change at the global scale. He writes (p. 30): 

“Static and rigid randomized control designs—emphasis on control—are irrelevant to the uncontrollable dynamics of complex systems. Indeed, these traditional approaches to evaluation can create barriers to systems change by forcing transformational visions into narrow project boxes amenable to methods evaluators are comfortable with (e.g., logic models and specific, measurable, achievable, relevant, time-bound [SMART] goals). Innovations in evaluation include eclectic approaches created by tapping into a vast array of many-splendored, diverse, and innovative knowledge-generating and learning-oriented processes.”

Part II adds eight Operating Principles: (5) Transboundary engagement principle, (6) GLOCAL principle, (7) Cross-silos principle, (8) Time being of essence principle, (9) Yin-yang principle, (10) Bricolage methods principle, (11) World savvy principle, and (12) Skin in the game principle. Ranging from page 39 to 148, this section contains the bulk of the book. I won’t attempt to summarize it but will highlight a few points that I found essential. Patton discusses the need to think and act at the global scale and what this means. He provides a thoughtful critique of the Sustainable Development Goals (SDGs) adopted by over 190 countries all over the world. He rightly points out that they are actually not global, given that they are based on country-level targets. He wisely notes that “simply aggregating nation-state data doesn’t generate a global picture” (p. 44). Another important insight that flies in the face of conventional evaluation (and development) thinking—and with which I couldn’t agree more—is that there are no “best practices” because the context in which any intervention takes place is so important. “Global context informs local actions. Local contexts make global understandings meaningful and actionable” (p. 50). This up and down engagement between levels is the essence of the GLOCAL principle. Patton discusses scaling up as the most common way of having global impact, i.e. that successful initiatives are scaled out, scaled up or scaled deep so that they can reach more people and places. Scaling up is also at the heart of the operational model of the Global Environment Facility (GEF) where I work and just last year my office completed an evaluation of scaling up at the GEF. 

Apart from integrating across scales and geographies, Blue Marble principles call for integrating across silos. It is worth noting, like Patton does, that the SDGs, while intended to be integrated, also easily lend themselves to new silos. Yet in real life, the SDGs are indeed all interconnected and “evaluating the interactions between the SDG targets and indicators, both actual and aspirational, positive and negative, and short term and long term, offers significant opportunities for Blue Marble thinkers, designers and evaluators to contribute to the 2030 Agenda” (p. 65). (In full transparency, in the discussion on the cross-silos principle, I was very flattered to find Patton citing my work.) 

With the time being of essence, Patton addresses another key notion in evaluation and development professions: sustainability. According to the established principles promoted by OECD’s Development Assistance Committee, sustainability refers primarily to the continuation of project benefits after the completion of the intervention. Although, OECD/DAC has recently updated the definition to incorporate more of the environmental sustainability, there has been a lively debate about a stronger articulation of the environmental dimension. Patton’s discussion is most welcome as he elaborates on adaptive sustainability and resilience, making a clear distinction between engineering resilience (a performance-related notion that focuses on stability, efficiency, control and predictability) and ecosystem resilience that focuses on persistence, adaptiveness, variability and unpredictability. Aiming for long-term resilience and sustainability forces us to concentrate on the latter, in which case evaluators, too, must focus on adaptability of the system, rather than a static endpoint after an intervention ends and the benefits remain (pp. 78-82). Patton illustrates this with concrete examples, including some in which short-term focus on accountability undermined long-term sustainability. 

The chapter on the bricolage methods principle describing the evaluator as a bricoleur, a traditional French traveling “jack-of-all-trades” who would use whatever tools were at hand to fix a problem, hit a note with me. Similarly, evaluators have to use an eclectic variety of methods to fit the evaluation situation that they need to address. The Canadian evaluator Andy Rowe has similarly made the call for All Hands on Deck for evaluators to put their methods wars behind them and join hands in combating the greatest challenges facing humankind. Claims of “gold standards” (often made by those promoting experimental methods) sound ludicrous. To be clear, Patton is no Luddite: he fully acknowledges the value of new methods that involve, i.a., geospatial tools, big data and AI, but emphasizes the importance of choosing the most appropriate methods from an eclectic toolkit. Here he elaborates on six “bricolaged Blue Marble evaluation methods lessons” (pp. 114-116).  

The chapters on the world savvy principle and skin in the game principle are highly personal. The former emphasizes the global competencies that all evaluators should possess. He notes that the “profession of evaluation is fairly obsessed with competency” (p. 122) but that the concept of competence is problematic and can lead to “exclusion, reinforcing the status quo and power of the status quo” (p. 124). These again are concerns I can easily relate to, having been involved in evaluation groups that sometimes feel like aiming to be guilds with their exclusive approaches and membership. Without getting into detail about what being world savvy means, some of the central concepts include reflexivity and ongoing learning. 

The skin in the game principle will be hard for some evaluators to swallow (and I’ve already witnessed discussions to this end), given that evaluators (most of whom are social scientists by training) have been indoctrinated to think that we need to be neutral observers of the object of evaluation, looking from the outside in, without taking sides or showing emotion. Patton challenges this by making it clear that we should not hide our values, especially in the current world situation where our common future is at stake. Independence is usually seen as one of the most important characteristics of an evaluator, but in some cases total independence can even undermine the credibility of the evaluator. This might be the case with indigenous communities where a total outsider would not carry any credence. The evaluator would still use her or his best judgment weighing the evidence (for example not assuming that the local communities would necessarily always know best). He makes a clear and useful distinction between caring and bias (pp. 139-141). His argumentation stands on the broad shoulders of two evaluation thought leaders, Robert Stake and Michael Scriven, as well as others. He also makes a strong case for evaluation as transdisciplinary science, and for science as intervention aimed at generating knowledge and solving an urgent problem. 

The final part of the book outlines three Global Systems Transformation principles: (13) Theory of transformation principle, (14) Transformation fidelity principle: evaluating transformation, and (15) Transformational alignment principle: transforming evaluation to evaluate transformation. The first of these chapters is the most theoretical in the book. In it Patton attempts to move from a theory of change to a theory of transformation, doing so by integrating multiple theories to explain transformation. The theoretical frameworks he most relies on are network theory and innovation theory. He also expounds on a hypothesis developed by Jerald Hage, director of the Center of Innovation at the University of Maryland’s Department of Sociology who also was Patton’s dissertation advisor at the University of Wisconsin decades earlier. Altogether, the theory of transformation that Patton comes up with is a plausible one. It claims that transformation flows from an “understanding that the status quo is not a viable path forward and that networked action on multiple fronts using diverse change strategies across multiple landscapes will be needed to overcome the resistance from those who benefit from the status quo” (p. 168). This will lead to critical mass tipping points and consequently transformations. Earlier in the chapter Patton reminds us of the old truth found in innovation theory that it is futile to try to convince (and despair over) the often powerful interests that benefit from the status quo and resist change; instead, focus on supporting the early adopters and spreading their message until a tipping point is reached. This is a very useful message to keep in mind when hopelessness sets in ahead of the Sisyphean task of moving towards a more environmentally benign future.

In the chapter on transformation fidelity principle, Patton starts by reminding us that transformation has become a new buzzword but not everything touted as such is in fact transformational. Of true transformations in relatively recent history he names three examples: the end of apartheid in South Africa, the fall of the Berlin Wall, and the significant fall in new cases of HIV/AIDS. Transformation is difficult to define, but easy to recognize when it happens by (borrowing the term from the statistician Fred Mosteller) interocular significance (i.e., it hits you in between the eyes). Transformation can be defined as a sensitizing concept, “a way of talking about something that is not yet well understood, precisely defined, or operationally measured” (p. 174). Evaluating transformational change, however, is more challenging. Patton sets out to define an evaluation framework for evaluating transformation with the theory of transformation that he developed in the previous chapter. He reviews an influential evaluation of transformational engagement by the World Bank’s Independent Evaluation Group (using an approach that we adapted to evaluate transformational change at the GEF Independent Evaluation Office a few years back). While this approach was fine, it was, according to Patton, a definition-based, rather than theory-based evaluation. Interestingly, Patton concludes that “evaluating transformation must involve capturing the story, communicating the process and results, interpreting meanings, making values-based judgments about what occurred, extracting lessons, and facilitating visioning the way forward both short term ad long term. Telling the transformation story will involve mixed methods” (p. 185).

In the final chapter, Patton lays out the need to transform evaluation itself to evaluate transformation. This chapter also acts as a summary of the book and his thinking around the Blue Marble principles. He concludes that global sustainability should become a universal criterion in evaluation. The book ends with a vision of a global network of Blue Marble evaluators, a vision that is in the making to become reality.

Blue Marble Evaluation is a powerful book that transmits the sense of urgency, the very caring for the planet and the state of affairs that Michael Quinn Patton writes about. He emphasizes that while he has pulled the text together, it has been a collaborative effort with contributions from many people, including fellow Blue Marble evaluators, like Pablo Vidueira and Glenn Page (the acknowledgements run over 3.5 pages). The book is also amply illustrated, including with cartoons by Mark Rogers, Chris Lysy, Simon Kneebone and Claudius Ceccon. All of this collaboration is further proof that Michael Patton practices what he preaches. The book is also very erudite, drawing upon literature and theories of science, philosophy, sociology, astronomy and more in addition to evaluation. There are many practical examples from interventions and evaluations of different kinds, some more profound than others. All of this makes for interesting, if occasionally rambling reading as the reader is left guessing how it all comes back to the theme (spoiler alert: it usually does). And as Patton states, this is not a book about methods, but evaluators tend to be methods people so he writes quite much about methods. All of this may make the book quite a ride for a traditional evaluator versed in project performance evaluation. Those are some of the people who most should read this book. Having said that, I would recommend the book to anyone interested in evaluation and applied social science in the age of the Anthropocene.