Blue Marble Evaluation is a call to
action for all evaluators (and others) concerned with the state of
planetary affairs in the new geological era of Anthropocene in which we live.
What defines Anthropocene is that human impact on the Earth is so pervasive as
to be the dominant force in modifying biosphere and atmosphere. Patton brings a
palpable sense of urgency to the task of transforming evaluation to deal with
transformative change. As an evaluator, Patton weighs the global challenges we
are facing—including climate change and related phenomena, growing
concentration of wealth and inequality, virulent infectious diseases and
evolving super-viruses (yes, the book was written before the emergence of
COVID-19), pollution, terrorism, refugee crises, etc.—against the positive
trends (such as reductions in poverty and illiteracy, hunger and
violence).
Striving toward a balanced
assessment, he concludes that the “evidence points overwhelmingly … to a severe
and growing crisis” (p. 19).
Blue
Marble Evaluation takes its name from the iconic photograph taken by the Apollo
17 astronauts on 7 December 1972, the first photograph of our home planet taken
from space. This signifies the view of the whole Earth, which is fundamental to
the Blue Marble Evaluation vision. Patton, a veteran evaluator and one of the
most respected thinkers in the field, elaborates on the premises and principles
of his vision for a renewed evaluation profession that is better able to
respond to the global challenges. In the process he provides a biting critique
of the narrow project mindset that dominates the evaluation practice today.
The
book is organized in three parts. The first, The Blue Marble Perspective,
presents four overarching blue marble principles: (1) Global thinking
principle, (2) Anthropocene as context principle, (3) Transformative engagement
principle, and (4) Integration principle. “The Blue Marble worldview
constitutes a paradigm,” Patton writes (p. 6). Taking a holistic perspective
and understanding global patterns and their implications is essential, as is
thinking about all aspects of systems change at all levels from local to
global. The integration principle emphasizes that transformation requires
multiple interventions and actions on many fronts by multiple actors. Blue
Marble principles should not only be applied to evaluation but also to design
and implementation. Bringing the four initial principles together, Patton takes
apart traditional project and program evaluation, as well as performance
measurement and monitoring, as insufficient to addressing systems change at the
global scale. He writes (p. 30):
“Static
and rigid randomized control designs—emphasis on control—are irrelevant to the
uncontrollable dynamics of complex systems. Indeed, these traditional
approaches to evaluation can create barriers to systems change by forcing
transformational visions into narrow project boxes amenable to methods
evaluators are comfortable with (e.g., logic models and specific, measurable,
achievable, relevant, time-bound [SMART] goals). Innovations in evaluation
include eclectic approaches created by tapping into a vast array of
many-splendored, diverse, and innovative knowledge-generating and
learning-oriented processes.”
Part
II adds eight Operating Principles: (5) Transboundary engagement principle, (6)
GLOCAL principle, (7) Cross-silos principle, (8) Time being of essence
principle, (9) Yin-yang principle, (10) Bricolage methods principle, (11) World
savvy principle, and (12) Skin in the game principle. Ranging from page 39 to
148, this section contains the bulk of the book. I won’t attempt to summarize
it but will highlight a few points that I found essential. Patton discusses the
need to think and act at the global scale and what this means. He provides a
thoughtful critique of the Sustainable Development Goals (SDGs) adopted by over
190 countries all over the world. He rightly points out that they are actually
not global, given that they are based on country-level targets. He wisely notes
that “simply aggregating nation-state data doesn’t generate a global picture”
(p. 44). Another important insight that flies in the face of conventional
evaluation (and development) thinking—and with which I couldn’t agree more—is
that there are no “best practices” because the context in which any
intervention takes place is so important. “Global context informs local
actions. Local contexts make global understandings meaningful and actionable”
(p. 50). This up and down engagement between levels is the essence of the
GLOCAL principle. Patton discusses scaling up as the most common way of having
global impact, i.e. that successful initiatives are scaled out, scaled up or
scaled deep so that they can reach more people and places. Scaling up is also at
the heart of the operational model of the Global Environment Facility (GEF)
where I work and just last year my office completed an
evaluation
of scaling up at the GEF.
Apart
from integrating across scales and geographies, Blue Marble principles call for
integrating across silos. It is worth noting, like Patton does, that the SDGs,
while intended to be integrated, also easily lend themselves to new silos. Yet
in real life, the SDGs are indeed all interconnected and “evaluating the
interactions between the SDG targets and indicators, both actual and
aspirational, positive and negative, and short term and long term, offers
significant opportunities for Blue Marble thinkers, designers and evaluators to
contribute to the 2030 Agenda” (p. 65). (In full transparency, in the
discussion on the cross-silos principle, I was very flattered to find Patton
citing my work.)
With
the time being of essence, Patton addresses another key notion in evaluation
and development professions: sustainability. According to the established
principles promoted by OECD’s Development Assistance Committee, sustainability
refers primarily to the continuation of project benefits after the completion
of the intervention. Although, OECD/DAC has recently updated the definition to
incorporate more of the environmental sustainability, there has been a lively
debate about a stronger articulation of the environmental dimension. Patton’s
discussion is most welcome as he elaborates on adaptive sustainability and
resilience, making a clear distinction between engineering resilience (a
performance-related notion that focuses on stability, efficiency, control and
predictability) and ecosystem resilience that focuses on persistence,
adaptiveness, variability and unpredictability. Aiming for long-term resilience
and sustainability forces us to concentrate on the latter, in which case
evaluators, too, must focus on adaptability of the system, rather than a static
endpoint after an intervention ends and the benefits remain (pp. 78-82). Patton
illustrates this with concrete examples, including some in which short-term
focus on accountability undermined long-term sustainability.
The
chapter on the bricolage methods principle describing the evaluator as a bricoleur, a traditional French traveling
“jack-of-all-trades” who would use whatever tools were at hand to fix a
problem, hit a note with me. Similarly, evaluators have to use an eclectic
variety of methods to fit the evaluation situation that they need to address.
The Canadian evaluator Andy Rowe has similarly made the call for All Hands on
Deck for evaluators to put their methods wars behind them and join hands in
combating the greatest challenges facing humankind. Claims of “gold standards”
(often made by those promoting experimental methods) sound ludicrous. To be
clear, Patton is no Luddite: he fully acknowledges the value of new methods
that involve, i.a., geospatial tools, big data and AI, but emphasizes the
importance of choosing the most appropriate methods from an eclectic toolkit. Here
he elaborates on six “bricolaged Blue Marble evaluation methods lessons” (pp.
114-116).
The
chapters on the world savvy principle and skin in the game principle are highly
personal. The former emphasizes the global competencies that all evaluators
should possess. He notes that the “profession of evaluation is fairly obsessed
with competency” (p. 122) but that the concept of competence is problematic and
can lead to “exclusion, reinforcing the status quo and power of the status quo”
(p. 124). These again are concerns I can easily relate to, having been involved
in evaluation groups that sometimes feel like aiming to be guilds with their
exclusive approaches and membership. Without getting into detail about what
being world savvy means, some of the central concepts include reflexivity and
ongoing learning.
The
skin in the game principle will be hard for some evaluators to swallow (and
I’ve already witnessed discussions to this end), given that evaluators (most of
whom are social scientists by training) have been indoctrinated to think that
we need to be neutral observers of the object of evaluation, looking from the outside
in, without taking sides or showing emotion. Patton challenges this by making
it clear that we should not hide our values, especially in the current world
situation where our common future is at stake. Independence is usually seen as
one of the most important characteristics of an evaluator, but in some cases
total independence can even undermine the credibility of the evaluator. This
might be the case with indigenous communities where a total outsider would not
carry any credence. The evaluator would still use her or his best judgment
weighing the evidence (for example not assuming that the local communities
would necessarily always know best). He makes a clear and useful distinction
between caring and bias (pp. 139-141). His argumentation stands on the broad
shoulders of two evaluation thought leaders, Robert Stake and Michael Scriven,
as well as others. He also makes a strong case for evaluation as
transdisciplinary science, and for science as intervention aimed at generating
knowledge and solving an urgent problem.
The
final part of the book outlines three Global Systems Transformation principles:
(13) Theory of transformation principle, (14) Transformation fidelity
principle: evaluating transformation, and (15) Transformational alignment
principle: transforming evaluation to evaluate transformation. The first of
these chapters is the most theoretical in the book. In it Patton attempts to
move from a theory of change to a theory of transformation, doing so by
integrating multiple theories to explain transformation. The theoretical
frameworks he most relies on are network theory and innovation theory. He also
expounds on a hypothesis developed by Jerald Hage, director of the Center of
Innovation at the University of Maryland’s Department of Sociology who also was
Patton’s dissertation advisor at the University of Wisconsin decades earlier.
Altogether, the theory of transformation that Patton comes up with is a
plausible one. It claims that transformation flows from an “understanding that
the status quo is not a viable path forward and that networked action on
multiple fronts using diverse change strategies across multiple landscapes will
be needed to overcome the resistance from those who benefit from the status
quo” (p. 168). This will lead to critical mass tipping points and consequently
transformations. Earlier in the chapter Patton reminds us of the old truth found
in innovation theory that it is futile to try to convince (and despair over)
the often powerful interests that benefit from the status quo and resist change;
instead, focus on supporting the early adopters and spreading their message
until a tipping point is reached. This is a very useful message to keep in mind
when hopelessness sets in ahead of the Sisyphean task of moving towards a more environmentally
benign future.
In
the chapter on transformation fidelity principle, Patton starts by reminding us
that transformation has become a new buzzword but not everything touted as such
is in fact transformational. Of true transformations in relatively recent
history he names three examples: the end of apartheid in South Africa, the fall
of the Berlin Wall, and the significant fall in new cases of HIV/AIDS.
Transformation is difficult to define, but easy to recognize when it happens by
(borrowing the term from the statistician Fred Mosteller) interocular
significance (i.e., it hits you in between the eyes). Transformation can be
defined as a sensitizing concept, “a way of talking about something that is not
yet well understood, precisely defined, or operationally measured” (p. 174). Evaluating
transformational change, however, is more challenging. Patton sets out to
define an evaluation framework for evaluating transformation with the theory of
transformation that he developed in the previous chapter. He reviews an
influential evaluation of transformational engagement by the World Bank’s
Independent Evaluation Group (using an approach that we adapted to
evaluate
transformational change at the GEF Independent Evaluation Office a few
years back). While this approach was fine, it was, according to Patton, a
definition-based, rather than theory-based evaluation. Interestingly, Patton
concludes that “evaluating transformation must involve capturing the story,
communicating the process and results, interpreting meanings, making
values-based judgments about what occurred, extracting lessons, and
facilitating visioning the way forward both short term ad long term. Telling
the transformation story will involve mixed methods” (p. 185).
In
the final chapter, Patton lays out the need to transform evaluation itself to
evaluate transformation. This chapter also acts as a summary of the book and
his thinking around the Blue Marble principles. He concludes that global
sustainability should become a universal criterion in evaluation. The book ends
with a vision of a global network of Blue Marble evaluators, a vision that is in
the making to become reality.
Blue
Marble Evaluation is a powerful book that transmits the sense of urgency, the
very caring for the planet and the state of affairs that Michael Quinn Patton
writes about. He emphasizes that while he has pulled the text together, it has
been a collaborative effort with contributions from many people, including
fellow Blue Marble evaluators, like Pablo Vidueira and Glenn Page (the
acknowledgements run over 3.5 pages). The book is also amply illustrated,
including with cartoons by Mark Rogers, Chris Lysy, Simon Kneebone and Claudius
Ceccon. All of this collaboration is further proof that Michael Patton
practices what he preaches. The book is also very erudite, drawing upon literature
and theories of science, philosophy, sociology, astronomy and more in addition
to evaluation. There are many practical examples from interventions and
evaluations of different kinds, some more profound than others. All of this
makes for interesting, if occasionally rambling reading as the reader is left
guessing how it all comes back to the theme (spoiler alert: it usually does). And
as Patton states, this is not a book about methods, but evaluators tend to be
methods people so he writes quite much about methods. All of this may make the
book quite a ride for a traditional evaluator versed in project performance
evaluation. Those are some of the people who most should read this book. Having
said that, I would recommend the book to anyone interested in evaluation and
applied social science in the age of the Anthropocene.
