In this session, Mark Cabaj speaks with Michael Quinn Patton about his new book on the topic of Developmental Evaluation – an approach that has proven particularly useful and effective at addressing the unique challenges of evaluating the real nature of community change work.
To deepen appreciation for the role of evaluation and evaluation thinking
To explore the distinctions between various types of evaluation
To understand more about developmental evaluation and its use
To investigate concepts, tools and resources available to support working with developmental evaluation methods
Michael Quinn Patton - Michael is an independent organizational development consultant who has worked with organizations and programs at the international, national, state, and local levels, and with philanthropic, not-for-profit, private sector, and government programs. He is a generalist whose aim is to improve human effectiveness and results. His work spans a full range of programs including: leadership development, education, human services, the environment, public health, employment, agricultural extension, arts, criminal justice, poverty, transportation, diversity, managing for results, performance indicators, effective governance, and futuring.
Michael has a doctorate in Sociology from the University of Wisconsin and spent 18 years on the faculty of the University of Minnesota. He is the only recipient of both the Alva and Gunner Myrdal Award from the Evaluation Research Society for "outstanding contributions to evaluation use and practice" and the Paul F. Lazarsfeld Award for lifetime contributions to evaluation theory from the American Evaluation Association.
Michael has authored six evaluation books including a 4th edition of Utilization-Focused Evaluation and co-authored Getting to Maybe: How the World Is Changed which applies complexity theory to social innovation and presents developmental evaluation as an approach for evaluating innovations. Michael’s latest book, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use was released in June 2010.
Mark Cabaj - Mark Cabaj is a founding Principal of Tamarack – An Institute for Community Engagement, an organization based in Waterloo, Ontario, focused on assisting people build strong communities through local action.
Mark joined Tamarack in 2002 and is currently the Executive Director of Vibrant Communities – a network of communities and national organizations that use collaborative, comprehensive approaches to substantially reduce poverty within Canada.
For some, the words “evaluation” or “evaluation thinking” generate negative images of something that we’d prefer to avoid. However, fundamentally evaluation is about increasing the effectiveness of change efforts of all kinds. Evaluation thinking involves a commitment to ask ourselves questions such as: are we doing what we wanted to be doing? Are we accomplishing what we wanted to accomplish?
While there is value in conducting evaluations to satisfy the needs and expectation of others, Michael believes that the highest form of accountability is to oneself – as someone involved in a change effort – to engage in considering whether or not your work is unfolding as you thought and hoped that it might. Ultimately, evaluation is a leadership function that involves being systematic about making developmental changes to the implementation of a plan based on feedback. Listen here as Michael offers his definition of evaluation and evaluation thinking.
Developmental evaluation is outcomes-oriented in its focus and is particularly useful in situations where the outcomes are emergent and changing. It emerged as a niche for use in situations of ongoing development. It is particularly well-suited for helping to monitor the connections between short-term outcomes and longer-term social change efforts. Programs that do not expect to create a standardized model, but rather anticipate the need to be continuously evolving are examples where Developmental Evaluation is particularly useful. The emphasis in Developmental Evaluation is on documenting decisions and formalizing the learning and the knowledge-bases that drive decisions. In this clip, Michael shares the story of how he coined the term developmental evaluation in response to a client’s frustration with the limitations of both formative and summative evaluation methods.
Historically, there have been two primary types of evaluation: Formative Evaluation and Summative Evaluation. Formative evaluation is used to help improve a program or policy. Summative evaluation is used to assess the merit or worth of a program or policy in order to determine whether it should be sustained, discontinued or scaled up. However, both formative and summative evaluations are quite rigid. They are also most effective when work is finite or can be paused while an evaluation is conducted. Listen here as Michael uses the metaphor of “making soup” to describe the distinctions between formative, summative and developmental evaluation.
The Value of Evaluation in Developmental Situations
Developmental situations are ones that require a willingness to move beyond making slight improvements and instead continually adapt or change in response to the current environment. Using this “lens”, one can begin to identify many developmental situations where evaluative thinking could be extremely useful. These include situations where people are innovating; people are trying out a new approach where no model exists; or, people do not anticipate their work ever arriving at a fixed model.
Developmental Evaluation works at the intersection between “top-down” models of social change which emphasize generalized knowledge about how change should occur and “bottom up” models which engage people in generating their own solution to problems. It adds value by helping to track the development of an innovation over time. Rather than being something apart, Developmental Evaluation is embedded into the innovation process, tracking changes over time and bringing some regular “reality-testing” to the innovation process. In the clip below, Michael summarizes Developmental Evaluation’s value-added role.
Evaluators are primarily trained to assume that all situations are simple and can be predicted at the outset. However, in real life, programs can be operating on multiple tracks simultaneously. The term “Patch Evaluation” is used to describe these types of situations: where evaluative thinking is being used simultaneously in different ways according to the different stages or types of work that is being undertaken. A critical beginning step in any large program or community effort is to analyze and determine which aspects of work are simple, which are complicated and which are complex and design an approach to evaluation that responds to these different needs. Listen to the clip below as Michael describes this process.
Concepts & Tools for Working with Developmental Evaluation
One of the most commonly asked questions about developmental evaluation is: what methods or tools are best used when working in developmental situations? Michael’s simple answer is, “You can use any evaluative method in a developmental context.” Listen here as Michael elaborates on this point.
Eva the Evaluator – This children’s book is written by Roger Miranda – the recently retired director of evaluation at the United Nations. It provides a clear and simple explanation of the role of an evaluator.