How many program
evaluation reports are simply placed on a shelf, never to
be read or used to shape change?
Too many, perhaps - but that doesn't have
to always be the case.
Evaluations can evolve as learning occurs;
they do not have to take place in situations where goals and
outcomes are pre-set. Developmental evaluation does not replace
other forms of evaluation. It seems best suited for initiatives
that are at an initial stage of development or undergoing
significant change, and can benefit from careful tracking.
 Michael
Quinn Patton is an independent organizational development
consultant and has written five major books on the art and
science of program evaluation, including the influential "Utilization-Focused
Evaluation," first published in 1978, in which he emphasized
the importance of designing evaluations to insure their usefulness,
rather than simply creating long reports that may never get
read or never result in any practical changes.
Michael's most recent book, "Getting to Maybe: How the
World is Changed," a collaboration with Frances Westley
and Brenda Zimmerman, was published in 2006.
Back to top.
Traditional evaluation is often about finding
a model that can work across time and space, that can be replicated.
The most wide-spread models of traditional evaluation include
summative and formative evaluations.
Summative evaluations are usually done at
the end of a program to assess whether the model that was
proposed for the program worked. Formative evaluations are
done at the beginning of a program while you're working out
the bugs and improving the model. You do a formative evaluation
in to order get ready for summative evaluation -- both assume
that you can work out the best model, then proceed to implement
it.
In developmental evaluation, you expect the
world to be dynamic, so that you will never arrive at a summative
state. In innovative processes, participants may find they
never get to a model for very long, or in a meaningful way.
Using a summative approach during a developmental stage of
an innovative process may do harm by imposing a static data
collection model on a very dynamic adaptive process. Looking
for a recipe may force your community and the people in it
to fit your model and your prescription - contrast this to
the metaphor of raising a child - where there is no recipe,
but you try different things until you find what works.
Developmental evaluation fits when a group
needs ways to get periodic feedback, reflect on it, then act
on it. The Stacey Matrix can be helpful to help a group determine
if you need a developmental evaluation approach. The matrix
measures the degree of certainty about the efficacy of an
intervention and the amount of agreement there is within a
group.
Back to top.
Sometimes there's the perception that evaluation
works against the creative thinking that innovation needs.
However, innovators need a strong commitment to test their
ideas against empirical data, not just against the hope and
vision that lies behind them. Developmental evaluation provides
a way to do that while valuing and documenting the creativity
and learning.
Developmental evaluation tests the passion
against reality - not to judge if it worked, but to pull out
the lessons and decide how to act on them. The tools might
be the same between kinds of evaluation, but the spirit in
which they are employed is different.
Back to top.
In developmental evaluation, the outcomes
are a map of the forks in the road - why the small decisions
were made that led to the eventual result. Developmental evaluation
documents the decisions and formalizes the learnings and the
knowledge bases that drove the decisions. Reporting what you've
learned can be a form of accountability - you can document
what you know now, that you didn't know before.
A developmental evaluation can sometimes
provide a compromise. If you can't agree where you need to
go, you can test different ideas, then come back and decide
on the next action.
Back to top.
If there is consensus about the longitudinal
research behind your issue, you don't need to recreate that
research in your evaluation. But you may need to generate
your own primary data about what works in your own context.
The criteria for data in a developmental evaluation needs
to be practical and usable. It has a different timeline -
it answers the question -- what do I need to know now, to
take the next action.
Developmental evaluation is an effort to
pull together data about the overall strategy,. Action research
looking for data on a specific issue or problem might be part
of it, but the key is to link the specific data to the big
picture.
Back to top.
In developmental evaluation, the evaluator
has to be present as the program unrolls to play a facilitative
role - to bring a full range of options to the group and help
them think about what kind and level of data will be helpful
to them. The evaluator also has a crucial role to play in
facilitating the interpretation of findings, so that they
are applied as the program proceeds. Research on use shows
that you can't wait till you have results - use of the data
has to be considered at the design stage.
Back to top.
Developmental
Evaluation - Check out the Developmental Evaluation Primer by Jamie Gamble, developed with the support of the J.W. McConnell Family Foundation, for a downloadable introduction to developmental evaluation.
The
Stacey Matrix - Michael referred to this helpful matrix, used to identify management decisions
on two dimensions: the degree of certainty and
the level of agreement.
An
Interview with The Evaluation Exchange - Michael was interviewed in Spring 2002 by the Harvard Family Research Project's evaluation periodical, The Evaluation Exchange. They asked him four questions about historical and emerging trends in evaluation practice.
Listen to our interviews with Michael's co-authors
of Getting to Maybe
Back to
top. |
CLICK
TO DOWNLOAD AUDIO!