Often evaluation processes are static and focus on end of project results. Instead, Vibrant Communities is asking key stakeholders about the questions that should be asked, the format of the end product and how, as a participant in the evaluation process, they might use the evaluation results to inform their work. This process has increased engagement in the evaluation design and development and potentially in how end-users will engage with results.
Learning Objective:
To become acquainted with the concepts of utilization-focused evaluation using Vibrant Communities as a case study.
Jamie Gamble - Jamie Gamble is the owner and principal of the New Brunswick based consulting company, Imprint Consulting Inc. He holds a Masters of Management Degree from McGill University. Through his work he is active in fostering innovation and development in a wide range of issues including poverty reduction, environmental protection, food security, youth leadership, and the arts. Jamie has extensive international experience and has worked in India, New Zealand and Mexico, and all across Canada. In 2008 the J.W. McConnell Family Foundation published Jamie’s Developmental Evaluation Primer.
Mark Cabaj - Mark Cabaj is a founding Principal of Tamarack – An Institute for Community Engagement, an organization based in Waterloo, Ontario, focused on assisting people build strong communities through local action.
Mark joined Tamarack in 2002 and is currently the Executive Director of Vibrant Communities – a network of communities and national organizations that use collaborative, comprehensive approaches to substantially reduce poverty within Canada.
Mark’s current focus is on developing approaches and techniques for understanding, planning and evaluating initiatives that address complex issues, such as neighbourhood renewal, poverty and homelessness, community safety, educational achievement and health. He is particularly involved in expanding the ideas and practice of developmental evaluation, a new approach to evaluation which places a heavy emphasis on learning and design in emerging and sometimes fast moving environments. Read more here.
Almost everyone has experienced an evaluation sitting on a shelf, gathering dust. The history of evaluation shows that “use” is a major challenge. Research spanning decades shows that evaluations are rarely used and rarely influence decisions. So, why measure or evaluate if no one cares?
Mark explained that Vibrant Communities engaged Jamie to try to avoid this problem - to do an evaluation in a way that those involved would actually use the results - that the findings would influence current decisions and future work.
Vibrant Communities started in 2002 and will end in 2011. Twelve initiatives across Canada have focused on reducing poverty using the following principles:
Poverty reduction versus poverty alleviation
Multi-sectoral collaboration
Comprehensive thinking and action
Building on local assets
An ongoing process of learning and change
Three national sponsors provide additional support:
J.W. McConnell Family Foundation provides matching grants
Caledon Institute provides policy writing and research
Tamarack provides cross community learning and coaching
In this clip, Jamie describes the key goals and characteristics of Vibrant Communities, then gives an summary of the outcomes, the dollars leveraged, and the learning generated by the initiative so far.
Jamie gave credit to Michael Quinn Patton and his book, Utilization-Focused Evaluation, for the best description of evaluations that focus on users’ needs.
Jamie emphasized that evaluations have to be useful; real-world application is the goal. Everything that is done, from beginning to end, affects use of the evaluation. Users need to be part of the design as well as part of the journey.
In this clip, Mark and Jamie discuss the decision-making styles of different U.S. presidents as a way of illustrating the different needs of evaluation users. Jamie makes the distinction between the form of an evaluation and the function - the purpose for which the evaluation will be used.
Jamie outlined the principles that are guiding the evaluation of Vibrant Communities:
Shaped by Primary Intended Users
The primary users include local communities, and funders and policy makers interested in place-based efforts to reduce poverty.
The secondary users include the current staff and sponsors of Vibrant Communities.
Participatory Sense-Making
Where and when possible, intended users will participate in efforts to interpret data, draw conclusions and make judgments.
Two Phases
The first phase (December ’09-May 2010) will use existing data or ‘easier to get’ sources.
A second round (June 2010 to early 2011) will address new questions generated through phase one and will try to address questions which are unanswerable with existing data. This phase will likely involve collecting additional data.
A fourth principle has emerged from the process:
Adaptive Design
The methods for gathering, analyzing, interpreting and judging data will evolve to reflect emerging questions, the preferences and needs of stakeholders and the requirements of good research.
Here, Jamie speaks about three groups that are the primary users of the Vibrant Communities evaluation, and how each group wants to use the evaluation.
Jamie has seen some advantages emerging from the process so far:
“Beta” testing methods and processes early in the evaluation
Using an inside-outside dynamic (VC staff dedicated to evaluation working with an external evaluator)
Not trying to fully develop things too quickly
Drawing on multiple perspectives for interpretation
Applying multiple lenses and analytical frameworks to the same question
Jamie explains in this clip how “the process is part of the data”. He describes how the evaluation plan and process changed significantly after he and Mark tested it with the steering committee of Vibrant Communities.
Jamie explained that Vibrant Communities conveners, coaches and some staff of sponsoring organizations will meet soon to determine what can be said with the data on hand– and what other data is needed. They also hope to collect ‘easy to get’ data and use it, along with a series of sense-making activities, to draw a preliminary set of conclusions.
The team is considering a “meta-evaluation” - an evaluation of the processes Vibrant Communities has used for nearly eight years to gather data on poverty reduction outcomes and community change, to see what parts are robust and which are not.
Down the road, they may also engage an expert panel to complement their internal sense-making.
To what extent do you understand the challenge of developing and implementing evaluations that are used by their intended users?
What are the challenges and characteristics of doing ‘utilization-focused evaluation’ for a diverse group of stakeholders for a pan-Canadian initiative? What are the implications for designing and evaluating an evaluation process?
How could organizations, networks and collaborations and networks use utilization-focused evaluations?
To reflect on these and other important questions, refer to the Resources and Links below.
Quotes on Evaluation Use - This handout summarizes and gives references to articles and research on use of evaluations, starting from the 1960’s to recent years. Access the handout here.
Utilization-Focused Evaluation (U-FE) Checklist - This checklist, by Michael Quinn Patton, explains utilization-focused evaluation, and offers a list of tasks and the underlying premises that are involved. Access the checklist here.
Developmental Evaluation Primer - The primer, prepared by Jamie Gamble, describes an alternative to formative and summative evaluation. Access the primer here.
Building Knowledge About Community Change - This 2004 resource from the Aspen Institute provides a summary of the challenge of evaluating comprehensive community change initiatives and describes an alternative approach. Access the resource here.
Evaluating Comprehensive Community Change - This 1997 Annie E. Casey Foundation document describes the challenges of evaluating complex change initiatives using traditional approaches. Access the document here.
Measuring Learning: Developmental Evaluation - This audio seminar with Michael Quinn Patton explains developmental evaluation. Utilization-focused evaluation is one form of developmental evaluation. Access the seminar here.
Often evaluation processes are static and focus on end of project results. Instead, Vibrant Communities is asking key stakeholders about the questions that should be asked, the format of the end product and how, as a participant in the evaluation process, they might use the evaluation results to inform their work. This process has increased engagement in the evaluation design and development and potentially in how end-users will engage with results. 
Jamie Gamble - Jamie Gamble is the owner and principal of the New Brunswick based consulting company, Imprint Consulting Inc. He holds a Masters of Management Degree from McGill University. Through his work he is active in fostering innovation and development in a wide range of issues including poverty reduction, environmental protection, food security, youth leadership, and the arts. Jamie has extensive international experience and has worked in India, New Zealand and Mexico, and all across Canada. In 2008 the J.W. McConnell Family Foundation published Jamie’s Developmental Evaluation Primer.
Mark Cabaj - Mark Cabaj is a founding Principal of Tamarack – An Institute for Community Engagement, an organization based in Waterloo, Ontario, focused on assisting people build strong communities through local action.
