Thematic summary:
Values-focussed monitoring and evaluation
1. Executive Summary
Intangible processes and outcomes relating to higher ethical/spiritual values, such as trust, empowerment or respect, are difficult to evaluate. The ESDinds Project, a two-year (2009-2011) EU-funded project bringing together research institutions and civil society organizations from five countries, has created an innovative toolkit for values-focused monitoring and evaluation. The WeValue toolkit consists of a set of 166 values-based indicators, together with guidelines for assessment tool development and data analysis. The project has trained a network of consultants to conduct values-focused evaluations, and a PDF handbook and web platform are also available at www.wevalue.org for evaluators (and CSO staff with research expertise) who wish to utilise the toolkit independently.
2. The need for values-focused M&E
“[Donors] want to see the impact of their gift now, not later. They are looking for accountability. They want evidence of success. They want metrics. And for some charitable projects this is eminently doable. In fact, gauging the effectiveness of some programs is deceptively simple. It’s easy to know how many children in Africa were inoculated by a particular program; or how many mosquito nets were distributed to protect against malaria… On the other hand, the impact of some types of philanthropy is much harder to measure. But the benefits, though more abstract, are no less real.” (Faye Wightman, CEO, Vancouver Foundation, 2010:9)
“Traditional methods of evaluation are often not able to capture or measure the ‘spirit of change’ in people, which is the very essence of human development.” (Crishna, 2007:217).
Civil society organizations (CSOs) include charities, non-governmental organizations (NGOs) and social enterprises. Many CSOs, especially those providing non-formal education, find it difficult to articulate their true impact to donors and the public. This is usually because key outcomes of their work are intangible or abstract in nature. Such outcomes often relate to the ways in which people live out or enact higher ethical/spiritual values such as justice, integrity, trustworthiness, unity in diversity, empowerment, or care and respect for the wider community of life.
Conventional evaluations generally focus, however, on quantitative outputs that are easy to count or measure. This means that donors tend to reward ‘quick-fix’, product-based solutions, sometimes at the expense of process-based approaches generating sustainable behaviour change (Ebrahim, 2003; Edwards & Hulme, 1996).
In an increasingly competitive funding climate, the need for innovative strategies to monitor and evaluate values enactment has never been more acute. The very survival of a CSO may depend on its leaders’ ability to understand, measure and communicate whatever it is that they value most.
3. Building on recent advances in evaluation theory
In establishing a values-focused approach to evaluation, the ESDinds Project and WeValue toolkit have drawn on several recent developments in the field of M&E:
(a) Process evaluation: a recognition that the journey matters as much as the destination, and that in addition to the outcomes it is also important to examine intervention processes (what is done within the project) and implementation processes (how things are done, i.e. how people interact and communicate with one another) (Cox et al, 2007; Ellis & Hogard, 2006; Hogard, 2008).
(b) Participatory evaluation: involving CSO managers, staff and/or beneficiaries in defining what to evaluate, which methods to use, how to collect data and how to understand the results (Crishna, 2007; Daigneault & Jacob, 2009).
(c) Utilization-focused evaluation: using reflection and ‘feedback loops’ to ensure that evaluation results translate into improvements in policy and practice (Flowers, 2010).
(d) Process-use benefits: the recognition that irrespective of its findings, the mere act of participating in an evaluation can confer important benefits for those involved. These include ‘learning how to learn’, creating shared understandings, strengthening the project, boosting morale, and developing professional networks (Forss et al, 2002)
4. Where the indicators came from
The indicator design phase of the ESDinds Project was based on an adaptive learning process combining ‘bottom-up’ and ‘top-down’ approaches (Reed et al, 2006) represented, respectively, by CSO leaders’ perspectives and researchers’ insights. The initial suggestions for indicators were drawn from CSO narratives about `successful’ projects, the ethical/spiritual values underlying their success, and the ways in which these values were enacted on a day-to-day basis. A team of six researchers analysed the interview data to identify priority values and putative indicators.
After successive rounds of prioritisation by the CSO partners, the chosen indicators were rigorously field-tested in five different CSO and business settings (Burford et al, 2013). As in the initial design phase, researchers and CSO leaders worked in close partnership to evaluate the indicators themselves. The CSO leaders commented on relevance, comprehensibility and practical usability, while the researchers assessed their measurability and identified validity issues, such as potential sources of bias. Following reflection on the initial field trials, some indicators were reworded, deleted or split into two parts to create the current set of 166 indicators. A second round of field testing did not result in any further changes to the indicator set.
5. The WeValue indicator set
The full indicator set is available to download on this website. These cannot be cleanly divided into ‘process’ and ‘impact’ indicators, as recent guidelines issued by the UK Government (HM Treasury, 2011) highlight the fact that what is a process indicator in one context may be an impact indicator in another, depending on what the project was aiming to achieve. Using the indicators to evaluate project implementation processes – how people interact and communicate – may provide vital clues to help CSO leaders understand, for example, why an intervention succeeded in one field location but failed in another. The indicators can “make the invisible visible” by highlighting unnoticed problems of trust, integrity, participation, etc.
6. Using the indicators
The list of WeValue indicators is intentionally broad and diverse, to enhance its usefulness, but no organization would be expected to use all 166 of them in an evaluation. The first step is selection of relevant indicators, with varying levels of participatory localization according to the context and evaluation priorities:
(a) At the most basic level, an organization might choose to transform each chosen indicator directly into a survey question, without any changes to the wording. This allows for simple comparisons between different projects or institutions, using statistical tests if desired. This method was used, for example, to compare values-related processes in three different Tanzanian schools, one of which had developed an intercultural curriculum and sustainability projects.
(b) While it is possible to use the indicators in their current form, field tests showed that most user organizations prefer to modify the wording of the chosen indicators, in order to maximise their applicability to the project or activity that is being evaluated. The modification may be as simple as replacing generic words with more specific ones, e.g. substituting ‘youth’ or ‘teachers’ for ‘people’, or it may involve more substantial changes in wording, as below: Original indicator: People feel a sense of power that they can effect change
New indicator 1: Participants feel a sense of power to effect a positive change
New indicator 2: Participants feel that their sense of power to effect a positive change has increased after doing the workshop
(c) The indicators can be used as templates for organizations to design their own customized indicators, expressed in their own language and based on their deepest aspirations. For example, a faith-based organization in Swindon, UK, used a set of WeValue indicators (“People are perceived to be trustworthy”, “People are perceived to be truthful”, etc.) to create the indicator “People are perceived to be mines rich in gems of inestimable value”. This was understood in two different ways:
• Children perceive themselves as `mines rich in gems of inestimable value’ (Awareness that they have the potential to be trustworthy, truthful, etc)
• Children perceive other people as ‘mines rich in gems of inestimable value’ (Broader vision: seeing the good in others, not focusing on negatives)
Individual indicators or sets of indicators may be incorporated into conventional M&E protocols in order to add a values dimension, and/or used as the basis of a systematic values-focused evaluation, as discussed below.
7. “Can we measure respect?”: Linking the indicators to core values
Values are intangible: they cannot be weighed, measured or counted directly, but only through processes of conceptualisation (defining conceptual terms) and operationalization (specifying phenomena appropriate to observe or measure) (Schlater & Sontag, 1994). It is also evident from management literature that different people can understand the same values-related words and phrases in very different ways (Cha & Edmondson, 2006; Lencioni, 2002).
Thus, in order for a value to be meaningfully measured, a specific word or phrase (such as ‘unity in diversity’ or ‘empowerment’) must be unambiguously linked to a set of measurable behaviours or perceptions. The process of linking sets of indicators to specific words or phrases, which we term value-labels, can be done by a single individual (`subjective values conceptualisation’) or by a group, through dialogue (`intersubjective values conceptualisation’). Some practical aspects are as follows:
• Organizations that are already clear about their ‘espoused values’ may wish to evaluate the extent to which these values are enacted in practice (Gruys et al, 2008). To do this, they can start with one of their value-labels and search for relevant indicators in the list. These can be supplemented with new indicators, if required.
• Organizations that do not have a clear values statement may prefer to start by choosing relevant indicators, then sort the chosen indicators into groups and select a value-label such as respect, communication or environment to define each group. This could be done either before or after the data collection phase.
8. Methods of collecting data
Mixed methods evaluation, i.e. combining quantitative methods such as surveys with qualitative methods such as interviews, focus groups and observation, has been recommended in order to enhance validity through triangulation (Mathison, 1989) and accommodate different people’s methodological preferences (Odendaal et al, 2006). Values-focused evaluation with the WeValue indicators does not necessarily require a mixed-methods approach, but it is strongly recommended.
The WeValue web platform and handbook contain information and guidelines on using a variety of conventional assessment methods, such as interviews, surveys, focus groups and structured observation, which can be used separately or together. The case studies also describe a number of creative, arts-based assessment methods that have been custom-designed to mainstream data collection unobtrusively into existing project activities. Here are some examples:
Spatial survey. Each indicator was converted into a question with a three-point scale attached to it, e.g. “Do you feel that women are valued in the group? A little / More or less / A lot.” In keeping with the frequent use of a spiral as a powerful indigenous symbol in the CSO’s regular activities, and the use of long coloured scarves for many of their exercises, a large spiral was formed on the ground with three coloured cloths - each colour representing one of the three possible answers. After each question, the youth were invited to stand on the colour that best reflected their personal response to the question. As the group was small, evaluators could record the responses of individual youth (assisted by video recording) as well as aggregating results for a group perspective. Additional qualitative information was provided by the speed of response: in some cases, the youth moved to their chosen answers without any hesitation, while in others, they took time to reflect before making their move. The survey was followed up with a focus group, to explore the underlying reasons for the responses.
Corporal survey. Similar to the spatial survey, but using body postures to represent responses (e.g. hands up for ‘a lot’, cross arms for ‘more or less’, sit down for ‘a little’)
Word elicitation with hand painting. Youth were asked to sit in a circle, paint something on their neighbour’s hand to illustrate how they felt after a reforestation exercise, and then describe their painting. Evaluators recorded the emotional vocabulary used.
9. Validity considerations in data collection and analysis
In WeValue evaluation, just as in action research (c.f. Peterson, 2010), scientific rigour needs to be balanced against practical considerations such as ease of use. Thus, the selection of a more or less ‘rigorous’ approach to data collection and analysis will depend largely on resource availability, and on the purposes of the monitoring or evaluation.
• For routine internal monitoring, undertaken primarily with the aim of learning and improving project activities, an informal process of data collection and analysis will often suffice. This might involve discussing the data informally as a group, noting overall patterns and unexpected results, and reflecting together on the implications for strategic planning or the future design of training materials.
• In the case of larger evaluations at key stages in the project lifecycle, greater confidence in the results may be necessary, especially if the evaluation findings will be used to influence major decisions about funding or personnel changes. In this case, care will be needed to identify and eliminate possible sources of invalidity, such as social desirability response bias, in which respondents say what they think evaluators or program staff want to hear (Arnold & Feldman, 1981) and group conformity bias, in which respondents ‘follow the crowd’ instead of thinking about their answers for themselves (Jetten et al, 1996).
• Where scientific rigour is paramount, a second evaluator may be required to corroborate interpretations in qualitative research, while significance testing (e.g. with Student’s t-test or Mann-Whitney ‘U’ test) may be helpful for quantitative data.
10. References
Arnold, H. J., & Feldman, D. C. (1981). Social desirability response bias in self-report choice situations. Academy of Management Journal, 24(2), 377-385.
Burford, G., Velasco, I., Janouskova, S., Zahradnik, M., Hak, T., Podger, D., Piggot, G., & Harder, M. (2013). Field trials of a novel toolkit for evaluating `intangible’ values-related dimensions of projects. Evaluation and Program Planning, in press.
Cha, S. E., & Edmondson, A. C. (2006). When values backfire: Leadership, attribution, and disenchantment in a values-driven organization. Leadership Quarterly, 17: 57-78.
Cox, T., Karanika, M., Griffiths, A., & Houdmont, J. (2007). Evaluating organizational-level work stress interventions: Beyond traditional methods. Work & Stress, 21(4), 348-362.
Crishna, B. (2007). Participatory evaluation (I) - sharing lessons from fieldwork in Asia. Child Care Health and Development, 33(3), 217-223.
Daigneault, P. M., & Jacob, S. (2009). Toward Accurate Measurement of Participation Rethinking the Conceptualization and Operationalization of Participatory Evaluation. American Journal of Evaluation, 30(3), 330-348.
Ebrahim, A. (2003). Accountability in practice: Mechanisms for NGOs. World Development, 31(5), 813-829.
Edwards, M., & Hulme, D. (1996). Too close for comfort? The impact of official aid on nongovernmental organizations. World Development, 24(6), 961-973.
Ellis, R., & Hogard, E. (2006). The Trident: A Three-Pronged Method for Evaluating Clinical, Social and Educational Innovations. Evaluation: The International Journal of Theory, Research and Practice, 12(3), 372-383.
Flowers, A. B. (2010). Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program. Evaluation and Program Planning, 33(2), 165-171.
Forss, K., Rebien, C. C., & Carlsson, J. (2002). Process use of evaluations: Types of use that precede lessons learned and feedback. Evaluation, 8(1), 29-45.
Gruys, M. L., Stewart, S. M., Goodstein, J., Bing, M. N., & Wicks, A. C. (2008). Values enactment in organizations: A multi-level investigation. Journal of Management, 34(4): 806-843.
Hogard, E. (2008). Purpose and method for the evaluation of interpersonal process in health and social care. Evaluation and Program Planning, 31(1), 34-40.
HM Treasury (2011). The Magenta Book: Guidance for Evaluation.
Jetten, J., Spears, R., & Manstead, A. S. R. (1996). Intergroup norms and intergroup discrimination: Distinctive self-categorization and social identity effects. Journal of Personality and Social Psychology, 71(6), 1222-1233.
Lencioni, P. M. (2002). Make your values mean something. Harvard Business Review, 80, 113-117.
Mathison, S. (1989). Why triangulate? Educational Researcher, 17(2), 13-17.
Odendaal, W. A., Marais, S., Munro, S., & van Niekerk, A. (2008). When the trivial becomes meaningful: Reflections on a process evaluation of a home visitation programme in South Africa. Evaluation and Program Planning, 31, 209-216.
Peterson, J. C. (2010). CBPR in Indian Country: Tensions and implications for health communication. Health Communication, 25(1), 50-60.
Reed, M.S., Fraser, E.D.G., Dougill, A.J., 2006. An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59, 406-418
Schlater, J. D., & Sontag, M. S. (1994). Toward the measurement of human values. Family and Consumer Science Research Journal, 23 (1), 4-25.
Wightman, F. (2010). Measures of success: Not everything that counts can be counted. Vancouver Foundation Magazine Retrieved 25 July 2011, Fall 2010, from http://vancouverfoundation.ca/documents/VanFdn-Magazine-Nov2010.pdf
