Should the role of the evaluator be reinvented? Martin Heaney considers the problem of how to turn anecdotal evidence into significant and useful data.
How can evaluators provide convincing evidence of participation in cultural activity? The increase of evidence-based policy-making and tighter budgets has created greater demand for ‘proof’ of positive social impacts. Creative Partnerships’ (CP) adoption of a National Evaluation Framework is one example of a trend towards a more rigorous approach to evaluation, with more emphasis on ‘hard’ quantitative data. Recent critical scrutiny of evaluation methodology, however, notably by Munira Mirza1, draws attention to weaknesses in arts organisations’ evaluation of social impacts. Some of the issues Mirza identifies include: how to transform anecdotal evidence into robust qualitative data; the challenge of the impacts of transformation occurring long after evaluation; and how to deal with negative impacts.
The rhetoric of instrumentalism and the pressure to respond with minimal reflection can be regarded as antithetical to more traditional patterns of arts evaluation orientated to inclusivity, learning and improving practice. Key theorist François Matarasso opposes recent excesses where “instead of being guided by the possibility of opening debate about culture within democratic society, arts evaluation is little more than an extension of private sector managerialism to a public service”2. At best, however, the evaluator can be repositioned as the architect of debate – without losing any rigour in reporting. This reinvention requires, I believe, the adoption of new creative tools and a revision of professional role and ethos.
REINTRODUCING CREATIVE PRACTICE
To initiate recent evaluations of large-scale projects, I draw increasingly on my practice as a drama-based facilitator to create spaces for reflection within programme delivery. Activities including mapping or visualising desired project outcomes can be converted into written benchmarks, which can be returned to for final reflections. Similarly, drama-based activities can be used as a stimulus for partners to co-define criteria for effective project partnership and management.
CO-DEFINITION OF EVALUATION STRATEGY
What is evaluation? What is good evaluative practice? The connotations of evaluation as judgement can cast dark shadows. Creative activity which opens up, at the outset of a project, how individuals ascribe value to their work can dispel antipathy and harness engagement in deeper reflection and participation in the evaluation cycle. It may also create potential insights into existing good practice, new ways to apply existing data-collection processes within the evaluation period and generate further opportunities for organisational and individual development.
OBSERVATION
Perhaps the most vital area in securing rich and reliable data is intensive exploration of indicators both before and during arts interventions. To benchmark and determine the value of activity undertaken, dialogue is essential to agree appropriate tools and measures. What, for example, might be the indicators of confidence-building in a dance class for young people? Here, external evaluators can facilitate and encourage the surfacing of ‘hidden’ understandings only accessible to participants. It is interesting to note increasing attention to this area. Arts & Business has been working with the London Benchmarking Group to provide a framework for partners to agree and set indicators. The MAP Consortium offers an experiential approach in its training for CP agents in different regions, focusing on the acquisition of tools for questioning, observation and dialogue on setting indicators, to enable agents and practitioners to link their activity to the CP evaluation cycle.
Through greater attention to inclusive dialogue, and participation in evaluation with stronger links to creative practice, it may be possible to go beyond the anecdotal and locate ‘immeasurables’ with more accuracy – with data sets that are textured and of greater value to both participants and funders.
It is not the intention here to create another ‘victory narrative’ – telling everyone how marvellous everything was. The approaches outlined above are common sense to a degree, but also point to other requirements and complexities: the need for evaluators to draw on a wider range of methods and facilitation skills; to be advocates and creative consultants who can challenge delivery timetables and demand insertion of space and time to reflect. Candour is required in negotiating timelines and adopting realistic expectations. Few funding arrangements permit the longitudinal approach necessary to evidence profound social transformation. What changes can be meaningfully identified and tracked? How can we steer project evaluation to capture the aspiration of participants and celebration of their work, and at the same time ensure that the reach of evaluation does not exceed its grasp in terms of claims of effectiveness? The art and science of evaluation is refreshingly young. A renegotiation of purpose with participants and attendant rigour in creating new frameworks is a timely advance.
Case study:
Oxfordshire Youth Arts Partnership (The OYAP Trust) has created a new Toolkit as a result of a one-year programme, ‘All Together Different’, led by the trust with Oxfordshire youth theatres. The programme included exchanges, master classes, a circus residency and a final platform performance as a means to develop youth theatre resources and to investigate the processes of collaboration between partner youth theatres with extremely varied constituencies. Much of the content was generated through a final creative consultation facilitated by the evaluator in an ‘Open Space’ model workshop, where project co-ordinators could reflect on (but not discuss) feedback from participants. This forum provided a means to reflect and learn from different perspectives and to learn from any negative impacts. Helen Le Brocq, Director of the OYAP Trust, said: “Creative consultation unlocked insights but required honesty, to face up to collective shortcomings in a positive and constructive manner. Without a facilitator leading a creative process, I think so much of this fertile thinking would have never been aired.” Feedback from participants indicated that the workshop provided a valuable space for review and a questioning of core principles and practices.
Martin Heaney is freelance consultant and Associate of The MAP Consortium. He is also a visiting lecturer at the University of Winchester and The Central School of Speech and Drama in Applied Theatre Practice and Creative Learning.
E martinh.1@btopenworld.com
T 07939 148420
This week Martin saw Michael Haneke’s film ‘The White Ribbon’, a lyrical study of pre-World War I rural Germany with dark undertows of patriarchy and its frightening legacy, and John Baldessari’s exhibition ‘Pure Beauty’ at the Tate Modern – art to make you think.
 |