Quantifying the success or failure of arts events is increasingly important for arts organisations and is often undertaken by consultants. Jess Tyrell asks what organisations can expect from this growing army of evaluators.
The Arts Council and many of the sectors other paymasters are increasing the emphasis they place on evaluation. As champions of the sector search for more evidence to prove the value of the arts, producers are being asked to justify their work. The demand for all bids to Grants for the Arts to show how they will evaluate their work has introduced a greater discipline, making producers think about how they will judge the effectiveness of their practice. But is this all a good thing and how should producers of small- to medium-size projects with limited resources respond?
Working on both sides of the fence evaluating projects and attempting to offer fresh approaches to help others reflect on their work, at the same time as evaluating our own year-long youth dance project were learning a lot: about the work, the importance of evaluation and its limitations. Recently, I organised an evening bringing together a group of freelance evaluators to discuss some of the challenges and approaches to evaluation, what it is for and what we hope to achieve as professionals. We agreed to meet again and set up an informal network to continue to discuss methodologies, share information and develop ways of doing things. This article is based on our initial discussions and some of my thoughts. I want to ask three questions: What is evaluation? Why do it? And what should you as a producer expect from the relatively unregulated group of individuals and small consultancies who offer their services?
Audiences matter
To start, lets play a philosophical what would you do? Youre an archaeologist excavating the foundations for a new building and you discover a never-seen work by Michelangelo buried in a box with an explosive device attached to it. If you open the box youll destroy the work of art, but if you dont open it the work will never be seen. Should you leave it buried, preserved forever, or open it destroying the work but making the site safe? It depends whether you think the work is important even if it will never be seen. Does art have value without an audience?
For an evaluator, the answer is no. Good art is all about the relationship between the work and the audience, and the evaluators job is to assess what the impact of it has been. All artists should have their audience as their focus, otherwise the work, in my view, is redundant. If a piece of art is never seen, of course, it has no value; its raison dêtre rests in what it is communicating and the way in which it does this.
Understanding what passes between artist and audience/participant is what evaluation in the arts is about, but there is much debate about what we should be looking for from the evaluation process. I have sympathy with the arts for arts sake argument, but this makes evaluation difficult. If art is subjective and about the individual experience, proving value becomes difficult, even impossible. This government has become more and more explicit about the arts being useful in achieving other public policy goals, justifying investment by showing how the arts contribute to other public goods. The arts are good because they improve education, divert young people from crime or improve the health of communities. Reducing art down to these, more concrete outcomes, makes the evaluators job easier. But is it right? Trying to capture what is inherent in an interaction between artist and audience is challenging. What do you measure? The degree of empathy? A warm feeling? A life-changing experience? And when do you measure? It might not be for years that the extent of the change is felt, and how can you tell it was just that piece of work that changed a persons sensibility?
Improving practice
Evaluation is the new required buzzword tacked on to applications and is the passport to securing funding. But does anyone really know what it means? And how much of it is done well? Given the difficulties of proving cause and effect, how should small- to medium-scale producers respond to the increased pressure to carry out evaluations given limited resources? At its centre, evaluation should be about learning, and keeping practice rooted in what the experience has meant to its audience. But there are dilemmas. Good evaluation can be expensive. Cause and effect is intangible, often there is no baseline against which to compare change, impact is messy and difficult to prove and data difficult to collect. And most of all, it is challenging for producers to be setting up methods of measuring work when all their focus is on producing it. This is why an outside, independent evaluator sympathetic to the process can add a huge amount.
But what are our expectations of evaluation and what do those funders, demanding that we do this, want from it? Comparisons between different peoples work are difficult to make as most evaluations remain private. If our funders are serious about this, shouldnt they be doing more to raise the evaluation game? For example, Arts Council England (ACE) could do so much more to share the knowledge it gains from the thousands of evaluations it receives each year best practice guidelines, case studies, a catalogue of errors to help us learn from others mistakes.
Collective intelligence
What does ACE do with all these reports? Does anyone actually read them, beyond the box-ticking exercise of making sure projects have met their output targets and spent the money as they said they would? Where does this collective intelligence go? Or is it that when reporting back to funders, producers are less than honest, wanting to paint their project in the best light, underlining their success and proving they can be trusted to deliver again, next time an application goes in?
To do the job well, a good evaluator should provide a balance between objective challenge, experience and clarity on what the evaluation is for. And as evaluation develops, we should also be searching for more creative ways of involving audiences in feedback, like the Explore, Expand, Exchange exhibitions in Manchester that made a cardboard structure for people to write their views on, that became an exhibit in itself. How many of us really get excited by the bog-standard questionnaire? What kills a great artistic experience more than filling in name, gender and how did you find out about us under duress on your way out?
Jess Tyrell is Director of Germination, which has recently produced the Reel Bollywood project, the evaluation of which will be published shortly. She runs a network for freelance evaluators which has its next meeting on 24 January. t: 020 7841 8970
e: jess@germination.co.uk
See The Pig That Wants to be Eaten by Julian Baggini (Granta Books, 2005) for this and other philosophical teasers.