Last month saw the launch of the Arts Council of Northern Ireland?s Evaluation Toolkit. Its author, Annabel Jackson, explains its genesis and the value of evaluation.
The Department of Culture, Arts and Leisure (DCAL) has, in recent years, been subject to financial pressure, which has been translated into a growing requirement that the Arts Council of Northern Ireland (ACNI) demonstrate that public money is spent on activities providing the greatest benefits to society ? and that it is spent in the most efficient way. Additional impetus for an evaluation toolkit came from a review we carried out in summer 2003.
This looked into the business-support needs of funded arts organisations and showed wide interest in evaluation. Some 90% of respondents thought that more or better evaluation information would help their organisation. Some 79% of arts organisations believed that evaluation would improve the effectiveness of the organisation. Motivating staff and volunteers and helping with fundraising were secondary objectives (56% and 49%, respectively). Arts organisations were keen to improve their own skills in evaluation. Some 70% thought that the best structure was to support arts organisations in self-evaluation, with only 21% thinking that ACNI should take responsibility for evaluation.
Practical
The Evaluation Toolkit is about action, not theory. It provides the minimum of background on the discipline of evaluation ? mainly debunking myths about evaluation. The body of the text concentrates on providing detailed practical guidance on the different activities in evaluation ? whether planning evaluation, identifying issues for investigation, writing questionnaires, or allocating tasks for evaluation. The Toolkit includes standard questionnaires that can be used by arts organisations. Some of the questionnaires are compulsory for arts organisations funded by ACNI.
Although the body of the text provides a comprehensive guide to evaluation (including issues such as staffing and quality control) it is not intended that this be read by all users. The Evaluation Toolkit includes a Quickguide for staff members (e.g. project administrators) who simply want to apply the questionnaires without any of the background. The Evaluation Toolkit is integrated with ACNI?s system for Regularly Funded Organisations (RFOs) and Creative Youth Partnerships. The RFO survey, for example, is basically a spreadsheet that collects project level data on each of the projects, performances, productions, exhibitions or publications carried out by funded arts organisations and each of the project participants. Collecting information in this disaggregated form is easier for arts organisations: the aim is that data is collected throughout the year by project leaders, as part of usual management information systems.
Early on it became apparent that fair measurement of the outcomes from community and voluntary arts projects would only be possible through carrying out some kind of follow-up survey six months or so after projects ended. Furthermore, outcomes do not lend themselves to survey through postal questionnaires: they are far more suited to telephone interviewing. The requirements for a large scale, valid telephone survey of participants in voluntary and community arts projects seemed well beyond the resources of most voluntary and community arts organisations. With this in mind, we have established the principle that this work will be carried out on arts organisations? behalf by a dedicated member of staff at the Arts Council.
Principles
- The Toolkit attempts to ask questions of interest to arts organisations and participants. These are intended to be questions whose answers have the capacity to change lives and elevate communities.
- We hope the Toolkit will make evaluation as easy as possible. The Toolkit?s evaluation framework aims to compress the largest amount of meaning into the smallest amount of work. It is essential to respect the time of staff, volunteers and participants.
- It is based on the principle of respecting the variation in voluntary and community organisations. We hope that it is flexible enough to suit arts organisations of different sizes, artforms, cultures, functions and stages in their development.
- It is intended that the Toolkit should operate at different levels. We have defined core indicators that are the minimum necessary to demonstrate the value of the arts sector. We also provide guidance to help arts organisations that want to do more to evaluate the full range of their social impacts. The Toolkit:
- is mixed method: quantitative and qualitative. This is because we believe the method should be chosen according to the question being pursued, not according to a pre-existing ideology.
- aims to respect the values of the voluntary and community arts. This sector adopts an open, developmental structure that gives importance to the process as well as the product.
The results from the pilot of the Toolkit have been very positive. Arts organisations have commented that the Toolkit gives them background information on their participants that they have never had before. It also helps to increase cohesion within the organisation by having one set of forms across different areas of work. This in itself has helped artists and other project workers to feel more valued by the organisation. Collecting information during projects rather than at one point in the year gives more accurate information and takes less time.
There has been a tendency for the arts sector to see evaluation as unimportant paperwork. Evaluation is better seen as the process of asking important and useful questions ? an activity entirely compatible with the creative process. Judging by the enthusiasm of the community and voluntary arts organisations in Northern Ireland, rolling out the Evaluation Toolkit may be hard work, but it will be worth it.
Annabel Jackson is an evaluator, specialising in the arts. t: 01225 446614; e: ajataja@aol.com
The Evaluation Toolkit was written and piloted with the active participation of a team of twelve arts organisations. After the Toolkit has been rolled out to the community and voluntary arts it will then be adapted for, and extended to, the rest of the arts sector in Northern Ireland.