Evaluating cities and capitals of culture

Bringing together academics, practitioners and policy makers, the Cities of Culture Research Network (CCRN), funded by the Arts and Humanities Research Council, was established to study, assess and improve the ways in which evaluation studies of Cities and Capitals of Culture (COCs) can inform cultural and urban strategies. 

The network includes partners involved with UK Cities of Culture (UKCoCs), European Capitals of Culture (ECoCs) and London Boroughs of Culture.* Over the last three years we have held several conferences and workshops and enabled a productive and mutually supportive Postgraduate and Early Career Researchers’ group that built connections between emerging researchers.

To encourage fruitful and engaging communication between policy making, research and evaluation, the network has developed a set of core values, principles and recommendations, which we would like to share as widely as possible. 

Six recommendations

1)    Recognise that the objectives of researchers and policy makers are different

The objectives of researchers and policy makers do not always coincide. Researchers aim to evaluate critically and with an awareness of complexity, multiple voices and places. They will pursue the evidence even if it leads to uncomfortable conclusions. 

Policy makers seek clear, cogent recommendations that may avoid or reduce complexity. Such differences vary in each situation, but the tensions that may result should be acknowledged. Policy makers should respect and guarantee academic freedom and complex and critical perspectives. 

Researchers should acknowledge the expectations, pressures and timescales of policy makers. Independent, research-based policy making based upon critical, open, and honest collaboration benefits all parties. It also benefits wider society. 

2)    Recognise potentially conflicting roles

Research-based evaluation should be: a) focused on learning (not on the celebration of CoC events or on PR to this end); b) polyvocal and inclusive of interested constituencies; c) critical, independent and resistant to boosterism, hype and any attempts at censorship. 

We argue that research-based evaluation should anticipate the challenges that may arise when evaluators have potentially conflicting roles, especially if universities act simultaneously as partners and evaluators of CoC projects in their own cities. 

If a research unit or university does undertake this role, the evaluation team should include members from outside the local university. The governance of the project should also ensure that all partners recognise the evaluators’ independence.

3)    Use appropriate methodology and approaches

Critical, flexible, and disinterested research-based evaluation requires appropriate methods and approaches throughout the different stages of the event’s development, planning and delivery. It also needs to select methods that anticipate, recognise, and acknowledge the interests of the different local stakeholders, partners and audiences in a CoC, while retaining critical independence.  

Research-based evaluation should therefore: a) recognise the strengths of longitudinal research, and plan for continuing and follow-up evaluation from the CoC award stage onwards; b) reflect upon potential audiences and their sometimes conflicting expectations and then consider which  interpretations and ways of communicating the findings of research-based evaluation best suit different audiences; c) engage a wide range of local actors and communities, including marginalised and ‘seldom-heard’ voices, to avoid over-representing some voices or minimising others, and to include excluded or marginalised actors on their own terms.

4)    Consider data priorities and hierarchies – and dissemination

Good research-based evaluation must consider thoroughly what kinds of data are more important in different situations, and how data priorities and hierarchies are decided and by whom. It is crucial to address questions of access to data: how it will be shared with stakeholders, and how will the integrity of the data be assured and maintained. Data should form an open resource, accessible to both researchers and stakeholders in the cultural sector who may then use it for their own research, evaluation and reflection. 

Therefore, both quantitative and qualitative data should be produced (rather than predominantly quantitative metrics), so that evaluation captures the full range of the voices, responses and impacts generated by CoC programmes. 

Equally, evaluators should plan for the extensive and imaginative communication of findings beyond established textual reports and conference formats. Potential dissemination partners include journalists, creative professionals, public health specialists, politicians, policy makers (operating in areas ranging from city planning to place marketing to social policy, economic development, and tourism) and other academics external to CoC evaluation. Evaluation and critical analysis should be communicated in ways that reach all these groups and, potentially, other groups we have not considered.

5)    Sustain open communication and cross sector understanding

Research-based evaluators and policy makers should encourage and pursue regular discussion and exchange to sustain open communication and cross-sector understanding. They should also focus on future research agendas, with particular attention to: a) evolving concepts and categories in the field (such as ‘seldom-heard communities’ or ‘left-behind places’); b) post-pandemic policy priorities and shifting expectations and contexts for CoC evaluation and research (such as the changing understandings of public space, of ‘workplaces’, and of the contribution of cultural activities to tackling aspects of the mental health crisis, and to reinventing local economies after the reduction of office, retail and night-time activities).

6)    Enable effective collaboration to share understanding and practice

Policy makers should provide placements, mentoring and training opportunities to enable closer, more effective collaboration with researchers, and to share understanding and practices surrounding issues like research impacts, and research presentation and dissemination. 

This kind of embedded, sustained communication will encourage mutual understanding. It will help researchers understand what policy makers want from research and its dissemination. It will also help all parties understand how research-based evaluation might acknowledge, and learn from, failure and disagreement.

David Atkinson is Professor, Cultural and Historical Geography, University of Hull.
Dr Franco Bianchini is Associate Director, Centre for Cultural Value, University of Leeds.
Glenn Burgess is Professor of History, University of Hull.  
Dr Alexandra Oancă is a Postdoctoral Fellow, Department of Social & Cultural Anthropology, University of Leuven. 

 https://citiesofculture.co.uk/
 @DavideAtkinson | @ffbianch | @AlexandraOanca | @cities_culture

*This initiative is a collaboration between the Universities of Hull, Warwick and Coventry. It includes partners from all three UKCoCs (Derry-Londonderry 2013, Hull 2017 and Coventry 2021) and both the UK ECoCs (Glasgow 1990 and Liverpool 2008). CCRN also encompasses Waltham Forest (London Borough of Culture 2019), Aarhus (Denmark, ECoC 2017) and Galway (Ireland, ECoC 2020).