Keywords

1 Introduction

The issue of performance measurement and evaluation is a recurrent topic in the cultural and creative sectors. The past decades have seen a proliferation of academic research attempting to define, measure and assess the different dimensions of the performance of arts and cultural organisations and their impact on society (Labaronne 2017). Not only academia has shown interest in the subject. Also supervising bodies, artistic and managing directors of various art institutions, project leaders of temporary cultural organisations as well as the general public have addressed the issue. This study builds analysis on the premise that caution is required when evaluating artistic and cultural activities outside the, organisational and broader institutional contexts in which they are embedded. Such a premise delineates the epistemological stance of this chapter. Positivist methodologies of performance measurement and evaluation, which assume the context neutrality of scientific methods, are not able to fully capture the intrinsic and intangible aspects of artistic, cultural and creative activities. These approaches do not make sense of the contextual complexity of such activities, ‘overestimating the general validity of methods and underestimating the richness and diversity of the contexts in which they might be applied’ (Chiaravalloti and Piber 2011, p. 242).

This chapter elaborates on the (often uncontested) assumptions underlying decontextualised conceptualisations of performance evaluation inherent to the positivist paradigm. We show why they are more than problematic when applied to cultural activities and organisations, as well as present a contextualised approach to performance evaluation as an alternative to positivist methodologies, which is better suited to capture the arts’ complexity and uniqueness.

The first section undertakes an in-depth analysis of the literature on performance measurement and evaluation in the area. In particular, we scrutinise the origins, inherent values, applied contexts and identified dimensions in the use of performance measurement and evaluation in the nonprofit corner of the arts sector and creative industries. In the section that follows, we narrow down the analytical focus to institutionalised and temporary cultural organisations, basing our analysis on document analysis and empirical data. Section 3 analyses the use, impact and relevance of performance measurement and evaluation practices for the European Capitals of Culture initiative. Section 4 offers in-depth insights into the internal practices of two long-standing arts institutions in Vienna and Berlin and the contexts in which performance is enacted. Building on these insights from the field, the fifth section presents a conceptual evaluation framework for more responsibly representing artistic performance that is derived from the field itself and is thus embedded in its specific artistic, organisational and cultural contexts. The last section highlights the specific challenges of defining and measuring artistic and cultural activities and offers a reflection on the status quo of performance measurement and evaluation in the arts and cultural sector. Finally, we raise crucial points for cultural policy as well as further research.

2 Analysing the Literature

2.1 Origins and Development of the Field of Evaluation

As an academic discipline and profession, the field of evaluation disseminated rapidly in the United States after the Second World War (Abma and Widdershoven 2011). New methods were developed to assess growing social projects and educational interventions; at the same time, public agencies’ and private foundations’ large expenditures on such programmes saw a growing demand for results (Rossi et al. 2004). By the late 1970s, evaluation research established itself as a distinct field in the social sciences in the US (Rossi et al. 2004). In addition, the qualitative paradigm of evaluation emerged. This pragmatic position originated as a challenge to the predominant scientific posture (Rossi et al. 2004), propagating a more holistic understanding of the complex contexts in which interventions take place by considering the perspectives of the affected parties (Abma and Widdershoven 2011). In Europe, the concept of modern evaluation arrived about a decade later, first to Sweden, England and Germany. The notion of evaluation was shaped and accelerated by governmental reforms in the context of New Public Management (NPM) as well as by European integration (Stockmann 2010a). Nowadays—it is argued—we live in an ‘evaluation society’ in which the routinisation of evaluations has proliferated across the entire realm of public life (Dahler-Larsen 2012).

2.2 Definitions and Purposes of Evaluation

The term ‘evaluation’ is associated with the assessment, appraisal or judgement of a particular unit of analysis. According to Mertens (1998), evaluation involves the ‘systematic investigation of the merit or worth of an object for the purpose of reducing uncertainty in decision making’ (219). More recently, evaluations are more broadly understood as ‘situations to stop and reflectively consider our experiences in the midst of a specific social practice’ (Dahler-Larsen 2012, p. 13). Such a broader conceptualisation of evaluation implicitly acknowledges the emergence of new evaluands, evaluees, and evaluation purposes (Labaronne 2019b), posing new challenges for which existing evaluation research struggles to offer solutions (Patton 2018, p. 20). The challenges are associated with the origins of the field of evaluation, which revolved around evaluating (closed) projects and programmes (ibid., p. 21). While tools and logic models work well for projects and programmes that presuppose a cause-effect chain of actions, they lead to misunderstandings, misconceptualisations and misrepresentations in complex dynamic environments (ibid.).

Further, evaluations are commonly associated with four, often interconnected, key purposes: knowledge generation as the basis for decision-making; verification of the degree of goal achievement; concept development as the initiation of learning processes for the design of programmes and organisations; and the exercising of legitimacy for institutions and policy-making (Stockmann and Hennefeld 2013). The different functions that evaluation might fulfil are intrinsically related to the underlying research paradigms. Following Stockmann and Meyer (2010), Table 1 summarises the underpinning assumptions of each research tradition.

Table 1 Paradigms in evaluation research
Full size table

2.3 Understanding the Relation Between Performance Measurement and Evaluation

A further aspect of evaluation and performance measurement concerns the goals, time horizons and subjects of the evaluation, which align with three basic types of evaluations, for which different methodologies tend to be applied (Stockmann 2010b). Table 2 summarises the key dimensions of evaluation research. In the context of institutions (as opposed to projects or programmes), ongoing or process evaluations are often related to performance measurement. As such, these terms tend to be used interchangeably. However, while the terms are related, they are conceptually different. ‘Performance measurement’ is a continuous process by which targets and goals are defined and subsequently operationalised through performance indicators, thus serving as a tool for control and accountability that focuses primarily on efficiency and effectiveness rather than on (long-term) measures of impact (Schober et al. 2012). That being said, evaluative practices have been integrated into NPM and managerial procedures through concepts such as Total Quality Management (TQM), benchmarking and the European Foundation for Quality Management (EFQM) model (Chiaravalloti and Piber 2011).

Table 2 Dimensions of evaluation research
Full size table

2.4 Values, Contexts and Dimensions of Performance Measurement and Evaluation

When it comes to NPM practices, managers try, on the one hand, to measure the performance of different art and cultural activities, enabling comparisons both within one genre as well as among different genres, and even across regions and countries. On the other hand, managers use NPM to set up strategies, visions, brands and philosophies and later compare them with the outcome. This process corresponds with the widespread management-accounting logic of ‘plan-act-control’. For the arts and humanities, this logic is generally unfamiliar, as the comparison between planning and implementation is primarily based on quantitative performance measures, such as the number of visitors and the percentage of self-earned revenue. But these measures usually fail to illuminate artistic excellence. For for-profit companies, it might seem quite straightforward to measure success in this type of one-dimensional construction; that is, at least on the surface, all relevant variables can be calculated in financial terms. Therefore, profit is a reliable dimension to evaluate success and to show the performance of an organisation on the markets. However, even in the for-profit sector, we understand that the use of financial metrics as the established best practice to measure performance is increasingly being challenged. That is, profit is no longer seen as the only indicator of performance. ‘Triple bottom line’ accounting (see Alhaddi 2015) clearly places social and ecological targets on the same level as financial targets. Another notion related to this new extension of the ‘single bottom line’ is ‘PPP’: people, planet and profit (ibid.). In the last few decades, conceptualisations of corporate (social) responsibility have raised the importance of employees and other stakeholders facing up to the ecological consequences of corporate decisions. Companies are also increasingly aware that they can lose sight of the content of their business strategy if they focus solely on the financial metrics that are meant to represent it (Harris and Tayler 2019).

The lack of a profit orientation leaves nonprofit arts and cultural organisations without a clear-cut goal. An overarching performance measurement or evaluation system for such organisations is lacking, and the puzzle of how to measure and evaluate performance remains a recurrent issue (Cairs et al. 2004). For art and cultural organisations, whether temporary or institutionalised, profit figures and other proxies for success and quality can be developed, but their informational value is definitely limited; sometimes they are even counterproductive, as they can generate unintended consequences. As Towse (2001) argues, there is an obvious temptation to fulfil easily measurable goals, such as raising attendance rates, rather than to pursue those goals that are difficult or even impossible to measure, such as increasing quality of output or reducing social exclusion. The performance of cultural organisations and the impact of art is in any case multidimensional, and to a certain degree intrinsic and intangible. In this regard, European Capitals of Culture are not only touristic and urban development projects, just as opera houses and museums do not only sell tickets and attract audiences. They are much more. This complexity is one of the reasons why cultural organisations are accountable not only in one dimension and not only for one stakeholder. Publicly funded institutions also have to satisfy their supervising bodies and legitimate the use of taxpayer money. Usually, they present their performance and their impact with reports of varying levels of detail as well as through performance measurement conventions.

On a political level, cultural budgets are evaluated in terms of their impact in relation to cultural policy goals. Additionally, performance indicators are used to monitor and evaluate the outcome of funding agreements between granting bodies and recipient institutions, providing a bridge between the goals of cultural and arts policies and the management of arts organisations (Towse 2001). Temporary projects, on the other hand, are usually evaluated ex post. After the completion of the project, the impact as well as the cost-benefit relation are investigated. At the level of the cultural institutions and projects, performance measurement and evaluation are used to make informed decisions about performances, projects and initiatives.

At the very end of the line, we reach the level of individual artists and the professional artistic field. Artists tend to establish an automatic and deeply personal judgement about the performance of a film, exhibition, opera, or ballet or the fulfilment of a role. Interestingly enough, this artist level of evaluation, which is inherent to the artistic process, has been traditionally backed only by a qualitative understanding of performance. The perspective of artists, who genuinely engage with evaluation as part of their artistic practices, tends to be underrepresented by most evaluation approaches at the organisational level (Chiaravalloti and Piber 2011). Indeed, the focus on importing management techniques from the for-profit world in an attempt to support the processes of arts organisations has led to the processes of the artists themselves largely being neglected in the arts management literature (Chiaravalloti 2016).

2.5 Evidence-Based Cultural Policy and Performance Evaluation in Arts Management

In the United States, the emphasis on performance measurement and evaluation that arose in the mid-1990s was motivated by a significant decrease in public funding for the arts and culture (Brooks 2000). Today, research into the value and impact of the arts constitutes a core function of the National Endowment for the Arts, the federal arts funding agency in the US. In German-speaking countries, in which the arts sector—in contrast to the USFootnote 1—is characterised by high levels of direct government support, cultural policy is also becoming increasingly evidence based. In general, cultural organisations are being required more and more to produce tangible results (Labaronne 2017; Gstraunthaler and Piber 2012). Under an increasingly evidence-based cultural policy, arts and cultural organisations are increasingly confronted with issues of evaluation and performance measurement. This situation is the case even though, as Svensson (2017) has argued, the arts and cultural sector, under the shield of artistic freedom, has been one of the last bastions to resist the evaluation imperative.

Hadida (2015) offers a comprehensive overview of the management literature on the issues of defining and measuring performance in the arts and cultural sector, albeit not with a focus on the nonprofit arts but rather encompassing the entire creative industries. The study puts forward a taxonomy of existing works, identifying four dimensions of performance (commercial/financial, artistic, societal, and managerial) at two levels of analysis: process and outcome (Hadida 2015). Most of the arts management literature identifies only two pillars of organisational performance: the commercial/financial dimension and the artistic dimension. Research investigating both of these dimensions together is far more abundant than works that define and assess artistic performance in isolation (ibid.). This is not surprising. Artistic achievement constitutes the most controversial aspect of the evaluation of organisational performance in the arts sector (Chiaravalloti and Piber 2011). A reasonable degree of validity is required to describe the ‘performance’ that evaluations aim to assess in order to responsibly account for its accomplishments (Rossi et al. 2004, p. 16). The evaluation of art might be influenced by specific conventions strongly related to its context, and thus the context-related uniqueness of artistic accomplishments should be taken into account (Becker 1982). Hence, clarifying the meaning of ‘performance’ in the specific context in which it is being enacted is particularly relevant in the cultural and creative sectors (Gstraunthaler and Piber 2012).

If we focus analysis on the (non-profit) performing arts for instance, in an effort to develop performance indicators for capturing artistic achievement, scholars have explored the various components of artistic quality and analysed the influencers of high achievement, focusing on the production and presentation as well as on the reception. Chiaravalloti (2016) defines the artistic dimension of performing arts organisations in terms of the artistic process of programming (selection of pieces for the season’s programme), production (staging of pieces) and reception (performance of pieces for an audience). He relates the quality of programming with the artistic director’s taste and repertoire, leadership style and ability to run artistic risk. Boerner (2004) looks at programming, referred to as an organisation’s ‘profile quality’, by considering the full selection of works presented by a performing arts company in a season. Over time, the programme of each season influences the overall profile of a performing arts organisation, and thus becomes a crucial factor for long-term artistic quality (Chiaravalloti 2016). During production, that is, the staging of new pieces, some structural characteristics work as positive influencers on achievement, such as an appropriate organisational culture, a good working atmosphere and the overall satisfaction of artists and staff (Abfalter 2010). The production of a piece is followed by the premiere and a number of successive performances. Unlike the previous phases, during which artistic quality is influenced and evaluated mainly internally, at the performance or reception phase, external stakeholders also judge performance quality. Boerner and Renz (2008) use the term ‘performance quality’ to refer to the artistic quality of a single performance presented by a performing arts company. Further, Boerner (2004) differentiates between the performing art (e.g., music, dance, singing) and the staging dimension (e.g., settings, costuming, lighting) and analyses the congruence between them. Other scholars focus on the collective performance, arguing that high performance quality is achieved when everything comes together in collective virtuosity (Marotto et al. 2007). Many research efforts attempt to objectivise and quantify the audience’s perception of performance quality. For example, Throsby (1990) identifies several components for artistic quality judgements of a single performance, which can be aggregated to evaluate a season, and Radbourne et al. (2009) propose the Arts Audience Experience Index, which measures artistic quality in terms of audience satisfaction.

Despite the efforts to capture artistic achievement, in the context of the performing arts, the presented short overview of the extant arts management literature shows that existing approaches are only able to capture some aspects. Other relevant aspects contributing to the artistic performance of institunionalized arts organisation, such as the development of the own ensemble, remain under the radar under such limited conceptualizations of artistic performance. This is can be partly explained with the dominance of positivist research tradition that tends to assume a context neutrality of methods and techniques—known as the ‘toolkit approach’ (Belfiore and Bennett 2010, p. 121)—and thus presume that similar practices can be applied to different contexts. Just as importantly—at least in the institutionalised arts organizations, that decontextualized approaches tend to neglect system-internal and long-term activities involved in realising artistic achievement Labaronne 2019a).

3 Mapping and Analysing Practices of Evaluation and Performance Measurement in Temporary Cultural Organisations: The Case of European Capitals of Culture

Not all cultural offerings are hosted by permanent organisations. Many festivals, biennials and other recurrent events are hosted and organized by temporary organizations. This raises several managerial challenges, especially in terms of staff, quality improvement and evaluation. Some examples include the music festivals in Aix en Provence, Baden-Baden, Bayreuth, Bregenz, Salzburg, Pesaro, Ravenna and Vienna. A special case are the European Capitals of Culture (ECoCs).

The scope of the empirical work for this case focuses on the evaluation frameworks for the ECoCs. It explores the assumptions, methodologies and theoretical and practical challenges embedded in the evaluation of ECoCs, considering the artistic, social, urban, regional and financial dimension of organisational performance at the process and outcome level of assessment. Although usually two cities host an ECoC project each year, a separate organisational structure is built for each project. Therefore, enabling knowledge transfer and an adequate feedback process among past, current and future ECoC cities and regions is a fruitful pursuit to allow the sharing of experiences and to enrich evaluative practices.

The ECoC programme was launched in 1985 by the Greek minister of culture, Melina Mercouri. The project was created as an intergovernmental initiative before turning into a project of the European Union. After that, many big cultural centres in Europe applied to become “cultural capital.” Since 1985, Athens, Florence, Amsterdam, Berlin, Paris, Stockholm, Prague and Dublin, among many others, have been European Capitals of Culture. The average budget of an ECoC project is between 30 million and 80 million euros. In this first empirical example, we will analyse the monitoring and evaluation framework of ECoCs at the level of the single project.

In terms of methodology, we have used a qualitative approach with three case studies: Marseille, France, in 2013; Valletta, Malta, in 2018; and Matera, Italy, in 2019 (for the cases of Valletta 2018 and Matera 2019 see also Habersam and Piber 2020). In total, we conducted twenty-three qualitative interviews with artists, organisers, members of the ECoC evaluation committees and other experts. We additionally analysed internal and external evaluation reports. The different data were triangulated and subsequently analysed (Eisenhardt 1989).

Basically, we can outline three different pillars of evaluation for ECoC projects:

  • First, each nominated city is subjected to a selection and monitoring process through a couple of preparatory meetings with an expert panel in the years before the event. This process ends well before the event itself, and the European Commission usually awards the Melina Mercouri Prize to the selected cities on the basis of the expert panel’s recommendations. Currently, the award is 1.5 million euros.

  • Second, each city is responsible for conducting its own evaluation of impact backed by common guidelines.

  • Third, the European Commission sets up an independent ex post evaluation after the project, which is usually commissioned to a private company (see, for example, Fox and Rampton 2016; Rampton et al. 2012).

As outlined by the above, one evaluative part takes place before, a second during and after, and a third mainly after the event. According to European Union Decision No. 445/2014 of the European Parliament, the overall target of the ECoC project is: ‘(a) to safeguard and promote the diversity of cultures in Europe and to highlight the common features they share as well as to increase citizens’ sense of belonging to a common cultural area; (b) to foster the contribution of culture to the long-term development of cities in accordance with their respective strategies and priorities’.

In more detail, the evaluative process involves the following. For the first pillar, European Union Decision No. 445/2014 discerns between general, special and operational objectives:

General objectives (GOs)

  • GO1: Safeguarding and promoting the diversity of cultures in Europe

  • GO2: Fostering the contribution of culture to the long-term development of cities

Special objectives (SOs)

  • SO1: Enhancing the range, diversity, European dimension of the cultural offerings

  • SO2: Widening access to and participation in culture

  • SO3: Strengthening the capacity of the cultural sector and its links with other sectors

  • SO4: Raising the international profile of cities through culture

Operational objectives (OOs)

  • OO1: Stimulating extensive cultural programmes of high artistic quality

  • OO2: Ensuring cultural programmes feature a strong European dimension and transnational cooperation

  • OO3: Involving a wide range of citizens and stakeholders in preparing and

  • implementing the cultural programme

  • OO4: Creating new opportunities for a wide range of citizens to attend or participate in cultural events

  • OO5: Improving cultural infrastructure

  • OO6: Developing the skills, capacity and governance of the cultural sector

  • OO7: Stimulating partnership and cooperation with other sectors

  • OO8: Promoting the city and its cultural programme

  • OO9: Improving the international outlook of residents

For the general objectives and the special objectives, the European Commission provides examples of ‘indicative indicators’ (see European Union Decision No. 445/2014, pp. 8ff) as well as ‘possible sources of data collection’. The criteria for the assessment of the projects are divided into the following six categories: ‘contribution to the long-term strategy’, ‘European dimension’, ‘cultural and artistic content’, ‘capacity to deliver’, ‘outreach’ and ‘management’.

In the final step of the selection process, the independent expert panel recommends the cities to be selected by the European Commission as ECoCs. After the selection, the expert panel ‘monitor[s] the preparation of the designated cities for the year of the title and provide[s] them with support and guidance from the time of their designation to the beginning of the year of the title’ (European Union Decision No. 445/2014, p. 9). After the selection procedure, the expert panel meets 36, 18 and 2 months before the year of the title. These meetings can be understood both as an ex ante as well as a process evaluation. The panel includes some people who were involved in past ECoCs, opening up pathways for knowledge transfer between different ECoC cities.

As the expert panel makes the constitutive recommendation to the European Commission whether to grant the Melina Mercouri Prize to a nominated city based on the aforementioned framework of targets, there is a certain motivation for applicant cities to adjust their projects to this framework.

The second evaluative pillar is the evaluation by the city itself, which is considered vital because “the Commission’s evaluations of the past European Capitals of Culture, which are based on data collected at a local level, have not been able to provide primary data on the impact of the title. Therefore, the cities themselves should be the key players in the evaluation process” (European Union Decision No. 445/2014, p. 3). In practice, the scope of evaluation differs from city to city. The organisers of Valletta 2019 established a ‘research team’ that was in charge of various impact studies. Partly in cooperation with local researchers, it conducted qualitative and quantitative impact studies as well as studies on international perception. Other cities, by contrast, did not invest much in evaluation.

Finally, the third evaluative pillar is the independent ex post evaluation (see, for example, Rampton et al. (2012) for more on the ECoCs Essen/Ruhr, Germany; Pécs, Hungary; and Istanbul, Turkey; and Fox and Rampton (2016) for the ECoCs Mons, Belgium, and Pilsen, Czech Republic. Both reports are based on slightly different evaluation regimes (see European Union Decision No. 1622/2006 for the regimes applicable until 2019)). The purpose of the ex post evaluation is also to provide an ‘external and independent evaluation of the results of the European Capital of Culture event of the previous year’ (European Union Decision No. 1622/2006).

In general, the celebration of an ECoC is a complex phenomenon. Therefore, the evaluation is likewise a multifaceted endeavour. Artistic, organisational, participatory and city and regional development issues have to be coordinated within the projects as well as evaluated. Furthermore, an ECoC project has multiple addressees: primarily local citizens and organisations, in a second tier artists and cultural practitioners and initiatives, and finally tourists as well as the destination as a whole. Different city projects naturally have idiosyncratic targets, aim at different stakeholders, and therefore the organisers also have to balance the overall orientation (strategy) and the composition of the individual projects. Any evaluative action has to look at the aims of the project and compare strategies and plans on the one side and their implementation on the other. From another angle, we can discern the planning and programming part (including the balancing of the individuals projects), the production part and the dissemination part (for a similar assessment of opera companies, see Chiaravalloti 2016).

Across all the ECoC projects analysed, we observed several problems with the evaluation process. We want to highlight and discuss four critical issues here. First, the three different pillars of evaluation (outlined above) are not connected. In practice this means that many resources are likely not efficiently invested and a lot of expertise is lost, as three different organisational entities are conducting the evaluation projects. Secondly, two of the three evaluative pillars do not have clear consequences; as one head of an ECoC project put it,

“It says obligatory evaluation, but what are the consequences? There aren’t any. It is pointless.” (Head of an ECoC project)

This is especially true for the ex post evaluation undertaken by the European Commission. Therefore, the applicants’ motivation to invest resources and time in evaluation is limited:

“Probably the word evaluation has a bad reputation. […] I mean, most of the time. It’s something that we are not used to dealing [with.]” (Programme manager of an ECoC project)

However, future ECoC cities can definitely benefit from the evaluations, that is, if the lessons learnt find their way to other organising cities. Thirdly, the long-term impacts of being an ECoC are very hard to assess. As regular evaluation (as well as funding) stops after the year of title, the long-term impacts remain under-investigated, or as a project manager of an ECoC put it,

“We would need studies five or ten years after the year of the title.” (Project manager of an ECoC project)

While the European Commission is not currently doing this research, some external studies are looking into these issues (see, for example, Garcia and Cox 2013).

Finally, the fourth critical issue of the evaluation process is that some intangible dimensions of ECoC-projects are difficult to evaluate. A major question that arises is: Are the initiatives for gratifying cultural elites, for attracting tourists or for affecting the everyday lives of citizens? In this vein, sophisticated evaluative methods are required to show outreach success, involvement levels, and the generation of “spirit” beyond simple participation figures for the many initiatives and projects.

4 (Re-)presenting Artistic Performance in Institutionalised Contexts: The Case of the Performing Arts

4.1 The Context of the Performing Arts

While the mission of performing arts organisations is articulated around the concept of artistic achievement, the sector often uses financial indicators instead of qualitative and long-term measures to evaluate organisational performance (Turbide and Laurin 2009). As mentioned earlier, metrics such as the number of performances, earned income and attendance rates might satisfy the needs of funding bodies but they offer little information about artistic accomplishments. Furthermore, a cultural performance is always a result of artistic, organisational and managerial efforts. Therefore, another problem arises in that many evaluation approaches separate these connected fields (see, for example, Krug and Weinberg 2004). As the Section “Analysing the Literature” discusses, scholarly efforts have developed performance indicators to capture artistic achievement. Yet, these efforts fell short to capture system-internal and long-term activities involved in realising artistic achievement in institutionalised artistic contexts (Labaronne 2019a).

Following the call for deeper insight into the artistic and institutional contexts of performing arts organisations and into how enacted performance can be better defined, captured and subsequently evaluated, this section elaborates on an alternative approach to existing decontextualised methodologies that is intended to better represent the artistic dimension of organisational performance in an institutional context. As such, the following addresses (part of) the challenges associated with evaluating artistic performance in the performing arts by aiming at a contextualized representation of artistic achievements (i.e., by aiming at a more appropriate definition of performance). Then it is argued, that a more responsible conceptualization of “performance” (the evaluand) in the different art disciplines can assist on overcoming part of the shortcomings related to its assessment (Labaronne 2019b). To this end, the scope of the following analysis focuses solely on the process level (as opposed to the level of outcome) of art production, that is, of the artistic dimension of organizational performance (see Hadida 2015).

The proposed framework is derived from a comprehensive ethnographic study that explores organisational practices in the performing arts (Labaronne 2019a). The concept of ‘practices’, as applied to the arts and reflected by Zembylas (2014, pp. 1–16), seems suitable as a unit of analysis (Labaronne 2019b). Practices are understood as configurations of cohesive activities for creating coordinated and collaborative relations that encompass actors as well as institutional settings, and the concept stresses that these practices are collectively constituted and regulated. Further, the notion materialises the body as an ‘aesthetic sense-making resource’ (Biehl-Missal 2017, p. 16), foregoing the positivist mind-body divide and challenging the traditional view of scientific management that is often represented through the metaphor of ‘organisations as machines’ (ibid., p. 29). In addition, the concept of practices overcomes the dichotomy between the individual and the collective, enabling social phenomena to be captured at different levels of analysis (Zembylas 2014, p. 2). By putting organisational practices at the core of the enquiry, the study conducted by Labaronne (2019a) captures the intrinsic, intangible and long-term aspects involved in realising artistic achievement and offers a framework to makes these aspects more explicit.

4.2 Realising Artistic Achievement in Institutional Contexts

A comprehensive ethnography study of world-renowned performing arts organisations was conducted during the 2017/18 season in Berlin and Vienna. The Appendix shows the key characteristics of the cases. The study focuses on dance companies as a case for the performing arts given that they represent an exceptional empirical setting from both an artistic and a management perspective. Dance productions feature a complex interplay between various elements such as choreography, music, costumes, sets and lighting, demanding a complex work schedule. Therefore, dance companies provide an ‘extreme case’ (Flyvbjerg 2011, p. 307) for the performing arts. The salient focal dynamics of extreme cases facilitate concept and theory building (Eisenhardt 1989).

For data collection, about 500 hours of structured observation were documented. While the ethnographic reports represent the main source of data material, informal and semi-structured formal interviews as well as relevant documentation add to the validity of the empirical material. Moreover, to further enhance the validity of the field research, multiple voices from the field were used (Denzin and Lincoln 2011, p. 5) and several types of data were triangulated (Eisenhardt 1989). Data analysis and data gathering overlapped, as is typical in studies informed by ethnography (Eisenhardt 1989).Footnote 2 Qualitative analysis followed contemporary treatments of the grounded theory approach (Charmaz 2006), in that both deductive and inductive procedures were used.

Putting the working practices of producing and presenting the performing arts at the core of the enquiry manifested the inward-looking and resource-oriented approach to the organisation of work in an institutional context. The findings were thus framed following a resource-based understanding of organisations. The resource-based view constitutes a theoretical lens that understands organisations as bundles of resources and capabilities (Prahalad and Hamel 1990). It looks at internal resources (rather than products) to analyse organisational performance (Kong 2007, p. 722). A resource-based understanding of an organisation allows the integration of any given single production (short-run) and the season’s programme consisting of parallel productions (long-run)Footnote 3 under a common analytical lens that acknowledges the resource interdependencies in the organisation of work and that offers novel insights into how such organisations deploy and develop resources to realise artistic achievement.

The conceptual framework developed from Labaronne’s study (2019a) builds on the dimensions of ‘resource disposition’ and ‘resource development’. Each dimension contains different subdimensions that comprise various aspects of organisational practices, reflecting competences and capabilities required to mobilise resources for artistic achievement in performing arts organisations (see Table 3).

Table 3 Dimensions and subdimensions of artistic achievement: The case of large state-funded dance companies
Full size table

According to the above table, artistic achievement in the institutionalised performing arts is not only about producing creative outputs that can be quantified, aggregated or objectified. The production of creative outputs, such as the creation and presentation of new dance pieces, represents only one aspect of what is traditionally (and limited) conceptualised as ‘performance’ in arts management scholarship. Basing evaluation procedures on a limited conceptualisation of performance, for example, using only financial metrics or audience satisfaction indexes, misrepresents managerial and artistic efforts such as those encompassed by the criteria outlined in Table 1. ‘Performance’ as a managerial term in institutionalised contexts also needs to reflect longer-term value-creating activities that might remain under the radar of decontextualised performance evaluation approaches. For instance, as important as the creation of new pieces is, so is the preservation of the existing repertoire, which might take up just as much resources. In the words of one managing and artistic director of the studied organisations:

“A successful season should bring new, interesting pieces, in which the dancers can develop themselves. It is not useful to create new ambitious pieces for the programme if the dance ensemble does not feel comfortable or the style is too unfamiliar to them. […] This is because we are a repertoire house with our own ensemble. […] We have to develop our dancers and also offer a season with an interesting mix of new pieces while still offering the existing repertoire.” (Managing and artistic director)

At the same time, realising artistic achievement is about the further development and motivation of a company’s ensemble. This implies that collaboration with guest choreographers for the co-creation of new pieces is not only about the development of new pieces for the season’s programme but also about the guest choreographers” contribution to the long-term artistic development (coaching) of the ensemble, and thus the retaining of dancers. As a company press release states: ‘the company invites internationally established choreographs to personally work with the dancers and develop the style of the company’ (Staatsballett Berlin 2017). Consequently, artistic achievement involves a complex interplay between short- and long-term considerations of resources. It involves developing an interesting season’s programme for the benefit of the public as well as of the company’s artistic staff (‘reconfiguration of resources’) while implementing this on a monthly basis (‘coordination of resources’) in a way that considers the vulnerability of dancers’ bodies as ‘aesthetic sense-making resources’ (Biehl-Missal 2017, p. 20), the need for further development (‘co-creation’ and ‘coaching’) and the ongoing practice of keeping repertoire pieces alive (‘preservation’).

The proposed conceptual framework can be used to design formative (ex ante), process (ongoing) or summative (ex post) evaluations to be undertaken by different stakeholders at different level of analysis. A newly appointed artistic director can use the framework as a formative assessment of past performances and an estimation of future ones in order to formulate an artistic vision and programme upcoming seasons. Organisations can use the framework to undertake a summative evaluation to review past seasons (e.g., to decide about future repertoire) while using it as ongoing assessment of the current one. Lastly, the framework can assist funding agencies and cultural policy makers to more holistically assess artistic achievement and more responsibly account for new appointments and reappointments of artistic directors or decisions about grant distributions.

That being said, it should be mentioned that the framework is grounded on the empirical and analytical context of large dance companies with permanent ensembles. The identified dimensions and subdimensions of the framework might provide insights into similar arts institutions, but a simple transfer of this contextualised approach to other types of artistic and organisational activities should be considered with caution—and most likely with a sophisticated adaption. As mentioned earlier, artistic accomplishments are embedded in specific conventions and a context-related uniqueness (Becker 1982). In this light, ‘the arts’ can only be considered together with velvet gloves (DeVereaux 2017). Taking this into consideration, the presented framework aims at a contextualised representation of artistic achievements in the performing arts through putting forward a more suitable definition of what constitutes performance in the very unique context they inhabit. Only a more considered and scrutinised conceptualisation of what constitutes ‘performance’ (the evaluand) in the different art disciplines and organisational settings can assist in overcoming some of the shortcomings related to its assessment (Labaronne 2019b).

5 Discussion and Conclusion

As the above elaboration reveals, some of the challenges associated with the definition and measurement of performance in the arts sector can be explained by the constructivist theories of art that remain prevalent today. In this light, art is not only a social construction but also a social product. Artistic accomplishments are embedded in specific conventions, organisational and managerial efforts and a context-related uniqueness. As such, also evaluation is strongly related to the generic context of art and culture (Becker 1982). The judgement of what is considered to be art and consequently what is considered to be good art can only be made by certain qualified experts—being part of the art world/s (Dickie 1984). This is the case for both examples presented in this chapter. In this vein, each ECoC project is situated in a special geographic, economic, social and cultural context, which on the one hand opens up manifold possibilities and on the other hand as well limits managerial action. This fact already shows that comparisons between ECoCs are not possible on a profound level. Only on an operative basis a limited benchmarking of clearly defined measures can be possible. For institutionalised arts organisations, particularly those with solid resource bases, the strong internal focus on the mobilisation of their own—sometimes unique—resources also makes comparisons inadequate. Taken the cases together, the analysis shows the inadequacy of the still widespread hidden assumption that quality and performance in the art and cultural sector are ‘ontological objects’ which can be measured in a methodologically sound way.

Another difficulty in defining artistic performance in the nonprofit arts sector has been described by Hadida (2015): as a product or outcome, artistic activities often exhibit characteristics of merit, have qualities of public or semi-public goods and might involve intangible individual or collective experiences that are not easily captured. Moreover, the outputs may be simultaneously aimed at different stakeholders and the outcomes may span several years—as we can see in the case of ECoCs (see also Brooks et al. 2002; Finocchiaro and Rizzo 2009) and repertoire-based performing arts organisations (Labaronne 2019a). In terms of the urban development of a city, an ECoC project may trigger new developments, which can only be seen and studied after a considerable span of years. This long time frame is also necessary to evaluate the sustainable development of an art institution’s ensemble, whether a corps de ballet or an orchestra. Insofar, we can conclude, that in both cases the current evaluation practices address primarily short-sighted plans and processes without touching the sustainable development and the long-term impact.

Concluding, we thus argue that challenges reside in the inherent intricacies associated with defining and measuring aspects of artistic and cultural performance that are, among others, contextual, subjective, intangible, long-term and often unquantifiable. Hence, researchers and cultural practitioners encounter difficulties when attempting to identify ‘measurable’ or ‘comparable’ indicators (Paulus 2003; Turbide and Laurin 2009) and define ‘objective’ measures (Boerner and Renz 2008; Radbourne et al. 2009). Although, rich qualitative data could be transformed or captured into one-dimensional figures, there are still the issues of aggregating several components and of how to merge the different rationalities that underline different financial, artistic, socioeconomic and political logics (Gstraunthaler and Piber 2012).

Performance and impact are the result of manifold efforts in artistic, managerial and organizational matters. Evaluation activities often separate the impact in parts and consequently they become blind for the big picture. Like a symphony can’t be evaluated via separately analysing each instrument, artistic achievements consist of various, mostly indivisible parts. For isolated areas, one-dimensional approaches might work, but calculating an overall aggregated performance seems to be impossible, as this usually destroys the portrayal of the unique qualities of an arts organisation as well as the big picture of a cultural or artistic performance. Similarly, it is difficult to compare different organisations, even ones in the same area. This chapter has shown that idiosyncratic value is difficult to quantify. However, we don’t argue entirely against metric- and financial-based measurement within the arts and cultural sector, but rather emphasise the importance of being aware of the inherent limits of such metrics, which can’t be overcome with more or better measures. Many examples show that as soon as overall quantitative performance measures are available, their special contexts of origin get lost, and both the meaning and the relevant benchmarks of the performance measures lose their relevance. That is to say, the ‘quantified performance’ becomes the ‘performance’ itself, and is taken for granted as such.

To avoid the above scenario, the evaluation of arts and cultural entities requires highly specific expert knowledge and therefore, it does not present an easy object of evaluation—especially if one looks into an organisation or a project from the outside. For the evaluation of cultural and artistic projects, events, performances and organisations, this means that we have to rely on the expertise of specialists and insiders. However, this requirement does not exclude the possibility of presenting metric- and financial-based performance measures for single areas of interest.

This chapter provides an empirically anchored reflection to better understand the limits of evaluations of both temporary and institutionalised arts and cultural projects and organisations. For those involved in evaluating institutionalised arts organisations, it has been suggested that assessing solely at the outcome level—that is, relying only on external measures of artistic quality such as audience surveys and economic analyses to determine the impact of arts organisations in their communities—conveys a limited and short-sighted representation of managerial and artistic efforts (Labaronne 2019b). Similarly, the impact of ECoC-projects is a complex and long-term issue. To evaluate the sustainability of these efforts, particularly in institutionalised contexts, intrinsic, system-internal and long-term activities need to be taken into account (Biondi et al. 2020; Labaronne 2019a).

From a methodological point of view, arts management research still has to develop methods that include inside expert knowledge without the disadvantages of biased opinions and the self-interest of the evaluators. Only by adequately representing performance in each particular artistic, organisational and cultural context will performance evaluation activities be able to more holistically and responsibly reflect what they intend to characterise. Arts management research that addresses these issues might attempt to more responsibly ‘bridge the gap’ between managerial performance measurement, cultural policy evaluation and artistic judgement.