Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Crowdsourcing initiatives are growing in popularity amongst museums and historical institutions as a means to collect new data and enhance their existing collections (Ridge 2014). As such, the need for reliable and valid data places constraints upon the design of crowdsourcing systems and an emphasis on error-free operation and other strictly functional aspects of usability. However, user experience as a whole encompasses much more than this. In this chapter, we argue that user experience has a fundamental impact on the nature of interactive engagement with cultural heritage, and consequently on the capacity of crowdsourcing to promote meaningful participation by existing and new audiences of volunteers.

Many initiatives past and present have provided meaningful work for communities of enthusiasts (Ridge 2014; Straub 2016). The motivations for taking part can be varied and are dependent on personal context and identity (Raddick et al. 2010; Rotman et al. 2012). However, it is only really a small number of deeply interested enthusiasts/hobbyists that make up the large percentage of contributions (Eveleigh et al. 2014; Suh et al. 2009).

We argue that if we are to take museums’ principle of institutional integrity into account, systems should be built that engender respectful and transparent relationships with their volunteers. From this ethical standpoint (see the UK Museum Association’s Code of EthicsFootnote 1), a museum has an active role in society to (a) provide meaningful work for public participation, (b) cultivate new audiences beyond those with existing deep interests, and (c) be respectful of their volunteer’s well-being. Placing emphasis on the amplification of positive aspects of user engagement when designing these systems may be a method of moving towards fulfilment of the three principles simultaneously.

We report a study carried out in collaboration with the American Air Museum, part of the Imperial War Museums in the UK. An image classification system was deployed in gallery to explore participants’ reflections on their user experience and to identify components of engagement that can be targeted for design work. We place particular emphasis on the emotional component of engagement due to its noted importance in prior museum visitor research (Falk and Dierking 2013). Our work addresses the lack of research relating to the role of emotional engagement in crowdsourcing tasks.

2 Crowdsourcing for Museum Collections

Digitisation of museum collections and advances in web technologies have allowed museums to open up their collections to the public. Many museum websites now have a searchable online collection such as the RijksmuseumFootnote 2 in the Netherlands, the Imperial War MuseumFootnote 3 in the UK or the Holocaust Memorial MuseumFootnote 4 in the USA. These collections are sometimes curated into online exhibitions, but for the most part they exist as searchable databases for academic researchers and interested individuals. In this way, they are primarily useful to people who know what they are looking for and not casual visitors.

Crowdsourcing initiatives provide an alternative means in which people can encounter a museum’s collections whilst also taking part in meaningful work on behalf of the museum. The popularisation of these systems can perhaps be seen in the success of the early ZooniverseFootnote 5 citizen science movement. Volunteers take part in transcribing and classifying media in order to aid scientific research. Some of the most successful of these projects have been Galaxy Zoo (classification of galaxies), Snapshot Serengeti (classification of wildlife) and Old Weather (transcription of ship logbooks).

Crowdsourcing initiatives can be classified in a number of ways (Oomen and Aroyo 2011). Wiki-type project participation, in which volunteers complement an existing collection in some way, usually requires a level of prerequisite knowledge. The American Air Museum website, for example, is collecting data about the military personnel, civilians and aircraft connected to the US Air Force Britain during World War II. In order to contribute to this project, the volunteer must either (a) know or possess information about a person or aircraft (perhaps be a family member) or (b) have research skills, enthusiasm and knowledge relating to the subject matter. These criteria raise the barrier to entry for general public engagement. Classification and transcription-type tasks, on the other hand, require little or no prerequisite knowledge. Participants simply need to analyse an image within the structure of the system, usually answering a classifying question or transcribing text within the image.

It is important to consider why museums are attracted to launching crowdsourcing initiatives. The (UK) Museum Association Code of EthicsFootnote 6 outlines ethical principles for museums as public institutions. The code was developed as a collaborative process between museum stakeholders, members of the public, funders and interest groups across the sector. The code adopts three principles:

  1. 1.

    Public Engagement and Public Benefit—Actively engage and work with existing audiences and reach out to new and diverse audiences; Promote meaningful participation with museums; Use collections for public benefit—for learning, inspiration and enjoyment.

  2. 2.

    Stewardship of Collections—Acquire, care for, exhibit and loan collections with transparency and competency in order to generate knowledge and engage the public with collections.

  3. 3.

    Individual and Institutional Integrity—Act in the public interest in all areas of work; Build respectful and transparent relationships with staff and volunteers to ensure public trust in museum’s activities.

Museums are attracted to crowdsourcing initiatives as a solution to high work load projects that are just not possible with limited resources available to them. When taking this perspective, a museum’s motivation to build a crowdsourcing project is to fulfil the stewardship aim and to enhance an existing collection through data collection. However, as Owens (2014) notes, crowdsourcing provides a convenient solution not only to fulfil stewardship-oriented aims of the museum but also satisfy the aim to engage the public with their collections. In doing so, the museum should not take advantage of the volunteers as cheap labour.

If we are to adhere to the principles set forth by the Museum Association, the work in which they participate should be meaningful, enjoyable and inspire learning. The initiative should actively seek to engage new audiences rather than rely on a small selection of enthusiast hobbyists. Designing initiatives that fulfil these criteria, in turn, may be said to fulfil the institutional integrity principle; ensuring a respectful relationship between the institution and the crowdsourcing volunteers. This is an active effort on the museum’s part to design engaging experiences that are inclusive, meaningful and satisfying.

2.1 User Engagement in Crowdsourcing

A user’s values, motivations, goals and interactions with a system contribute to their user experience. User experiences are multifaceted since they describe matters of relevance and value to users. As such, one of the key facets of user experience is the extent to which a person finds their interactions with a system engaging. User engagement in Human–Computer Interaction is a term used to refer to the emotional, cognitive, behavioural and temporal relation between a user and a system (O’Brien 2010; O’Brien and Toms 2008). In simpler terms, it is a quality of a user’s experience that gives emphasis to positive aspects of interactions (Attfield et al. 2011). How we term positive engagement is dependent on the user and the system designer’s goals. If there is a mismatch of perspectives, users or designers may not see the values and meanings that matter to one another.

From the museum’s perspective, we can use the principles set out by the Museum Association’s Ethics Code as mentioned in the previous section. Desirable outcomes towards public engagement and public benefit may be learning, inspiration, enjoyment and meaningful participation. The user’s goals may or may not be congruent with these ideal ethical aims of the museum. We can look to existing literature to better understand volunteers’ motivations and positive aspects of their engagement with crowdsourcing systems.

O’Brien and Toms developed a framework for engagement through a study based around user encounters with web search, online shopping, gaming and webcasting applications (O’Brien and Toms 2008). The framework consists of four phases: a point of engagement, a period of sustained engagement, disengagement and re-engagement. Within the period of sustained engagement, they identified a number of attributes of engagement that contributed towards a user’s positive engagement with a system: focused attention, perceived usability, endurability, novelty, aesthetics and felt involvement.

2.1.1 Designing for Reliability

In crowdsourcing system design, challenges relating to data and system usability and functionality take precedence over user experience related attributes. In some ways, the need for reliable and valid data may overbear upon an experience-centric approach to design. Classification systems are simple and structured systems because they need people to give reliable and valid information as a primary function. There is perhaps a dichotomy between designing for usability and designing for user experience. Using video game design as an example Cook (2008) has noted, if the game Super Mario Bros was designed to be usable the result would simply be a button labelled ‘Rescue Princess’. The user would click the button and achieve their goal. This is a purely usable system that streamlines user activity towards its main goal but not one that would result in a positive experience for the user/player. In a similar way, crowdsourcing systems designed for usability around data validity may not support positive attributes of engagement that give rise to positive user experience.

2.1.2 Designing for Engagement

Designing for engagement may have to be underspecified and open to customisation to support dynamic personalisation (Deterding 2015). However, valid and reliable data is at the core of crowdsourcing system and so should not be compromised for the sake of user experience. Without the collection of useful data, the crowdsourcing work being undertaken loses its meaning. Due to the importance of usability to classification tasks, crowdsourcing system designers employ a number of mechanisms to promote engagement without the need to compromise data validity.

The Old Weather ship logs transcription project uses gamification and social features to encourage user engagement with the project. The system awards points and ranks to top contributors allowing volunteers to become ‘captain’ of a ship if they made the most transcriptions from the relevant log book (Eveleigh et al. 2013). This competitive mechanism was found to be motivating and rewarding to some users, ignored by others and even contributed to the decision to cease participation by some.

Another mechanism used by the Old Weather project is the use of narrative feedback. As the coordinates of the ship are transcribed from the logbook, a ship’s course is plotted across the map. This led to some volunteers becoming more emotionally attached to a certain ship and gave way to a sense of identification and exploration of the past (Blaser 2014).

The addition of social features has also been used to promote social engagement around a project’s content. The American Air Museum website incorporates a user activity feed on their front page to engender a feeling that the site is an active social space. The Zooniverse Galaxy Zoo project encouraged the use of forums to promote work and conversation around the project’s content and inspire volunteer-led scientific discovery (Straub 2016).

Motivations for participation amongst volunteers can vary and so cannot be distilled to a single underlying motivational factor. Intrinsic and extrinsic factors, such as altruism, personal interest, learning, competitiveness and sociality can all be motivations to take part in an initiative (Raddick et al. 2010; Rotman et al. 2012). It is typical for a crowdsourcing project to have a skewed pattern of participation; most of the work is done by a small percentage of the system’s user-base. For example, 94% of volunteers only contribute to 15% of the input in the Old Weather project (Eveleigh et al. 2014). In this way, the ‘crowd’ can be thought of as a group of dedicated enthusiasts supported by a large number of ‘dabblers’. Referring back to the code of ethics, we might consider a museum’s role to actively reach out to and engage with new audiences. With regards to crowdsourcing initiatives, this could extend to ‘flattening’ this skewed pattern of participation by actively designing to engage those beyond the core enthusiast group and thus making the resources spent on building such systems of greater value to a larger percentage of the public.

The present study sought to understand the energising and attenuating factors that lead to engagement in the context of historical classification systems. We have proposed that designing for user engagement can be a mechanism for meeting the public engagement, stewardship and integrity aims of a museum.

2.2 Emotional Engagement in Museums

As discussed in the previous section, designers of crowdsourcing systems can target particular components of user engagement to energise through design mechanisms (For example, the forums on the Zooniverse project aim to energise social engagement around the content). In the present study, we draw focus to the emotional component. This is based upon prior research in visitor emotion in the physical museum and findings from our previous exploratory study. The emotional aspect of engagement can broadly be thought of as the affective experiences of user’s interactions with a system (O’Brien and Toms 2008). Falk and Dierking note that emotion has long been thought of an important part of the physical museum experience but to a large extent has been poorly understood (2013, p. 191).

Dierking connects emotions, motivation and interest in informal learning environment of science and technology museums (2005). She argues that emotion is a vital part of learning and the formation of memory. She goes on to provide three reasons for effective free-choice learning; (a) visitors experience ideas on their own terms with an autonomy to attend exhibits depending on prior knowledge and interest, (b) the institution provides a safe, comfortable environment in which they are free to encounter and pursue ideas and (c) visitors can see and experience how the encountered ideas apply in the context of the real world. To Dierking, emotions experienced as part of this personal context are fundamental constituents of learning.

There is some evidence to suggest that a visitor’s emotional state is an important factor in exhibition design. Studies show that after visiting an emotionally arousing exhibition a visitor’s level of reflection and long-term learning is higher when compared to visitors of a less arousing exhibition within the same museum (Falk and Gillespie 2009).

In a previous study (Wrigglesworth and Watts 2016), we explored museum visitors’ sensemaking process as they encountered a set of historical images. Each image was accompanied by a set of verbal prompts to elicit meaning-making by the visitor. The responses were analysed through a grounded theory methodology (Strauss and Corbin 1994) to identify components of engagement with historical information. The qualitative responses were analysed to expose the salient components of engagement. Several of these emergent components can be said to be pertinent to emotional experience:

Empathising—The participant talks about how they think the people in the photograph are feeling.

Responding emotionally—The participant makes reference to their own feelings about the photograph content.

Seeking personal connection—The participant seeks to know more personal information about the subject.

The personal context seems to be important in this sensemaking process. Although a photograph can be understood as a factual representation of circumstances, when actively engaged by an individual it becomes a mirror in which their own experiences, prior knowledge and feelings are reflected. This is exemplified by two individual’s different reactions to the same photograph during a pilot study; the photograph depicts US bombers on a bombing run.Footnote 7 One lady was moved to tears and was noted saying ‘You sometimes forget about the people involved’. A man took a contrasting view and cheered the bombers on, ‘Well, they got what was coming to them’. In both cases, the participants were emotionally involved with the content of the photograph, but their emotional reaction was a result of their own personal context. The man’s reaction was perhaps more understandable when later in the museum cafe he was heard reflecting on the photograph and explaining to family members that his relatives were heavily bombed in Coventry during the war. Such emotional connections to content may serve as a mechanism for engaging visitors with historical content.

People visit a war museum knowing that they will likely encounter media that evoke negative emotions; images of violence, death and suffering. The lady that cried may have experienced unpleasant emotions as she was moved to tears. These negative emotions, however, do not necessarily mean that her experience as a whole was negative. People have been found to enjoy and appreciate experiencing negative emotions when playing video games (Bopp et al. 2016). Entertainment audiences voluntarily expose themselves to unpleasant feelings (Bartsch 2012; Oliver and Raney 2011). People seek entertainment to satisfy their psychological needs and move towards well-being. Sad and moving films can satisfy a viewer’s eudaimonic motivations; an individual’s search for deeper insight, meaning and self-actualisation (Bartsch 2012; Bartsch et al. 2014; Tamborini et al. 2010).

The present study aims to explore the findings of this prior research in relation to museum crowdsourcing activities. We explore the relationship between the felt emotional involvement to those depicted in historical content (as suggested by our previous study), reflective and meaningful entertainment and enjoyment (as suggested by museum motivations and the prior work in entertainment experience studies). As covered in previous sections, by targeting components of user experience (in this case emotional) in this way, we aim to explore how crowdsourcing can provide an opportunity for a museum to fulfil its aims of public engagement, stewardship and integrity simultaneously.

3 Study

3.1 Study Rationale

We have argued that public engagement and integrity should be reconciled with the data-driven aims of a museums mission when designing crowdsourcing systems. In the study reported herein, we partly focus on one particular aspect of user experience: emotional engagement with the crowdsourcing task content. Our previous studies (Wrigglesworth and Watts 2016) have suggested that museum visitors respond emotionally to historical content and prior research in entertainment psychology and media communication has shown that users have a psychological need for emotional gratification.

The first part of the study focused on five operationalised components of engagement: positive and negative affect, connectedness, reflection and enjoyment. The relationship between these components has been explored to test the hypothesis that feelings of connectedness with people depicted in the photographs relate to enjoyment and reflection of the crowdsourcing task. Further to this, we used a post-task interview to provide qualitative data and provoke participant’s reflection about their experience of the task. From this, we hoped to acquire a complementary and rich data set to explore emergent components of engagement that may be targeted for design work.

3.2 Quantitative Measures

Five measures were developed from the relevant literature. All survey measures consisted of a 7-point Likert scale. A reliability analysis was conducted for each scale (see Table 8.1). All measures had a high-reliability rating (Cronbach α > 0.8).

Table 8.1 Reliability analysis results for each scale
Full size table

3.2.1 Panas

The Positive Affect Negative Affect Schedule (PANAS) is a widely used set of two 10-item mood scales to measure both positive and negative dimensions of affect (Watson et al. 1988). In the present study, we used the scale to assess the positive and negative dimensions of a participant’s mood during the task as a whole. It treats positive and negative affect as orthogonal to each other rather than opposing poles on the same scale (Ekkekakis 2013).

3.2.2 Connectedness

A scale was developed to assess the extent to which participants felt empathetic and connected to the people in the photographs. The scale was adapted from Bartsch’s emotional engagement with characters scale (2012). The wording of the scale items was changed to be relevant to photographs rather than apply to television and films as they do in the original. For example, the general statement ‘… because I identify with the characters’ outlook on life’ was adapted to ‘… I was able to put myself “in the Shoes” of those depicted in the photographs’.

  • … I had a connection to the people shown in the photographs

  • … I understood how the people in the photographs were feeling

  • … I was able to put myself ‘in the shoes’ of those depicted in the photographs

  • … I was able to relate to those depicted in the photographs

  • … I cared about what happened to those shown in the photographs.

3.2.3 Reflection

A scale was used to measure the extent to which a participant was provoked to think about meaningful subjects. This was adapted from a contemplativeness scale developed in prior research (Bartsch 2012; Bartsch et al. 2014). This measure is pertinent because, as well as being a measure of contemplative entertainment, museums aim to provide the public with opportunities for inspiration and learning.

  • … I was inspired to think about meaningful issues

  • … I was inspired to gain new insights

  • … I thought about meaningful events in the world

  • … I thought about myself in relation to others

  • … I found the task thought-provoking.

3.2.4 Enjoyment

Enjoyment was measured using a scale developed and validated for measuring enjoyment of computer games (Ryan et al. 2006). As with reflection, it was used in this instance as a measure that is desirable from the museum’s point of view to satisfy public engagement motives. The scale was reduced to five measures for brevity in the survey.

  • … I enjoyed doing this activity very much

  • … This activity was fun to do

  • … I would describe this activity as very interesting

  • … While I was doing this activity, I was thinking about how much I enjoyed it.

3.3 System Design

The study was conducted using a custom-built web-based image classification. The design of the system was largely influenced by two existing designs: the Zooniverse citizen science classification platform and the American Air Museum’s in-gallery image classification interactive tables.

A user was presented with a photograph and a simple multiple-choice question regarding the content of that photograph. For example, ‘Is there a person in this photograph?’ A simple question logic tree determines which question will be asked next. There were two main logic branches: questions regarding people and questions regarding aircraft. When the user reached the end of a logic tree, the system moves onto the next photograph. This approach is used in most classification systems including the Zooniverse Labs systemFootnote 8 and can be said to be paradigmatic of such systems.

The questions were adapted from the AAM’s own in-gallery crowdsourcing system. This was so they were emblematic of a museum’s desired classification-tag output in relation to the content. The photographs were from the AAM’s Roger Freeman Collection; a collection of ~15,000 photographs of the US air force and civilians in Britain during World War II. A total of 12 were used in the present study to keep the study session time to a reasonable length. Six were chosen that depicted people and six to depict aircraft. This was to keep the set balanced in regards to the question logic tree focus (People and Aircraft). The order in which the photographs appeared was randomised to combat order biasing effects.

3.4 Study Context

The study took place in the American Air Museum’s Georgia Frontiere gallery, a room separate from the main museum hall. The gallery contains two crowdsourcing interactive tables that were designed by the museum (and on which the study’s task questions were based) and the roll of honour for the 8th Airforce. It is regarded as a quieter and more reflective space than the main museum hall.

The museum does not have a strong narrative trail and is open plan. The gallery is positioned at the back of the hall, so visitors are therefore likely to have passed by and looked at exhibits on the way. Therefore, there may be framing effects introduced by this configuration, as people think about the exhibits they encounter before reaching the study location.

3.5 Participants

Participants were recruited from visitors to the American Air Museum. They were approached by the researcher and asked if they would like to take part in a short experiment for the museum. A total of 31 participants took part in the study (74% Male, 26% female). The age demographic was largely skewed to people aged 45–55 and 55–65 age categories (39% and 32%, respectively). This is in line with the demographics for the American Air Museum website visitors as reported by their Google Analytics. The sample can therefore be said to be representative of the American Air Museum’s current audience.

3.6 Approach

A mixed methods approach was used for the present study. Quantitative measures were used to explore correlations between, positive negative affect (PANAS) and the emotional connectedness, enjoyment and reflective nature of the crowdsourcing task. Data was gathered by a survey when the user completed the task. The survey consisted of three pages. The first was a PANAS survey headed by the question ‘How did you feel during the task’ and given a Likert scale ranging from 1 to 7 for each affective word (1 being ‘not at all’ and 7 being ‘very much’). The second page was the engagement component survey measuring connectedness, reflection and enjoyment. Participants were asked the extent to which they thought the statements were true on a Likert scale of 1–7 (1 being ‘not at all’ true and 7 being ‘very true’). The third and final page asked participants for their demographic data (gender and age category). The order of the Likert scale terms was randomised for the first two pages for each participant, in order to reduce ordering bias effects.

The quantitative survey was followed by a short semi-open interview. This was conducted to gather rich data about the participant’s engagement with the system to complement and elucidate any findings from the quantitative data. Participants were asked about their experience using the system in regards to the task as a process and to elaborate upon their thoughts and feelings about the specific content they encountered. The length of the interview ranged from 5 to 10 min depending on how much the participant had to say.

The audio recordings of the post-task interviews were transcribed and then coded line-by-line. An open-coding process was used to allow for codes to emerge from the data (Saldaña 2009). These open codes were then developed into concepts as the coding progressed. The categories were in constant flux as the analysis proceeded; concepts were created and merged as they were constantly being compared to each other.

3.7 Study Procedure

The study procedure took the following form: Participants were asked to read the study briefing on the introduction screen. This briefing explained that their help was needed to classify the museum’s large collection of photographs. They were told that they would have to complete 12 photographs and the task would be followed by a survey and short interview.

The participants were then left to complete the task. Upon finishing, they were thanked and then asked to proceed with the survey. On completion of the survey, they were asked to take part in the interview.

4 Results

4.1 Survey Results

A Pearson’s product-moment correlation coefficient was computed to explore the correlational relationships between the five variables. The results can be seen in Table 8.2.

Table 8.2 Pearson’s correlation results for the five factors
Full size table

There is a strong positive correlation between enjoyment and positive PANAS (r = 0.745, p < 0.001), and moderate positive correlations between connectedness and positive PANAS (r = 0.544, p < 0.01), and to a lesser extent, reflection and positive PANAS (r = 0.440, p < 0.05). Negative PANAS has no significant correlation with any of the variables.

Connectedness, reflection and enjoyment all hold moderate correlations to each other; enjoyment and connectedness (r = 0.671, p < 0.001), enjoyment and reflection (r = 0.647, p < 0.001) and connectedness and reflection (r = 0.681, p < 0.001). This may suggest that connectedness, reflection and enjoyment are all constituent parts that make up a positive experience with a crowdsourcing task.

Figure 8.1 shows a flattened three-dimensional graph that visualises the correlations between connectedness, reflection and enjoyment. The vertical line on the graph represents the reflection mean (17.58) and the horizontal line represents the connectedness mean (18.0).

Fig. 8.1
figure 1

Three-dimensional graph showing the correlation between connectedness, reflection and enjoyment

Full size image

4.2 Post-task Interviews

The coding process for the transcribed interviews resulted in a pool of 28 concepts. The concepts from this pool were further analysed and abstracted into six categories with an interpretive sensitivity for engagement (See Table 8.3 for results). Sensitivity to engagement refers to interest or disinterest in respect of the activity. These categories often contain what can be considered opposed concepts, for example, the category of motivations contains both motivating (altruism) and demotivating (unclear goals) concepts.

Table 8.3 Categories and the concepts they were developed from
Full size table

4.2.1 Altruistic Motivations

Participants often cited altruistic motivational reasons for enjoying or appreciating the task. Taken as a whole, this category can be thought of as the motivation to take part in meaningful work for the common good. Helping a museum was seen as a good thing to do in itself, ‘I felt like I was I thought I was helping you guys out by doing it’. To some, identification with the particular museum fuelled this altruistic motivation, ‘If I got to it because I was looking at the Duxford website that might sway you, because you might think you want to support them’. Some participants liked to feel part of something bigger than themselves, ‘I think it’s kind of neat to play a part and to assist’.

The task was meaningful because the work was focused on important world events, ‘World War II is a huge point in our history and I think it’s becoming obsolete and, not being forgot, that’s not the correct word, but we need to continue on a legacy because it’s such a massive, important thing, so, that’s why I would do it’. In contrast to this, some people seemed to find that unclear goals had the opposite effect, ‘I didn’t know how I was contributing really, I suppose’. This was either from them not understanding the project aims as a whole, or that they struggled to understand how the answers they gave related to the classification tags.

4.2.2 Novelty

Participants were particularly engaged by the content of the photographs that seemed to be novel to them in some way. ‘You see things you don’t normally see. You see them doing their everyday things… you know’. Much of this novelty factor was relating to the social history aspect of the photographs; seeing people act out their lives and duties rather than looking at the aircraft or the technical side of war, ‘I guess I’m not into all the planes side of things, I’m more into looking at the social side of things, how they relaxed and all that kind of thing’. This can perhaps be a consequence of finding certain types of images overly familiar, ‘I’ve seen a lot of similar photos before, so I wasn’t very surprised by any of it’. Images of grounded aircraft or bombers flying in formation seemed to have a lower novelty factor than the behind-the-scenes photographs such as those showing men and women relaxing together. However, the influence of this component may be dependent on the participant’s prior interests. Many were pushed on in the task by the curiosity of which photograph they would see next.

4.2.3 Resonance with Prior Experience or Interest

Engagement was reported to be energised by photographic content that was in accordance to the participant’s identity and personal interests. This could be directly related to what the photograph was depicting, ‘I like the technical ones because I’m a bit of a nerd on the technical ones with the planes and stuff’. Sometimes a participant’s familial connection was a bridge to amplify engagement with the photograph content and the task as a whole, ‘Both my grand-fathers fought in world war two on the American side, one of my grandpas landed at Omaha beach on D-Day, so it all kind of ties in, so I found it all very interesting’.

In contrast to this, engagement may be attenuated by a participant not identifying with the task, ‘I guess it’s more for people that are more interested in the war museum’, or identification with only a certain aspect ‘[I liked] Looking at the crew, the uniforms, the equipment; the actual airborne bombing didn’t do much for me’. The most common expressions of personal interest in the photograph content included interest in social history, the aircraft depicted and vocational connections.

4.2.4 Emotional Involvement

When participants reflected upon the photographs that they found most interesting, they often talked about their engagement from an emotional perspective. Some people described how seeing certain photographs affected them, ‘Some of the ones of planes that were dropping bombs I thought was shocking kind of to see the action and the flak you know’. This was often due to the visual impact of the photograph depicting dramatic scenes such as bombers in action, ‘The planes bombing and that are just impressive, you know it’s a horrible thing they’re doing, but they just look impressive’.

The social–historical scenes provoked more contemplative emotional involvement as participants would think about and empathise with the people they saw, ‘Just looking at things like the clothes and the food they ate; I guess just thinking about how they felt when they were in the situation’, and ‘I liked to see some of them where there were people going to have their food, they looked exhausted, so that you know that’s what I mean you can see, you can sort of get into it’. This deeper reading into each photo perhaps led some participants to think about the stories behind those in the photographs, ‘I mean yeah, they are all interesting, each one tells a really specific story, it’s really interesting’ and (when asked what they found interesting), ‘Well, a planes a plane. Shouldn’t say that too loud should I? I think it’s individual’s stories and how things were different’.

In contrast to this, some people found that their emotional involvement was attenuated by their focus on the classifying task, ‘So mostly I was trying to answer the questions—I’d turned off the emotion button [laughs]’.

4.2.5 Opportunities for Deeper Engagement

Some participants found that there were no opportunities for deeper engagement with the subject matter or that the task was not meaningful to them, ‘I felt like I was just doing a service for you guys’. Their concentration was, for the most part, focused on the task activity; analysing the photographs to reliably answer the questions, ‘I was just answering the questions as opposed to looking at the photograph with any meaning’. Some expressed an interest to learn more about the subject matter such as where the photograph was taken or who the people in the photograph were, ‘You didn’t know if that was at Duxford or if it was in the other US airbases or it was in North Korea or South Korea so the interesting bit would be a direct connection to Duxford’.

4.2.6 Task Temporality

The temporal aspect of engagement emerged in a number of ways. Some noted that they felt that their attention to the task waned as the time went on, ‘A bit too long, but I would have gone to 10 rather than 12, I was starting to lose attention at the end’. This may have been due to the repetitiveness of the simple questions and structure of the system, ‘You could almost predict the questions, you almost went into autopilot slightly’. Some people may have been happy to go along with the repetitive nature of the task because of altruistic motivations, ‘A little bit mechanical, but obviously you told me there was a purpose to it so I was quite happy to go along with that’.

In contrast, some reported that as they proceeded with the task it became easier once the initial ambiguity had been overcome, ‘As I went along, the more you got into it, the more you actually studied it’.

5 Discussion

The present study has explored the role of engagement in crowdsourcing using an historical image classification system. We found that the emotional connectedness to the people depicted in the images was positively correlated to two desirable elements of volunteer experience (from the museum perspective): reflection about the subject matter and enjoyment of the task. All three of these factors were significantly correlated with positive affect. Negative affect had no correlation to any of the factors. This may suggest that connectedness, reflection and enjoyment are all components that result from or contribute to a positive experience with a crowdsourcing task.

However, the data does not suggest causality. From this data alone, we cannot yet say that designing for this type of emotional engagement leads to higher reflection and enjoyment, only that if one is present we are likely to find the others. Given the prior research in emotional engagement in physical museums (Dierking 2005; Falk 2009; Falk and Dierking 2013; Simon 2010) and the emergent component of ‘emotional involvement’ and ‘resonance with prior interest’ (perhaps better termed as personal context), it is reasonable to suggest that it is a salient part of visitor’s experience that carries over into crowdsourcing applications.

To further explore the volunteer’s perspective of their engagement with the task, we collected qualitative data by conducting semi-structured interviews. The interpretation of this data yielded six categories of user engagement: altruistic motivations, novelty, resonance with prior interest, emotional involvement, opportunities for deeper engagement and task temporality. These components may not be considered exhaustive but they are the most noteworthy and prominent in the present study. We can operationalise these findings by developing the categories into design lenses: ways of viewing design work from a given perspective (Deterding 2015).

5.1 Emotional Involvement

The emotional involvement construct comprises of feelings of connectedness, emotional impact and narrative involvement in regards to the crowdsourcing task and its content. This might manifest in design work in a number of ways. As mentioned previously, the Old Weather project encourages volunteers to become attached to the ships they encounter as they transcribe log books. This is done through visual feedback of the ship’s journey as the coordinates are transcribed. The narrative unfolds as they work and the volunteers feel involved in the story (Blaser 2014).

5.2 Resonance with Prior Interests

As we have suggested before, emotional involvement is influenced by a volunteer’s personal context and the content’s resonance with prior interests. Interests in regards to the material encountered in the present study varied from social history, technical and aircraft, vocational interests or familial connections. Designing for a variety of interests can be quite challenging. Some Zooniverse projects enable multiple work flows to be followed. This is akin to setting up different sets of classification questions that focus on different aspects of classification. For example, The American Air Museum’s in-gallery system uses four branches of classification enquiry: people, aircraft, places and documents. Giving users a choice for the branch to follow may help them direct their work towards their own personal interests. Another possible solution would be to give the volunteer a choice of images to tag next rather than just present them with a random one.

5.3 Altruism

Museums are social institutions, so people regard their crowdsourcing work as being part of that social enterprise. They see value in it and are therefore motivated by the altruistic aspect. A number of design features can be employed to amplify this component of engagement. Community features and social translucent mechanisms can help engender the feeling that the volunteer is working meaningfully as part of something bigger than themselves. This could be implemented through community activity feeds, project progress bars, records of contributions, messages of thanks from the museum, clear project goals, and updates about the how the museum is using the output from the project.

5.4 Novelty

Novelty is a component of engagement that relates to seeing something new, a sense of discovery or feeling behind-the-scenes in some way. Previous research has shown that when a volunteer is informed that they are the first to encounter a particular image, their contribution increases (Jackson et al. 2016). Informing the volunteer with a notification is a simple design mechanism to this end.

5.5 Opportunities for Deeper Engagement

A problem with crowdsourcing and especially classification systems is that the museums do not know much about the specific content being shown. That is why crowdsourcing is being used in the first place. This makes it difficult to provide opportunities for deeper engagement such as learning. Beyond providing links to general learning resources about the subject matter, social design features can be employed. Integration of forums with the system allows volunteers to further develop their interests through conversation around the crowdsourcing content. Zooniverse’s Galaxy Zoo project is an exemplar of the benefits taking this approach can have. Volunteers led their own research activity resulting in a new category of ‘Green Pea galaxies’ being discovered (Straub 2016).

5.6 Task Temporality

Over time, attributes of engagement with a task may attenuate. This can be due to the loss of attention and tiredness, the repetitive nature of the task, or perhaps a degrading of positive components of engagement such as loss of novelty. We may be able to frame this in terms of balancing volunteer skill level and the challenge level of the task, a building block of flow theory (Csikszentmihalyi 1996). As volunteers master one level of analysis such as answering simple questions (Is there an aircraft in this photograph?) we might offer them a more challenging variation on the task that requires a higher level of skill (What type of aircraft is this? Here are some reference photographs to help you).

These perspectives aim to align design of crowdsourcing systems to user motivations and desired outcomes. The context and content of a specific system may ‘naturally’ accommodate certain components more than others, e.g. the personal aspect of transcribing an artist’s notebook may more easily afford emotional involvement. Design work that amplifies the positive aspects of these components (and removes the attenuating aspects) should lead to volunteer satisfaction and well-being. The ultimate aim of this approach is that a museum is able to fulfil its remit to cultivate engaging experience for volunteers, which in turn means that the museum is respectful of the volunteer’s needs and going some way towards fulfilling the institutional integrity principle.

6 Limitations and Future Work

The study was limited by a number or factors. First, due to the exploratory nature of the study, the data only shows correlational relationships and not causal relationships between the factors. There was a relatively low number of participants (n = 31), therefore certain statistical methods such as factor analysis could not be applied reliably. The visualisation of the data in the graph, the lack of correlation to negative PANAS, and the consistency of connectedness and reflective components appearing in the qualitative data, all suggest the correlations to be plausible. However, now that this exploratory study has exposed these correlations, larger scale studies should be conducted to confirm these findings and to further explore the underlying causes.

Another limitation was that the study took place in a physical museum context and not online as the target system would in practice. This may have coloured what people might have experienced and so lays a constraint on what we might be able to say in terms of designing engagement for crowdsourcing. The survey results may also have been influenced by this. Museum visitors may be more receptive to or reflective about experiences at the museum as opposed to at home. However, the use of museum visitors as participants is perhaps justified because by visiting the museum, they have demonstrated an interest in the subject matter of the system and so their motivations and interests may be more representative of a typical crowdsourcing crowd.

The study was conducted using historical photography and so the exposed components of engagement may be more relevant to humanities projects in which people and the past are present in the crowdsourcing content. We have tried to generalise these findings in the discussion and design lens perspective. Also, in reporting this study, we have outlined an approach to conducting research into investigating user engagement which can be applied in multiple contexts.

7 Conclusion

We have taken the position that museum crowdsourcing projects should be designed to fulfil the ethical principles of public engagement and benefit, stewardship of collections and institutional integrity. We have argued that the cultivation of positive user engagement can lead to volunteer satisfaction and well-being. A particular focus to our perspective is the emotional component of engagement due to prior research in physical museum settings, prior exploratory work and the lack of related research in the context of crowdsourcing systems.

The present study explored the relationship between five potential factors that contribute to positive user experience with a historical classification system: connectedness, reflection, enjoyment and positive and negative affect. Further to this, we explored the felt engagement from the volunteer’s perspective through the conduct of interviews and qualitative analysis.

Our quantitative study has shown emotional connectedness and relatedness to be components of enjoyable and positive user experiences. Six categories emerged from the data that reflected a user’s engagement with the classification system: emotional involvement, altruistic motivation, novelty, resonance with prior interest, opportunities for deeper engagement and task temporality.

We reason that using these components of user engagement as design lenses when designing crowdsourcing systems can lead to more positive volunteer experience and thus fulfil the public benefit and institutional integrity principles of a museum’s ethical remit.