Abstract
We present the results of an exploratory player experience study on the game Quantum Moves, a citizen science game where players move quantum particles to help create a quantum computer. Eight-hundred-and-seventeen players responded to a 13-question exploratory survey constructed to understand how players relate to the game, what are their motivations, and how could the game be improved. We analyzed the data using descriptive statistics and thematic analysis. Specifically, the thematic analysis helped identifying two cross-cutting themes amongst the players: (a) learning and (b) the opportunity to contribute to science. Results indicate that the opportunity to help science, along with game design, game elements, involvement of players with the scientific community, and players’ strategies influence experience. Implications of the particular findings for the research on player experience on citizen science games and development of evaluation methods are discussed.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Citizen science is established on the involvement of volunteers in scientific research with the goal of increasing the scope, scale, or diversity of scientific practices [1]. However, the sustainability of citizen science depends on the degree to which members of the public are interested and provide continued contributions. As such, one of the biggest challenges in citizen science is recruiting and retaining sufficient participants to achieve a project’s goals. Participation in citizen science varies greatly across projects and platforms, but projects can be at risk to be cancelled if they cannot reach a critical mass or keep their volunteers engaged [2]. Within the citizen science community, several researchers including Newman et al. [3] and Sørensen et al. [4] have pointed to games as an opportunity for doing citizen science. While the existent literature has mainly examined motivation, quality of player contributions, aspects related to content learning and education, and system/task design in online citizen science games [5], both player experience (PX) and quality of experience (QoE) of games of this type appears to be under explored.
In this paper, we focus on understanding the QoE of Quantum Moves (QM), an online game contributing to problem optimization in quantum physics [4, 6]. To do this, we conducted an exploratory study on the individual interaction with the game by players who took part in the study. This premise is consistent with the argument that the individual (socio-) psychological level is the constituent aspect of QoE [7]. The paper is structured as follows. First, we identify research gaps from existing research and then we will introduce the exploratory survey and the analytic methods we used. Then we present our findings. We conclude with a discussion and reflections on the implications of the findings for the game QM.
Player experience and quality of experience
In this section, we discuss the concepts of player experience (PX) and quality of experience (QoE). Both refer to the notion of experience—or experiencing—which has been defined as “the individual stream of perceptions (of feelings, sensory percepts and concepts) that occurs in a particular situation of reference” [8]. In terms of the application domain of this paper, experience results from the encounter of a player with a serious game. This encounter is what we can refer to as PX. PX has been a growing field of research in the last decade [9]. In the literature, there are a number of different understandings of the term PX in games. Here, we define PX as “the individual and personal experience of playing games” [7]. For Wiemeyer et al., PX can be described in terms of the characteristics of the interaction between a player and a game and is typically investigated during and after the interaction with games. These authors distinguish three levels of PX: (1) the (socio-) psychological level (individual experience), (2) the behavioral level, and (3) the physiological level. The individual and personal experience comprises a number of dimensions, including intrinsically motivated actions, ambivalence and openness to both procedure and outcomes, immersion, satisfaction and fun of gaming, among others. The behavioral level describes specific observable behavior (like laughing, smiling or frowning), while the last level describes physiological reactions (like increased heart rate or blood pressure) [7]. According to Wiemeyer et al., PX should be distinguished from player types, because the former denotes a transient and dynamic construct (state), while the latter denotes a more or less stable and static construct.
PX is different from QoE. The notion of experience—and similarly that of PX—as defined earlier does not imply an evaluation of the quality of experience. As noted by Raake and Egger [8], evaluating the quality of experience results from cognitive processes related to the experience. They proposed a definition in which QoE results from the individual evaluation of personal expectations and needs with respect to the enjoyment or annoyance experienced while interacting with an application, a service or a system. As the application domain in this study is a serious game, this definition is adequate because it goes beyond the mere use of the game and embraces non-instrumental, hedonic and affective aspects that are central to understanding the quality of experience when playing the game. In addition, the inherent subjective and individual character and the context-dependency of QoE are also implied by this definition [8].
QoE is a highly complex and subjective concept to evaluate. QoE concerns the individual evaluation of a system performance. Such evaluation is influenced by contextual factors, culture, psychological characteristics and individual expectations and needs, among others, with respect to a system. In the context of games, previous studies have mostly focused on single dimensions of the experience with the game. Positive dimensions have included enjoyment [10], flow [11, 12], immersion [13], and effect and control [14]. Negative dimensions have included boredom and tension relating to the difficulty of a game or the competition a player feels. Apparently, game research has not paid much attention to these negative dimensions, even though they contribute to the challenge of a game and are likely to influence the QoE [15]. A reason for the focus on single dimensions might rest in the need for a multi-disciplinary and multi-methodological approach to the study of the QoE [16] Methods to assess the experience with a game include psychophysiological player testing, eye tracking, game logs, surveys and questionnaires, among others [17].
QoE in online citizen science games
Online citizen science games are a subset of citizen science projects conducted entirely via the Internet, in which citizens play an enjoyable game and at the same time generate useful data by performing a computation or task which cannot (yet) be performed by computers alone [18]. Both full-fledged games and elements of games or ‘gamification’, where game-related and external motivators (such as points, leaderboards, or achievements) are applied to non-game contexts [19], are used in citizen science to develop applications that invite citizens to collect data, annotate images or documents, or solve difficult scientific problems. Research on the use of games in citizen science is relatively novel but expected to grow rapidly (for a more comprehensive review on online citizen science games, please see [20]). Most of the existing studies thus far have explored the value of games for motivating and engaging volunteers (e.g., [2, 21,22,23,24]). A systematic mapping review of empirical studies on citizens involved in research through gaming showed that, besides motivation, the topics examined included the quality of player contributions, aspects related to content learning and education, and system/task design [5], but not PX and/or QoE of citizen science games. Even though studies of PX have reported that the various features designed into games can influence experience significantly and can further result in the retention of players (e.g., the speed of player interactions and time pressure affected game immersion [25]), there has been very little research looking specifically at citizen science games [26]. A search of the Scopus and Web of ScienceFootnote 1 databases retrieved no articles about QoE of citizen science games and only two empirical studies of PX in citizen science games. The first study was conducted by Prestopnik and Tang [26], who examined how two different reward structures, one point-based and the other story-based, impacted PX in two citizen science games. Their findings showed that participants strongly preferred the story-based game over the points-based game because in the former the focus on story-motivated activities and rewards made the citizen science task more enjoyable and gave participants various reasons to continue to play. The authors argued that a story-based game allows players to engage with an entertainment-oriented game world that only occasionally requires them to act as a “citizen scientist,” unlike a point-based game where players earn points and other rewards specifically for engaging with the science, with these activities comprising the majority of the game experience. The second study was conducted by Hess et al. [27], who conducted a small user study (20 participants using convenience sampling) with a short version of the Game Experience Questionnaire to measure PX through the improved mechanics of a computer game aimed at solving protein sequence alignments. The results indicated that players had fun playing the game and also had a substantial feeling of competence and challenge in the game. The majority of the players were neither bored nor found the game tiresome.
After conducting the exploratory research on QoE for the game QM, we recognized some findings that could help to address the research gap in the existing literature regarding motivation and engagement of citizen science game players.
The game: Quantum Moves (QM)
QM [4, 28] is a dynamic puzzle game about single-atom transport problems related to the development of a so-called quantum computer [6]. QM is the first citizen science game developed by ScienceAtHome (SAH). SAH is a research group working at Aarhus University where scientists and game developers are developing a platform which includes a variety of tools and games for research in different fields in citizen science. QM presents players with the task of finding the optimal (efficient-yet-quick) way of moving an atom (visualized as a liquid-like substance in the game due to its interpretation as a quantum wave phenomenon) from an initial location to a target area (Fig. 1). The path (‘hit’) created by the player is stored in a database and post-processed with further optimization as detailed in [4]. QM has been played by over 250,000 players around the world, generating over 6 million solutions to the 23 puzzles (levels). For each trial of a QM level the player gets a score and a star rating (zero to three stars). The score positions players on a leaderboard ranking as a motivation to improve their performance. The user research presented in this paper builds on two previous studies of QM. In 2014, Lieberoth et al. [29] conducted a study with 1190 players of QM characterizing the types of players who play the game. Lieberoth et al. found that there were two types of players who play QM, the ‘heroes,’ who play methodically, are intrinsically motivated and connect with the science behind the game and the casual players whose motivations and skill levels are fluid. In 2017, Pedersen et al. [28] conducted a study with 150,000 players of QM inquiring about the influence of different types of leaderboards on motivation and performance in the game.
Data and methods
As part of our research on the QoE of QM, we constructed a 20-question post-play questionnaire to assess players’ motivations for playing the game and possible improvements for future updates of the game. Initially, we decided to build a custom questionnaire as the main goal of the research was to assess the game from a development perspective while addressing the challenges that pose to study citizen science games. That is, questionnaires targeting commercial entertainment games inquire for PX from a marketing perspective, and questionnaires targeting learning games inquire for the relationship between gameplay experience and content learning. Players of citizen science games can be driven to play by different motivations not explored in those questionnaires, such as interest in helping the development of science or the scientific topic behind the game.
The authors who worked at SAH and were familiar with the study population defined and discussed the questions. The purpose was to gather information about the way players relate to the game. Between March and April 2017, we distributed a cross-sectional, self-administered online questionnaire to approximately 25,600 people included in the SAH mailing list. The questionnaire included two parts (Table 1). The first part consisted of 13 compulsory questions, of which three were only for participants who play games regularly. The first four questions were about general gaming habits and were included for enquiring into how gaming preferences were related to recruitment and retention in QM. The next six questions enquired about player perceptions of the game, and player motivations before and during gameplay. The last three questions probed inquired on the experiences with the SAH platform in general to investigate the efficacy of the infrastructure around the game. The second part of the questionnaire was voluntary and contained four questions about sociodemographic player characteristics and one free text question (Q19) enquiring into the individual and personal PX, which is considered the constituent dimension of PX [7]. In this paper, we focus on the results from the analysis of the responses to the free text question, while characterizing the players of the game by using the sociodemographic data provided. The purpose of these analyses was to provide feedback to the project from a scientific, design, and engagement perspective; which in turn generated a set of limitations of generalizing our findings. This study follows all the ethical regulations for research in the Social Sciences within the European Union [30]. Survey respondents were asked to sign an electronic consent form. The survey data did not contain personal identifiable data and was stored in a password protected electronic format.
In addition, Q20 prompted our respondents to take the Bartle test for players who play MUDs (designed by Erwin Andreasen and Brandon Downey). The Bartle test is of wide use in the game design industry as it gives a general perspective on what players like from the game and how players relate to it, allowing for balancing and improving retention on a game based on its target population. However, due to the reported limitations of the test [31] and the lack of accessibility to its psychometric proprieties, we decided to do a basic report on this data. We highlight that out of the 743 respondents, 630 took the Bartle test and reported back to us, suggesting that these type of self-assessment tools are of interest for our players. Many players reported belonging to more than one category (probably because the test reports on more than one category). The category of ‘Explorer’ was the most prevalent (528 participants) followed by the categories of ‘Achiever’ and ‘Socialiser’, (126 and 115 participants respectively) and with only 32 players belonging to the category of ‘Killer’ (distribution of the players provided in “Appendix 1”). This suggests that players whose motivation is to explore might be the ones most attracted to QM.
We used two methods to analyze the survey data: (a) descriptive statistics for analyzing frequency and distribution of responses according to age, gender and occupation, and (b) iterative thematic analysis [32] for identifying themes within the responses to the free-text question (Q19) about what players liked or disliked from the game, what they would like to improve, and how they played the game and created strategies. Out of the 743 respondents, 121 (16%) did not answer Q19 at all, and 163 (22%) gave a response containing less than eight characters. Seventy-nine responses were longer than 50 words, with one response being 577 words long. The shortness of responses and the “bittiness” of data made the identification of patterns challenging. Two of the authors coded the entire corpus of the free-text responses, using a coding frame developed inductively and iteratively after discussion. The coding frame represented the topics and subtopics identified in the responses (see “Appendix 2”). Most responses were related to multiple codes. For example, a response could be associated with “Engagement” as well as “Strategy Making.” To validate the coding consensus between the two researchers, we followed the proportion agreement method by Campbell et al. [33], finding an agreement of 70%. Additionally, we used Mezzich’s Kappa [34] as an extra reliability method, finding an agreement better than chance (p < .05).
Results
Sociodemographic characteristics of participants
We obtained responses from 817 participants (3.2%) from a total of 25,587 people who received the invitation to participate in the survey. Eight-hundred-and seventeen participants filled the first part of the survey, from which 743 also filled the second part. Eighty-two percent (N = 607) of the respondents were male and 16% (N = 116) were female; 2% (N = 20) of the respondents did not disclose their gender. The most common age bracket was 26–35 years (N = 193, 26%) (Fig. 2). The majority of participants stated working full time (N = 310, 42%) or studying at school or university (N = 268, 36%) (Fig. 3). Lastly, 80% (N = 660) of participants reported playing games in general, while the remaining 20% reported that QM was an exception.
QoE and units of meaning
We summarized the frequency of topics identified in the responses to Q19. Although most of the answers described more than one topic and subtopic, several key topics and subtopics predominated, as depicted in Tables 2 and 3 (for a full view of the codes and units of meaning identified in the analysis please refer to the “Appendix 2”).
Additionally, we performed a co-occurrence analysis on matching pairs of topics. As Table 4 shows, “engagement” and “strategy making” appear to co-occur relatively often (23 times in total), suggesting that players who enjoyed the game the most were also those more focused on finding and creating strategies to solve the challenges of the game. As a close runner-up, the co-occurrence of “engagement” and “game design suggestions” accounts for the fact that players who engaged either positively or negatively with the game made suggestions for improving the game.
Finally, using a thematic analysis we identified recurrent topics and subtopics in the responses to question Q19. As noted earlier, the shortness of responses to this question and the “bittiness” of data, rather than a longer narrative, made the identification of patterns challenging. However, we identified three broad, cross-cutting, themes at the socio-psychological level, which are reflected on the results of the coding: (a) engagement; (b) learning, and (c) contribution to science. We found that aspects of the participants’ reported experience overlap across these themes. This is consistent with the complexity of construction of understandings and meaning of experiences, which are not built on isolated concepts but are relative to each other. To illustrate each theme, we selected a sample of poignant and representative responses from the survey.
Engagement This theme is defined here by how players perceived: their involvement to the game in terms of time, effort and interest; their balancing match between skills and challenges; their feelings when playing the game and about their achievements (or lack of); their appreciation of the game design features, the game purpose of the game, and whether it was valuable to them. The analysis of responses indicates that a positive—or negative—experience relates primarily with the aspects of engagement, which, in turn, is often associated with the ability to play successfully, the actions chosen to perform well, and how well the design of the game is perceived. The following four responses illustrate how engagement embodies both negative and positive types of experiences. In the following response, the co-occurrence of positive engagement and game design suggestions indicates an interest of the player to improve the game, through the lens of his/her own experience, and provide player support.
R01: “I liked the simplicity in the display and interface, if I were given the opportunity, i would try to add another layer to the game, would a 3d model be more accurate than a 2d in the quantum world? Perhaps more examples of quantum mechanics through mini games for players to better understand the attributes associated with the ‘liquid data.”
In the following excerpt, perceived enjoyment is associated with the challenge of the game, although the player feels being hampered by his/her limited knowledge of quantum physics:
R02: “I liked the challenge of the game. I think more help or explanation could be given early on with the option to read about how Quantum Moves relates to quantum physics. Explanation would be nice but I do concede I may not fully understand the physics anyway, that is why you made the game. [I] tried out a couple different strategies. Mostly i thought speed, displacement affected the movement. I considered playing around with oscillating the mover but I couldn’t find a easy to approach it systematically.”
This response is grounded in the observation made by the player about his/her poor understanding of physics. The player struggled to reconcile his/her appreciation for the game (“I liked the challenge of the game”) with the unsuccessful experience of trying different approaches based on a limited understanding of the game mechanics. Knowing quantum physics is not a prerequisite for playing Quantum Moves; however, learning a bit about the science can enrich the gameplay experience, as for R03 the player stated:
R03: “I liked being able to experiment and move around in random ways to see how it reacts, and I found the game more enjoyable after learning a bit more about quantum physics and having a better understanding of what i’m doing. Though I would like to see a more clean and simpler interface.”
This player used trial and error, the most frequent strategy reported by respondents (7.46%). Players seem to resort to this approach they do not understand the rules of the game. When using trial and error, players proceed mainly by exploring and manipulating elements of the game in an effort to sort out possibilities and to run across steps that might carry them closer to the solution. This behavior is most likely to be reported when players lack knowledge about the character of the solution, or when no single rule seems to underlie the solution.
Responses often offered an assortment of experiences, linking one or more elements of appreciation with unmet expectations. In R04, the player appreciated the representation of a complex problem in what s/he considers a casual game, but s/he ended up quitting the game early on:
R04: “It conveyed a complex problem with a simple visualization, the graphics were complex enough to give a sense of ‘high tech’ but still not distracting. I think it did a fantastic job in presenting the issue(s) and made the challenges compelling for a casual game. Not fun enough to return more than once since the task was quite monotonous and ‘special mindset’ type of problems.”
Learning This theme encompasses three main strands of learning identified in the responses: learning about the game (subsuming both learning about the purpose of the game and how to play the game), learning about quantum physics, and learning and developing one’s own skills. The theme is defined by how well players with various levels of experience and skills could approach and play the game, and how they felt about the opportunity to learn something about quantum physics and the how the game works. The following five responses illustrate the three strands in the theme. For example, R05 displays an appreciation for the challenge the player faced when playing the game, which s/he saw as an opportunity to learn to be creative and use deductive reasoning, without much scaffolds:
R05: “What I really liked about the game was that it takes some time to really figure out how to complete a level. And that the instructions gave you just a little hint about it, but the rest is about yourself, creativity and deduction skills.”
In contrast with this response, the player in R06 perceived an imbalance between the complexity of the game and the support provided, resulting in a steep learning curve:
R06: “Both too simple tutorials and challenging game, too steep learning curve.”
In most of the answers under the learning curve label (4.9% of the total answers), players reported being stuck in the game because the curve was not adapted to their abilities. Within this set of responses, we were able to identify several instances of frustration resulting in quitting the game. We also found also some instances of enjoyment of the challenge leading to learning about a difficult topic, as stated eloquently in R07 and R08:
R07: “It was really challenging and I loved that aspect. I like problem solving and trying to come up with new ways to solve problems. I also liked learning more about quantum physics so that was really a bonus!”
R08: “I liked the steep learning curve and how it creates insight in a visual way in quantum mechanics.”
When the challenges associated to the learning curve did not result in frustration, they were turned into appreciated learning opportunities. For example, in R09 the player engaged with “discovery learning” [35, 36], drawing on his/her own experience as a method through which s/he interacted with the game by exploring and manipulating objects and wrestling with strategies.
R09: “I liked the process of creating and refining strategies to improve the score on the level. I would try something weird to see what happened, then repeat the level while slightly modifying the strategy to see what change that may have made. I discovered that smooth moves are much better than sharp, jerky moves, and that any change in the speed of the mouse would induce some ‘waves’.”
Contribution to science This theme is defined by players’ reported intrinsic interest in contributing to science, how clearly players perceived the scientific purpose of the game and how they felt about the interaction with scientists at SAH. The analysis of responses indicates that the interest in contributing to science is often combined with aspects of engagement. We discuss a sample of four responses to illustrate this theme. Of the respondents, 5.9% expressed their interest in contributing to science, as this player stated clearly:
R10: “I heard of this research on the NPR Program Science Friday. I love the idea of crowdsourced research and have a laypersons interest in the theories of quantum physics.”
To feel they are part of the scientific project, some of these respondents also made demands to access the gameplay data and receive reports, and wanted the ability to communicate with the scientists at SAH, as the following two quotes illustrate:
R11: “To improve the game experience for me would be to have periodic reports on the research and any insights, progress or problems made regarding the objectives. This would enhance the feeling of being part of the research which is the main reason I played.”
R12: “I like seeing the data being collected, it made it clear that i was actually doing something which could be analyzed. I would have liked to know more about what was being collected, though.”
Responses also revealed that the interest in contributing to science could be discouraged by a lack of understanding of the relationships between the game and its scientific purpose, which can result in players quitting the game. Below is a selection of relevant answers:
R13: “Explain how the game works, make a link with the part of physics which it concerns, it was all a bit unclear what is really all about. It worked for me but have not a clue what you accomplished with all data that is gathered. The idea to turn to the public is great, but explain more.”
R14: “Fun but frustrating, it would be nice to have a better understanding of how the data helps real life research. Enjoyable game, but I lost interest due to the perceived disconnect from the science behind it.”
The qualitative themes emerging from Q19 were also supported by the quantitative analyses of Q6, Q7, and Q10. The answers of Q6 inquiring on why the player choose QM indicate that the main motivation was the scientific nature of the game, particularly the interest on contributing to and learning from the science behind the game (Fig. 3).
In addition, the answers to Q7 highlight the main motivations for the players to contribute to science, to learn about the science in the game (including problem solving in a scientific environment), and to learn about the science in the game (including problem solving in a scientific environment), and the challenge posed by the game (Fig. 4).
The answers to Q6 and Q7 create a cross-sectional cut on the three main themes we found as motivators to play QM (Engagement, Learning, and Contribution to Science), highlighting even more relevance for exploring these concepts when researching PX in Citizen Science games.
Finally, from Q10 we can take a quick view of which are the most common ideas/words associated with the game. The 10 most common words (as depicted in Fig. 5) depict that most people found the game to be interesting, associating it with good feelings such as ‘good’ or ‘cool.’ However, ‘challenge’ and ‘frustration’ also fulfill a role in the engagement of people.
Discussion
Given the limited literature on QoE of online citizen science games, we engage mainly with relevant literature in game studies to present the underlying meaning of our results.
Main themes
While the initial aim of our study was to explore PX of QM for improving the game design, our findings prove compelling for understanding gaps in the QoE studies involving online citizen science games. We identified three main themes (Engagement, Learning, and Contribution to Science) helping to understand this experience. As regards to engagement, multiple interrelated factors, including game design (particularly levels and features, the appeal of the task, and the strategies that players can choose within the game) appear to influence longer-term engagement. These findings were quantitatively supported by the answers to Q6 and Q7 (see Figs. 3, 4), and resonate with Tinati et al. [37], who found that memorability/interestingness of subjects, speed, task difficulty, and amount of feedback provided, influence sustained engagement. New or improved game design features for players to use or to overcome seem to influence longer-term engagement [38]. For example, the request by a player of “more examples of quantum mechanics through mini-games for players to better understand the attributes associated with the ‘liquid data” indicates a wish for re-designing, turning a complex problem into smaller problems [39] without reducing the importance of the project’s scientific purpose.
In several other instances of the survey responses, players suggested the implementation of new or improved features for enhancing the gameplay experience, including adding sound or introducing a story in the game. Our interpretation of the co-occurrence of engagement and game design suggestions we found in the data is that features influence the perception of the interestedness [37] of the game. The finding that QM is perceived as monotonous and uninspiring by several players is consistent with what Prestopnik and Tang [26] argue about citizen science tasks as being mundane or repetitive sometimes. Making challenging scientific tasks interesting, worthwhile and achievable is critical for a successful citizen science system [40], as it impacts the decision of players to leave or stay.
Our findings also indicate that the strategies used by players seem to relate to the type of engagement and player decision to stay. The prevalent use of trial-and-error can arise from various sources. For example, it can arise from the perception of the game as “hard,” in the sense that there are several choices that appear to be good, even when they are not [41]. Instances of player responses indicate that this approach is more common when players experiment with various options in the game. Nonetheless, trial-and-error has proven to be a challenging concept to define, as the term can be associated with two different definitions. The first definition [42] refers to a learning process involving inductive reasoning. In this process, the person interacts with the problem as a way for probing it, analyzing the results of their interactions to form a mental model and infer effective rules for solving a problem. The second definition refers to trial-and-error as solving a problem using ‘brute force’ (i.e., trying every possible combination coming to a solution without reflecting or learning from it). Unfortunately, in the current research, we could not discern the extent to which the meanings could correspond to the first or the second connotation due to the short length of the units of analysis. More research is required to clarify this aspect.
When examining learning more closely, we found that learning about the game, in particular, learning how to play QM, can be “hard” for several players. Players reported that the learning curve can be steep, thus driving away players who do not want to exert big effort to the game, as well as players who are interested in the game and may be willing to make some effort but get frustrated because they have a low level of ability. Our analysis suggests the importance of achieving a feeling of flow [11] by balancing the skills required to solve a task, the rewards from solving it, and the continuous improvement of the skills required to solve it. The results also indicate that the provision of in-game support such as tutorials or guided task interfaces could influence the choice of strategies by equipping players with a better understanding of the game mechanisms, which could further increase interest in the game and support longer-term participation.
Several players were interested in learning more about quantum physics but did not report whether they thought they had learnt something about it. These findings were both supported by the qualitative thematic analysis and the quantitative answers to Q6 and Q7 (see Figs. 3, 4). According to Wiemeyer et al. [7], game developers should consider whether players may actually be interested in learning a subject, for example, learning about the environment of serious games, because it may add to the attractiveness of games. It should be noted that learning quantum physics is not at the forefront of the design objectives of QM and our results do not provide enough evidence of whether, through gameplay, players concurrently develop an understanding of the science behind the game. On the contrary, our data provide evidence of players who did not understand whether they were engaged in a scientific practice. We contrast this result with [43] study of Foldit (a citizen science game that gets people to understand protein folding through a collaborative puzzle-solving approach [www.fold.it]). The Foldit Citizen Science game could be an entryway for some players into the science of proteins as well as scientific practice. This may arise from the fact that the embodied activity of playing Foldit mirrors one of the activities in a biochemistry lab [43]. In QM, players who enter the game are unlikely to learn the science behind the game and struggle to understand its scientific significance. Arguably, this could be related to the complex nature of quantum mechanics: QM cannot afford to mirror the ‘real’ quantum behavior; instead, it provides a simplified representation of the problem for players while keeping the difficult components inside for the physicists. While QM does not constitute an entryway into quantum physics, it is an opportunity to develop skills, use intuition and think outside the box. This is especially evident in those responses where players report experimenting with the game features to see what they can achieve. The appreciation of this opportunity for trying and discovering things complements the enjoyment of the mental challenge created by the game and the pleasure of seeing improvements in one’s skills over time.
As regards to contributing to science, this interest is an important intrinsic motivation for QM players. This is particularly visible in the answers to Q6, Q7, and Q10 (see Figs. 3, 4, 5). While this anticipated result is in line with previous studies [2, 22, 24, 44], we emphasize here that this aspect can contribute to enrich the experience of a serious game. The use of games in citizen science suggests that game designers tend to rely on players’ civic zeal to trigger the excitement of ‘participatory science’ and the rewarding feeling of achieving something in a field considered prestigious by the general public [45].
Player motivation
Based on the analysis of the three themes, we present two main aspects that need consideration for improving the design of QM: contribution to science and extrinsic rewards.
Contribution to science When players make an effort to contribute to science, they wish to know what happens “behind the scenes,” either for better understanding the scientific problem they are asked to solve, for understanding the extent of their contribution, or for helping improve the game (Tables 2, 3, 4). We suggest the creation of an online community (e.g., Facebook or Reddit group) where the scientific team and developers participate actively in the discussion about the scientific background of QM, the way their contributions help achieve the specific scientific goal, and the upgrades that can be implemented to make the game more fun. Additionally, we suggest the involvement of players in scientific outreach, by reporting via newsletters how they have helped achieve the scientific purpose.
Extrinsic rewards Some elements, such as leaderboards, help engage certain types of players, but are not extensible to every player. Moreover, being an artificially introduced motivator it quickly wears weak, so it does not engage players for long periods of time [26, 46]. However, in our research, we found that game elements can be engaging to a certain extent for those players whose main motivation is not to help science, but to have a game experience. The responses to the free-text question of the survey suggest that not only were these players happy with the game elements used in QM (i.e., leaderboards and achievement stars) but also asked for more of these elements (e.g., unlockable content). In this case, the players’ qualitative experience contrasts with other findings [28, 46, 47] on the absence of correlation between player motivation and performance, and game elements. However, these suggestions are valuable for game designers because they seem to give players a subjective feeling of progression within the game, appropriation and agency, and engagement with the community, which might contribute to long-term engagement with the game.
Further steps
Although this exploratory study contributes to understanding the QoE of QM (therefore, it is limited to specific experience and demographical data), we consider some of our findings to be an important stepping-stone for understanding the QoE in other citizen science games. A further study mapping experience data onto player demographics would allow a richer representation and optimization of aspects of the QoE of a single player or a group of players with similar characteristics. In relation to the generalizability of our results, two other limitations need to be addressed. First, as the focus of this survey was on the experience of players participating in QM, our findings are informed by—and limited to—the respondents who completed the survey. These respondents were individuals who were interested in this study and may not be representative of the entire population of players. Arguably, our results are somehow biased towards the players who actually liked the game and/or to participate in citizen science projects. Second, our results are based on the analysis of a single game, although we have compared them and shown similarities with findings from previous studies of citizen science projects.
The reach of our current research allows us to point to specific aspects that are particularly present in the QoE of citizen science game players and which are not present in other types of games such as commercial or learning games [48, 49]. Acknowledging the presence of these topics in citizen science games can improve the construction of future scales or subscales used in PX evaluation. Construction and standardization of these scales will allow for (a) a better evaluation of citizen science projects in the aspects of engagement, learning, and contribution to science; and (b) designing and developing better tools for engaging citizen scientists as a particular population type.
As further research on the subject, we suggest that our findings are re-tested using standardized questionnaires [7, 49]. Also, we suggest the application of our coding schema (see “Appendix 2”) to the analysis of open questions in different citizen science games as a way to assess the occurrence of the three cross-cutting topics (i.e. engagement, learning, and contribution to science) found in the current research, as well as other possible topics that are particular of citizen science games.
Conclusion
We designed this survey study to investigate the participation in and the QoE of QM, an online game reproducing a problem optimization in quantum physics. Understanding how players participate and experience a citizen science project provides another lens for examining such projects. We found that most of the contributions were provided by a small share of players, although several players return to QM after long periods of absence. Regarding the QoE, our findings indicate that the perceived QoE goes beyond fundamental design issues [50], including affective aspects, such as having fun, feeling rewarded for learning something and for contributing to science; while experiencing other emotional responses, such as getting frustrated for not progressing in the game. These results highlight the importance of design interventions that increase both the perceived and factual effectiveness and efficiency of task performance. The perceived effectiveness and efficiency being crucial for the interaction and thus for the quality of experience, while creating something that players find meaningful and scientifically relevant, influencing both the attraction and retention of the game. The factual effectiveness and efficiency being important for generating more and better-quality scientific data. The trade-off between the need for collecting sound scientific data and ensuring gamers a rewarding experience has been recognized as a challenge not only for designers of serious games, but more generally in all citizen science projects started by domain experts [51].
In addition, the fact that the game attracted such a broad audience, either by awakening the curiosity of people wanting to try a different type of game, or by attracting those who wanted to help science, proved to be a powerful tool for collecting scientific data. Data collected in this way was rich both transversally (many different people playing) and longitudinal (few players playing for a long time and trying to get the best solutions) [4]. The findings of this research will help not only the scientists behind QM to improve the QoE in the game so that more and better answers can be generated, but also future studies using Citizen Science games.
Notes
We searched the databases Scopus and Web of Science. Both databases are well-established, multi-disciplinary research platforms, including a wide variety of peer-reviewed journals, and they are being updated regularly. We chose these two databases to ensure relevant papers are included, since it is possible that one database omits relevant research. The Scopus database retrieved 11 articles when searching for TITLE-ABS-KEY (“citizen science” AND “gam*” AND “player experience”). The oldest article dates from 2014. No articles were found when searching (TITLE-ABS-KEY (“citizen science” AND “gam*”)) AND (“quality of experience”). The Web of Science returned one article from 2015 with a similar search string: TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience*”) and no results when searching TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience”). Search conducted in 4th September, 2019.
References
Bonney R, Shirk J, Phillips T et al (2014) Next steps for citizen science. Science 343:1436–1437. https://doi.org/10.1126/science.1251554
Tinati R, Luczak-Roesch M, Simperl E, Hall W (2017) An investigation of player motivations in Eyewire, a gamified citizen science project. Comput Hum Behav 73:527–540. https://doi.org/10.1016/j.chb.2016.12.074
Newman G, Wiggins A, Crall A et al (2012) The future of citizen science: emerging technologies and shifting paradigms. Front Ecol Environ 10:298–304. https://doi.org/10.1890/110294
Sørensen JJ, Pedersen MK, Munch M et al (2016) Exploring the quantum speed limit with computer games. Nature 532:210–218
Magnussen R (2017) Involving lay people in research and professional development through gaming: a systematic mapping review. In: Proceedings of the 11th European conference on games based learning, ECGBL 2017, pp 394–401
Weitenberg C, Kuhr S, Mølmer K, Sherson J (2011) A quantum computation architecture using optical tweezers. Phys Rev A. https://doi.org/10.1103/PhysRevA.84.032322
Wiemeyer J, Nacke L, Moser C, Floyd’ Mueller F (2016) Player experience. In: Dörner R, Göbel S, Effelsberg W, Wiemeyer J (eds) Serious games. Foundations, concepts and practice. Springer, Cham
Möller S, Raake A (2014) Quality and quality of experience. Advanced concepts, applications and methods. In: Möller S, Raake A (eds) quality of experience. Springer, Berlin
Canossa A, Drachen A (2009) Patterns of play: play-personas in user-centred game development. In: Proceedings of DiGRA 2009. DiGRA
Klimmt C (2003) Dimensions and determinants of the enjoyment of playing digital games: a three-level model. In: Proceedings of level up - 1st international digital games research conference 2003, Netherlands
Csikszentmihalyi M (1990) Flow. The psychology of optimal experience. Harper & Row, Manhattan
Sweetser P, Wyeth P (2005) GameFlow: a model for evaluating player enjoyment in games. Comput Entertain CIE Theor Pract Comput Appl Entertain 3:3. https://doi.org/10.1145/1077246.1077253
Brown E, Cairns P (2004) A grounded investigation of game immersion. In: CHI’04 extended abstracts on human factors in computing systems. ACM, Austria, pp 24–29
Klimmt C, Hartmann T, Frey A (2007) Effectance and control as determinants of video game enjoyment. Cyberpsychol Behav 10:845–848. https://doi.org/10.1089/cpb.2007.9942
Poels K, de Kort Y, Ijsselsteijn W (2012) Identification and categorization of digital game experiences: a qualitative study integrating theoretical insights and player perspectives. Westminst Pap 9:107–129. https://doi.org/10.16997/wpcc.153
Le Callet P, Möller S, Perkis A (series ed) (2013) Qualinet white paper on definitions of quality of experience (QoE) and related concepts [white paper]. Retrieved from http://www.qualinet.eu/
Nacke L, Drachen A, Göbel S (2010) Methods for evaluating gameplay experience in a serious gaming context. Int J Comput Sci Sport 9:40–51
Law E, von Ahn L (2011) Human computation. Morgan & Claypool, San Rafael
Deterding S, Sicart M, Nacke L, et al (2011) Gamification: using game design elements in non-gaming contexts. In: CHI’11 extended abstracts on human factors in computing systems. ACM Press, Toronto, pp 2425–2428
Ponti M, Hillman T, Kullenberg C, Kasperowski D (2018) Getting it right or being top rank: games in citizen science. Citiz Sci Theory Pract 3:1–12. https://doi.org/10.5334/cstp.101
Bowser A, Hansen D, He Y, et al (2013) Using gamification to inspire new citizen science volunteers. In: Proceedings of the companion publication of the 17th ACM conference on computer supported cooperative work and social computing. ACM Press, Toronto, pp 18–25
Curtis V (2016) Motivation to participate in an online citizen science game: a study of Foldit. Sci Commun 37:723–746. https://doi.org/10.1177/1075547015609322
Greenhill A, Holmes K, Woodcock J et al (2016) Playing with science: exploring how game activity motivates users participation on an online citizen science platform. Aslib J Inf Manag 68:306–325. https://doi.org/10.1108/AJIM-11-2015-0182
Iacovides I, Jennet C, Cornish-Trestrail C, Cox A (2013) Do games attract or sustain engagement in citizen science? A study of volunteer motivations. In: CHI’13 extended abstracts on human factors in computing systems. ACM Press, Paris, pp 1101–1106
Cox A, Cairns P, Shah P, Carroll M (2012) Not doing but thinking: the role of challenge in the gaming experience. In: Proceedings of CHI’12 SIGCHI conference on human factors in computing systems. ACM, Texas, pp 79–88
Prestopnik N, Tang J (2015) Points, stories, worlds, and diegesis: comparing player experiences in two citizen science games. Comput Hum Behav 52:492–506. https://doi.org/10.1016/j.chb.2015.05.051
Hess M, Wiemeyer J, Hamacher K, Goesele M (2014) Serious games for solving protein sequence alignments—combining citizen science and gaming. In: Göbel S, Wiemeyer J (eds) Games for training, education, health and sports. Springer, Berlin
Pedersen MK, Rasmussen N, Sherson J, Basaiawmoit R (2017) Leaderboard effects on player performance in a citizen science game. In: Proceedings of the 11th European conference on games based learning
Lieberoth A, Pedersen MK, Marin A et al (2014) Getting humans to do quantum optimization: user acquisition, engagement and early results from the citizen cyberscience game quantum moves. Hum Comput 1:219–244. https://doi.org/10.15346/hc.v1i2.11
Directorate-General for Research Science, Economy and Society (2010) European textbook on ethics in research. Publication office of the European Union, Luxembourg
González Mariño JC, Cantú Gallegos ML, Camacho-Cruz HE (2018) Redesigning the Bartle test of gamer psychology for its application in gamification processes of learning. In: Proceedings of the 12th international multi-conference on society, cybernetics and informatics. IIIS, Florida, pp 35–40
Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa
Campbell J, Quincy C, Osserman J, Pedersen O (2013) Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol Methods Res 42:294–320. https://doi.org/10.1177/0049124113500475smr.sagepub.com
Eccleston P, Werneke U, Armon K et al (2001) Accounting for overlap? An application of Mezzich’s kappa statistic to test interrater reliability of interview data on parental accident and emergency attendance. J Adv Nurs 33:784–790. https://doi.org/10.1046/j.1365-2648.2001.01718.x
Stokke A (2015) What to do about Canada’s declining math scores. C. D. Howe Institute Commentary. https://doi.org/10.2139/ssrn.2613146
Dean D, Kuhn D (2006) Direct instruction vs. discovery: the long view. Science Education 91(3):384–397. https://doi.org/10.1002/sce.20194
Tinati R, van Kleek M, Simperl E, et al (2015) Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, Seoul, pp 4069–4078
Habgood J, Overmars M (2006) The game maker’s apprentice. Game development for beginners. Apress, Berkeley
Seaborn K, Fels D (2014) Gamification in theory and action: a survey. Int J Hum-Comput Stud 74:14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006
Franzoni C, Sauermann H (2014) Crowd science: the organization of scientific research in open collaborative projects. Res Policy 43:1–20. https://doi.org/10.1016/j.respol.2013.07.005
Elias G, Garfield R, Gutschera R et al (2012) Characteristics of games. MIT Press, Cambridge
Azwihangwisi M (2015) Piaget’s theory of human development and education. Int Encycl Soc Behav Sci 18:125–132
Horstman T, Chen M (2012) Gamers as scientists? The relationship between participating in foldit play and doing science. In: Presented at the American educational research association (AERA), Canada. Retrieved from https://www.researchgate.net/publication/258294401_Gamers_as_Scientists_The_Relationship_Between_Participating_in_Foldit_Play_and_Doing_Science
Jennett C, Kloetzer L, Schneider D et al (2016) Motivations, learning and creativity in online citizen science. J Sci Commun 15:1–23
Lafourcade M, Joubert A, Le Brun N (2015) Games with a purpose (GWAPs). Wiley, New York
Prestopnik N, Crowston K, Wang J (2017) Gamers, citizen scientists, and data: exploring participant contributions in two games with a purpose. Comput Hum Behav 68:254–268. https://doi.org/10.1016/j.chb.2016.11.035
Skarlatidou A, Ponti M, Sprinks J et al (2019) User experience of digital technologies in citizen science. J Sci Commun. https://doi.org/10.22323/2.18010501
Brockmyer J, Fox C, Curtiss K et al (2009) The development of the game engagement questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634. https://doi.org/10.1016/j.jesp.2009.02.016
Johnson D, Gardner J, Perry R (2018) Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ). Int J Hum-Comput Stud 118:38–46. https://doi.org/10.1016/j.ijhcs.2018.05.003
Preece J (2016) Citizen science: new research challenges for human–computer interaction. Int J Hum-Comput Interact 32:585–612. https://doi.org/10.1080/10447318.2016.1194153
Kim S, Robson C, Zimmerman T, et al (2011) Creek watch: pairing usefulness and usability for successful citizen science. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’11). ACM Press, New York, pp 2125–2134
Acknowledgements
This work was supported through funding from Marianne and Marcus Wallenberg Grant No. 2013.0020, Lundbeck Foundation, Grant No. R139-2012-12633, Carlsberg Foundation Grant No. CF18-0019, and John Templeton Foundation Grant No. 60969. J.F.S. Acknowledges funding from the ERC, H2020 Grant No. 639560 (MECTRL) and the Synakos foundation.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
Distribution of players according to the player types as assessed by the Bartle Test (Q20). NA stands for the players who could not take the test. The total sum of players per category is more than the players who answer the question; this is because several players reported that they belong to more than one category.
Appendix 2
Coding schema of player responses to question Q19
Topics and subtopics (indented) | Definition |
---|---|
Not defined—no response | Players did not answer the question, or their answers were irrelevant |
Game type | Players use a name that defines the game as a genre and explain why they like it or not |
Casual game | Players state the game is a casual game or they play it casually |
Zen game | Players state the game is a Zen game or having calming properties |
Puzzle | Players state the game is a puzzle |
Single player game | Players state the game is a single-player game |
Distinct game | Players state the game is different from other games on the market and/or other citizen science games |
Educational effects | Players state the game has educational effects, or it is good for learning/teaching science. |
Interest in learning | Players state the game taught them something about science |
Generating curiosity | Players state the game generated curiosity for them to learn more |
Engagement | Assigned to indicate the way players engaged with the game |
Enjoyment | Players state they have enjoyed the game |
Monotony | Players state the game is monotonous or repetitive |
Boredom | Players state the game is boring |
Frustration | Players state the game is frustrating, either for its mechanics or for problems with the game build |
Challenge | Players state the game is challenging, either in a positive or a negative way |
Confidence | Players state they are confident when solving the problems proposed by the game |
Modes of learning | Applied when players mention how the game engaged them in learning, or they suggest a way in which they can learn about the game or from the game |
Vicarious learning | Players would like to see other players’ solutions to learn better ways to solve problems |
Interest in quantum mechanics | Players state they have learned more about the specific field of science of the game |
Game design suggestions | Applied when players suggest design ideas for improving the game |
Graphic design suggestions | Players suggest changes to the graphic aspects of the game |
Opportunities for player customization | Players state they would like to be able to change certain features of the game ad libitum (e.g., re-skin the game) |
Send reminders | Players state they would like to receive periodic reminders to play the game |
Sound design suggestions | Players suggest sounds designs for the game, such as auditory cues for success or background music |
New mechanics suggestions | Players suggest new or improved mechanics for the game |
Enlarge game | Players suggest developing the game further, for example by adding levels |
Improving tutorials | Players state tutorials should be improved, or they evaluated the current tutorial as insufficient |
Influence of leader board on motivation | Players evaluate how the leaderboard influenced their motivation when playing the game |
Interest in contributing to citizen science | Players state their interest in the game for their contribution to citizen science |
Intuitive (easy to understand) | Players state the game and game mechanics were easy to understand, or they did not have to make mental effort to understand it and play it |
Intuitive solution | Players state the game was easy to play and not much knowledge was required to solve the problems |
Knowledge representation | Players state the way the game represents information on quantum physics makes it easy to understand this topic |
Lack of game feedback | Players state not having enough feedback from the game |
Lack of information regarding quantum physics (scientific topic) | Players state the game lacked of more and deeper information about quantum physics |
Lack of interaction with the scientific team and the game developers | Players state the need for more interaction and conversations with the team of scientist and developers behind the game to better understand how the data gathered is being used |
Lack of understanding | Applied when players state not being able to understand certain aspects of the game |
Lack of understanding of the game | Players state they did not understand the game |
Lack of understanding of the game mechanics | Players state they did not understand the game mechanics |
Lack of understanding of the procedural understanding | Players state they did not understand the procedures required to solve the problems |
Lack of understanding of the UI interface | Players state they did not understand the user interface or they were confused by it |
Lack of understanding of the purpose of the game | Players state they did not understand the purpose of the game apart from the partial goals in each level |
Learning curve | Players make a statement on the learning curve of the game. Statements about game progression, level progression, and blockages in the gameplay fall into this category |
Parallel thinking | Participant state having to think in non-conventional ways to solve the problems proposed in the game |
Player types | Players state they engage or not engage with the game due to a personal trait |
Competitive | Players state they are competitive or like competition |
Non-competitive | Players state they are not competitive or dislike competition |
Not a gamer | Players state they are regular players of video games |
Puzzler | Players state they like solving puzzles |
Psychophysical constraints | Players state having problems performing tasks requiring fine motor skills |
Relationship between game and the scientific topic (QP) | Players state that it would be a gain for the game if it highlighted more its scientific relevance |
Relevance of controllers | Players state controllers were relevant for the implementation of the strategy, usually in a negative way |
Relevance of gaming experience | Player state that their experience playing games helped them solve the challenges proposed by the game |
Relevance of science knowledge | Player state their knowledge of science helped them solve the challenges proposed by the game |
Rewards other than the score system | Players state they find rewarding aspects of the game that are different from the score system, e.g., learning science or helping the development of a scientific tool |
Science dissemination | Players state the game is good for science dissemination |
Social aspects | Players state interest in the inclusion of social aspects in the game |
Strategy making | Applied when players talked about the strategies they used to solve the challenges proposed by the game |
Lack of strategy | Players state not having a strategy |
Scientific reasoning/experimentation | Players state experimenting and finding patterns which allow them to overcome the game challenges |
Serendipity | Players solved the problem by chance or luck |
Trial and error | Players solved the game challenges using trial and error |
Technical issues | Applied when players state having technical issues with the game |
Problems with responsiveness and accuracy | Players state having problems with the responsiveness or the accuracy of the game depending on the device they used |
Game bugs | Players reported bugs in the game |
Relevance of OS | Players reported incompatibility of the game with some operative systems |
Relevance of console | Players state the device (e.g., PC or mobile device) they used for playing influenced their performance |
Problems with data transmission | Players reported to stop playing because data transmission was very costly |
Too busy to play | Players state not playing much because they had other activities to do |
Rights and permissions
About this article
Cite this article
Díaz, C., Ponti, M., Haikka, P. et al. More than data gatherers: exploring player experience in a citizen science game. Qual User Exp 5, 1 (2020). https://doi.org/10.1007/s41233-019-0030-8
Received:
Published:
DOI: https://doi.org/10.1007/s41233-019-0030-8