Keywords

1 Introduction

There is palpable interest in the potential of educational games to engage students and enhance learning. Teachers are particularly excited about the use of educational games in their classrooms, with a high percentage having their students use games for learning at least once a week (55% of 513, according to [9]). At the same time, young people are playing computer games at an ever increasing rate. For instance, [15] reported a playing frequency of 4.9 to 5.8 h per week for children age 7 to 12. The enthusiasm for educational games, combined with the increasing interest of young people in playing computer games, is leading to a revolution in education, with educational games in the forefront.

At the same time, scientific research has started to provide evidence that games can be effective for learning [2, 4, 14, 19, 31]. For instance, research has shown the benefits of learning decimal mathematics with an educational game called Decimal Point, which was designed based on theory and evidence about common student misconceptions [8]. In a study involving more than 150 middle school students, Decimal Point led to significantly more learning and was self-reported by students as significantly more enjoyable than a more conventional computer-based tutoring approach [20]. Other studies, in the areas of mathematics [11, 22], science [1, 12], and language learning [29, 32], have shown similar learning and/or engagement benefits for educational games.

The search is now on for the specific features of games that lead to engagement and learning benefits, as well as how we can best leverage those features to maximize the potential of educational games [2]. Potential game features to explore include game challenge, fantasy, in-game actions, in-game objects, animation, game environment, and feedback [6, 18]. For instance, Lomas and colleagues have explored the benefits of increasing challenge in an educational game, finding that increasing challenge doesn’t necessarily increase motivation and learning; instead students were generally more motivated by “easy” versions of games [16, 17]. Another key feature of educational games is agency, the capability for students to decide how to play games - what aspects of a game they will explore, how long they will play, and when they will try out various game features. Agency is often seen as a component of engagement [23], which in turn leads to fun. Yet, agency, which is closely related to self-regulated learning [34], may or may not be helpful for learning. While the best learners, those who demonstrate achievement on tests, are often good self-regulated learners [26, 34], many students are not good at regulating their learning, benefiting from instructional choices being made for them through direct instruction [33].

Several past studies have explored the effects of agency by giving students control over the way they play an educational game. For example, [30] showed that letting students decide the amount of time to work on different lessons can lead to higher learning outcomes. On the other hand, allowing students to make choices on instructionally irrelevant components of a learning environment has also been shown to make a difference to learning. For instance, [3] let students customize game icons and names in their arithmetic tutor, while [27] provided in-game currency, which could be spent on personalizing the system interface or extra game play. In these studies, the idea was to provide students with a sense of control without the risk of making poor pedagogical decisions. The general results across these studies showed that students who exercised their agency became more involved and learned more.

A notable example of studying student agency, in which students were given instructionally relevant choices, comes from Sawyer et al. [24] who explored variations in agency within the game Crystal Island. In Crystal Island, students are tasked with exploring an island and interacting with people and objects to gather knowledge about a spreading disease, learning about microbiology along the way. Three agency conditions were present in this study: high-agency, which allowed students to navigate to locations in the environment in any order; low-agency, which restricted students to a prescribed order, and no-agency, where students simply watched a video of an expert playing the game. In their study Sawyer et al. found that students in the low-agency condition attempted more incorrect submissions but also attained significantly higher learning gains, which might be attributed to their extensively engaging with instructional materials. Their results suggest that limiting agency improves learning performance but can also lead to undesirable student behaviors, such as a propensity for guessing.

In our study we explore variations of agency within Decimal Point, the educational game briefly described above. Our study compares a low-agency version of the game, in which students are guided to play in a prescribed sequence, playing all possible “mini-games,” to a high-agency version of the game, in which students can choose how many and in what order they will play the mini-games. The study is comparable to Sawyer et al. [24] in its exploration of agency; students are compelled either to play the game in a lock-step order or to exercise autonomy by making their own choices about game play. In addition, the choices students are presented with in both Decimal Point and Crystal Island are pedagogically relevant. In the high-agency version of Decimal Point, a student can choose to play mini-games that focus on specific aspects of the content domain (e.g., adding decimals, comparing decimals, completing decimal sequences), as well as choosing to get more or less practice with decimals (and game playing). Our research questions and hypotheses for the study are as follows.

Research Question 1. Is there a difference in learning between students who play the low-agency version of the game versus the students who play the high-agency version of the game? Given the results of the Sawyer et al. study [24], as well as the similarities between our implementation of agency and theirs, we hypothesized that the low-agency version of the game would lead to better learning outcomes than the high-agency version of the game.

Research Question 2. Is there a difference in enjoyment between students who play the low-agency version of the game versus the students who play the high-agency version of the game? Given past research on agency showing that students prefer to make their own choices, regardless of whether those choices are pedagogically beneficial [3, 24], we hypothesized that the high-agency version of the game would lead to higher levels of enjoyment than the low-agency version.

2 The Educational Game: Decimal Point

Decimal Point is a single-player game designed to help middle-school students learn decimals. The game is based on an amusement park metaphor (Fig. 1), with the student traveling to different theme areas (e.g., “Haunted House,” “Wild West”), playing a variety of mini-games within each area (e.g., “Western Shooter,” and “OK Corral,” within the Wild West theme area). The mini-games are targeted at helping students overcome common decimal misconceptions [10, 13, 28]. Students do not score points or compete with their fellow students. Instead, they simply play the mini-games in the amusement park and are commended upon completing the journey. There are no other activities within Decimal Point beyond playing of the mini-games.

Fig. 1.
figure 1

The Decimal Point game (low-agency version).

Full size image
Fig. 2.
figure 2

A student playing the “Space Raider” mini-game.

Full size image

An example mini-game, “Space Raider,” is shown in Fig. 2. This game challenges the student to use laser guns (lower left and right of Fig. 2) to shoot decimal-labeled spaceships (e.g., 0.3234, 0.5, 0.82, 0.634) in the order from smallest to largest decimal. The “Space Raider” mini-game is targeted at “whole number thinking,” a common misconception in which students think longer decimals are larger than shorter decimals [28]. The student tries to shoot the spaceships in the requested order and, if they make mistakes, are prompted to correct their solution by dragging and dropping the numbers into the correct sequence. Feedback in this mini-game is provided at the end of the game, once all spaceships have been shot. In Fig. 2, the student has exhibited the misconception by shooting the spaceships in length order: 0.5, 0.82, 0.634.

In the original version of Decimal Point, students are prompted to play each mini-game in a pre-specified sequence, according to the dashed path shown in Fig. 1, starting from the upper left. In the study discussed in this paper, this setting is referred to as the low-agency version of the game, since student choice is limited. In order to explore agency, we extended the game to a high-agency version that allows students more control over their experience and learning. As depicted in Fig. 3, students are given several choices. First, they can play the mini-games in any order they choose. Students are also presented with a dashboard that displays the five different categories of mini-games (i.e., (1) “Addition - Add decimals”, (2) “Bucket - Compare decimals”, (3) “Sequence - Complete a decimal sequence”, (4) “Number Line - Place point on number line”, (5) “Sorting - Ordering decimals”), as well as the specific mini-games within each category. In Fig. 3 four mini-games have been played, indicated by their icons being colored in the map and their names shown in red font in the dashboard. By mousing over the various game icons, the student can learn about each game and thus access information and gain knowledge to make informed choices.

Fig. 3.
figure 3

High-agency version of the Decimal Point game. The mouse cursor is currently over the “Thirsty Vampire” icon, displaying that mini-game’s info in the dashboard.

Full size image

Second, students can stop playing Decimal Point once they have finished playing at least one-half of the mini-games, as shown in Fig. 4. When they reach the halfway point, they are presented with a dialogue that says “You have finished playing half of the mini-games. You can keep playing until all games have been played or stop at any time by clicking on the Stop Playing button,” and a new “Stop Playing” button appears in the upper left, as in Fig. 4. At any time from this point until they finish playing all of the games, students can click on “Stop Playing” to quit playing and proceed to the next item in the materials.

Finally, once students have completed every mini-game once (2 problems per mini-game), they can play more mini-games, any of the original 24 games, for one additional problem each. They are also presented with a dialogue telling them they can keep playing (i.e., “You have played all of the mini-games. You can now either quit (by clicking on Stop Playing) or replay some of the mini-games”) and a “Stop Playing” button that allows them to stop playing at any time (Fig. 4). Taken together, these changes mean that students playing the high-agency version of Decimal Point can play from 24 up to 72 mini-games (compared to the standard 48 in the low-agency condition), in any order of their choosing.

Fig. 4.
figure 4

High-agency version of the Decimal Point game after the student has played one-half of the mini-games and is given the option to stop. A “Stop Playing” button appears on the dashboard in the upper left.

Full size image

As mentioned, the original (low-agency) version of the game has been empirically demonstrated to be effective for engagement and learning. In our previous study [20], students were assigned to one of two conditions that were compared: the game condition or the non-game condition. Students in the game condition were presented with two problems to solve for each of the mini-games shown in Fig. 1. The non-game condition presented a more conventional user interface to the students, prompting students to solve precisely the same decimal problems, in the same order. [20] compared 75 game-playing students to 83 non-game-playing students and found that the game-playing students enjoyed their experience more and learned more. In subsequent data analyses, we also found that female students benefited more from Decimal Point than male students, and the game made difficult problems more tractable for all students, as students in the game condition made significantly fewer errors on the difficult problems [21]. In this paper, we describe a study to explore how extending the agency feature of the game might alter student learning.

3 Method

3.1 Participants and Design

The original participants were 197 students from two schools in a large U.S. city (45 fifth and 152 sixth graders). Students were randomly assigned to either the high-agency (HA) or the low-agency (LA) condition. Thirty-two (32) participants (19 HA, 13 LA) were excluded from the analyses because they did not fully complete all materials and measures in the study. An additional seven (7) participants were removed due to having gain scores 2.5 standard deviations above or below the mean between the pretest and the immediate posttest or between the pretest and the delayed posttest. The remaining 158 students (79 HA; 79 LA; 81 male, 77 female) had a mean age of 11.15 (SD = 0.60).

3.2 Materials

A web-based learning environment was used to deploy the experiment, and the instructional materials were assigned to each group as outlined in Table 1. Materials included three tests (pretest, posttest, delayed posttest), the two game versions, and two questionnaires (demographic, evaluation). Details about the materials are provided in the remainder of this section.

Pretest, Immediate Posttest, and Delayed Posttest. The pretest, immediate posttest, and delayed posttest (one week after the posttest), were administered online. Each test consisted of 24 items, some of which had multiple parts, comprising 61 possible points. Participants received points for each correct part. There was an A, B, and C form of the base test, which were isomorphic to one another and which were positionally counter-balanced within condition (i.e., approximately 1/3 of the students in each condition received Test A as the pretest, 1/3 received Test B as the pretest, and 1/3 received Test C as the pretest; likewise for the posttest and delayed posttest). According to the result of an ANOVA, there was no difference, in terms of the difficulty level, among the three versions of the test: A, B and C (\(F(2, 155) = 1.14,\, p = 0.322\)).

Table 1. Conditions and Materials used in the study. Italicized items vary across conditions.
Full size table

Test items were designed to probe for specific decimal misconceptions and took a variety of forms, for instance: adding, multiplying, and dividing decimal numbers (e.g., ), choosing the largest of a given set of decimals (e.g., “Choose the largest of the following three numbers: 5.413, 5.75, 5.6”), and placing a given decimal number on a number line.

Questionnaire. After the lesson, an online evaluation questionnaire was presented to the students, prompting them to rate their experience of interacting with the instructional materials. Students could respond on a 5-point Likert scale, ranging from 1 = “strongly disagree” to 5 = “strongly agree”. For the purpose of our analysis, the eight items in the evaluation questionnaire were combined into the following three different categories:

  1. 1.

    Lesson Enjoyment indicates how much students like the lesson, such as “I liked doing this lesson” and “I would like to do more lessons like this.”

  2. 2.

    Ease of Interface Use indicates how easy it was to interact with the intervention items and interface for the lesson, such as “I liked the way the material was presented on the screen,” “I liked the way the computer responded to my input,” “I think the interface of the system was confusing,” and “It was easy to enter my answer into the system.”

  3. 3.

    Attitude towards Math indicates how students felt about math after completing the intervention, such as “The lesson made me feel more like I am good at math” and “The lesson made me feel that math is fun. ”

4 Results

First, a repeated measures ANOVA showed that students in both conditions learned from the pretest to the immediate posttest (LA: (\(F(1, 78) = 70.803,\) \(p < 0.0005\)); HA: (\(F(1, 78) = 49.043, \ p < 0.0005\))) and from the pretest to the delayed posttest (LA: (\(F(1, 78) = 67.931, \ p < 0.0005\)); HA: (\(F(1, 78) = 109.64, \ p < 0.0005\))). Next, we conducted analyses related to our two research questions, as well as conducting other post-hoc analyses.

Research Question 1. Is there a difference in learning performance between students who play the low-agency version of the game versus the students who play the high-agency version of the game? With pretest scores used as a covariate, we ran an ANCOVA for both pretest-immediate posttest and pretest-delayed posttest. Results are shown in Table 2. The LA group performed slightly better than the HA group on the immediate posttest (\(F(1, 155) = 1.8, \ p = 0.182, \ d = 0.13\)) but worse on the delayed posttest (\(F(1, 155) = 2.465, \ p = 0.118, \ d = -0.14\)). Neither result is significant.

Interestingly, when broken down by separate schools - one in which we experimented with sixth graders (School 1) and a second in which we experimented with fifth graders (School 2) - we see a significant result. Again, applying ANCOVA with pretest scores as covariates, in School 1 (6th graders) the low-agency group (\(n = 61\)) performed significantly better than the high-agency group (\(n = 61\)) on the immediate posttest (\(F(1, 119) = 4.534, \ p = 0.035, \ d = 0.18\)). However, the low-agency group performed slightly worse, but not significantly so, on the delayed posttest (\(F(1, 119) = 1.286, \ p = 0.259, \ d = -0.15\)).

Table 2. Learning results across conditions.
Full size table

For School 2 (5th graders) the low-agency group (\(n = 18\)) performed slightly better, but not significantly, than the high-agency group (\(n = 18\)) on the immediate posttest (\(F(1, 33) = 0.621, \ p = 0.436, \ d = -0.08\)). The low-agency group performed slightly worse, but not significantly so, on the delayed posttest (\(F(1, 33) = 1.68, \ p = 0.204, \ d = -0.12\)).

In summary, our hypothesis that the LA group would learn more than the HA group was not confirmed, except with respect to one of the schools, and only on the immediate posttest at that school.

Research Question 2. Is there a difference in enjoyment between students who play the low-agency version of the game versus the students who play the high-agency version of the game? Three categories that roughly summarize how students enjoyed and felt about the game (i.e.. Enjoyment in the Lesson, Ease of Interface Using, Attitude towards Math) were assessed by one-way ANOVA (Table 3). For lesson enjoyment, the high-agency and low-agency groups did not differ significantly (\(F(1, 156) = 0.007, \ p = 0.935, \ d = -0.01\)). For Ease of Interface, again, the two groups did not differ significantly (\(F(1, 156) = 0.285, \ p = 0.594, \ d = -0.085\)). Also, for Attitude towards Math the groups did not differ significantly (\(F(1, 156) = 0.584, \ p = 0.446, \ d = 0.122\)).

In summary, our hypothesis was not confirmed that the HA group would experience a higher level of enjoyment with the game than the LA group.

Table 3. Engagement results across conditions.
Full size table

Posthoc Analyses. Given the results and answers to our research questions, we performed post-hoc analyses to better understand why we did not see the differences we expected between the high-agency and low-agency game conditions. In particular, we were interested in exploring what the high-agency students did with the additional control they were given in game play. Did they take advantage of it, to explore the game and have more fun? Did they leverage their autonomy to make self-regulated learning choices?

To explore these questions, we looked at the specific mini-games and mini-game sequences that were chosen by players in the HA condition. The number of high-agency students who did less than, exactly the same, and more than the canonical number of problems was 15, 54, and 10, respectively. Thus, a significant majority of the HA students (68%) played the same mini-games as the LA students. Furthermore, we found that 22% of students in the HA condition precisely followed the canonical sequence (i.e., the sequence prescribed in the LA condition, as shown in Fig. 1). To get a sense of how different the sequences of students in the HA condition were from the canonical sequence, we calculated Damerau-Levenshtein distance between each students’ sequence and the canonical one. Damerau-Levenshtein distance counts the number of insertions, deletions, substitutions, and transpositions needed to turn one string into another [5]. In addition to raw Damerau-Levenshtein distance we also calculated a length matched distance, which measured either the edit distance between a student’s sequence and a subsequence of the canonical sequence in the event the student played less than 24 mini-games or a subset of the student’s sequence of equal length to the canonical sequence in the event they played more mini-games. This modified edit distance avoids inflating the distance for those students who chose to play more or less while still giving a qualitative sense of how similar their path was to the standard path.

In general, the distributions of edit distances were lopsided, due to so many players following the prescribed order. On average, players’ sequences differed by about 13.07 edits (SD = 9.73) from the canonical sequence, meaning roughly half of the mini-games they played followed the expected sequence. When controlling for sequence length (i.e., when students chose to play more or less mini-games) this effect is further tempered to 10.77 edits (SD 8.83) from the standard order.

In addition to the distributional information we also checked to see whether a student’s edit distance from the canonical sequence had any effect on their pretest - posttest or pretest - delayed posttest learning gains. Repeated-measures ANCOVA showed no significant effects for pretest - posttest (\(F(1,78) = 0.18, \ p = 0.67\)) or pretest - delayed posttest (\(F(1,78) = 0.00, \ p = 1.00\)). This means that the amount of difference between a student’s chosen ordering and the prescribed ordering had little effect on their learning gains.

5 Discussion

We hypothesized that our findings would (roughly) replicate those of Sawyer et al. [24], given the similarities in implementation of agency between the two games. However, other than the finding at School 1, in which the low-agency students did exhibit a better learning outcome than the high-agency students, we did not replicate their findings. Instead, we found no overall differences in learning between the low- and high-agency students who played Decimal Point. Given that our results only partially replicate those of Sawyer et al., we are left with the question of explaining why this happened.

As stated earlier, our implementation of agency has key aspects in common with [24]. For instance, students are compelled either to play the game in a specific order (low-agency condition) or to have autonomy to make choices about the order of game play (high-agency condition). In addition, the choices students are presented with in both games are pedagogically relevant, unlike other studies of agency, such as [3, 27], in which the choices were unrelated to learning elements in the games. However, one key difference between Decimal Point and Crystal Island is that the Decimal Point high-agency game displays student progress (through the dashboard - Figs. 3 and 4), allowing students to make informed choices about their next step. Perhaps more importantly, in Decimal Point the high-agency students are given the choice to stop or continue playing, a game feature that is not part of the high-agency version of Crystal Island. In particular, the Decimal Point high-agency students were in control of how much practice on the relevant domain problems they wanted, by choosing to play anywhere from 12 less to 24 more mini-games than the standard number.

Interestingly, however, it appears that the high-agency Decimal Point students did not exercise their given agency very much. As mentioned above, 68% of the high-agency students tackled precisely the same problems as the low-agency students. Additionally, the sequences of mini-games chosen by the students did not substantially vary from that of the low-agency condition. One potential reason for this lies in the visual design of the amusement park map (Fig. 2). In particular, the dotted line connecting all the mini-games could have implicitly, yet unintentionally, communicated the sequence of mini-games to follow. This could be a case of “indirect control” [25], where subtle pieces of visual design can draw students’ attention and guide behavior without explicit direction.

Another interpretation points to the nuanced nature of agency versus autonomy. In our study, while the students were given the choice of which mini-games to play (autonomy), they may not have felt their choices were consequential (agency). When the students finished a mini-game, the only change visible to them was a game icon “blocked” on the map. In other words, playing Space Raider before Thirsty Vampire or vice versa would lead to the exact same state, where both games are blocked, so which game being picked first would not matter. A recent study by Flowerday and Shell [7] also shows that providing instructional choices alone does not increase motivation and learning outcome; instead, the key motivator is situational interest. Based on this finding, we could redesign the high-agency condition so that student choices have a more significant impact on their subsequent game experience. On the other hand, the freedom to stop early or play additional games is an instructionally relevant instance of agency. Yet, students did not take advantage of this, possibly because they are often not good self-regulated learners [34]; younger students are likely to be even weaker in this aspect.

Finally, we note that our study was conducted in a classroom setting, where students’ performance on the tests were used for class grades. Thus, there may have been some felt pressure, causing students to be hesitant about finishing early and skipping some materials. As noted in [23], when one feels controlled in pursuing an activity, one’s sense of autonomy and motivation becomes diminished. Conducting the same study in a pressure-free environment could potentially yield more pronounced differences between the two agency conditions.

6 Conclusion

This study of student agency was intended to explore the earlier results of [24], investigating whether providing students agency in a game context would increase or decrease their learning. In this study, we did not find that students in a low-agency condition learned more than the students in a high-agency condition (except for one school). As is often the case, shifting the context of instruction can change the results. In our study, the effects of indirect control or teachers’ pressure might play an important role. It could also be the case that students did not experience much agency or were not good self-regulated learners. Nevertheless, educational game studies such as this are important in helping us better understand and make decisions about how to implement adaptivity in games, which ultimately artificial intelligence will control. Once we have a better understanding of the specific context in which, for instance, agency leads to better learning, we can develop AIED-infused games that adapt to those contexts.