Introduction

Digital games have captured the attention, interest, and time of youth in our society. According to a recent report, 81% of teens have access to game consoles and 72% play games online or on their phones (Lenhart 2015). What’s more, teachers are increasingly choosing to use digital games in their classrooms; 57% of surveyed teachers used digital games in their classrooms at least once a week, and over 80% said they were moderately comfortable using games in their classrooms (Fishman et al. 2014).

Unfortunately, findings are still mixed about whether or why games designed to connect with academic content and labeled “educational” are actually effective at supporting thinking, learning, or enjoyment. Digital games, as a whole, can be difficult to study because there are many different kinds of games that leverage radically different core mechanics (Clark et al. 2015; Young et al., 2012), making it difficult to consider “digital games” as a single, coherent instructional approach. Those that have been studied are almost always games developed by university researchers and not commercially available. In commercially available educational games, claims to support a particular kind of disciplinary idea (for example, “fractions,”) rarely consider the nuances in what it means to legitimately understand that idea.

As a consequence, there are hundreds of games now available that claim to be “educational,” but which have not been evaluated to ascertain whether they support learning. Contributing to this problem is the current state of research available about educational games, which, beyond generally establishing that digital games can support learning (Clark et al. 2015; Kebritchi et al. 2010; Wouters et al. 2013), has yet to develop a core set of design principles for educational games that are promising. Developing such an understanding would benefit not only future game designers, but would also help us to rank the quality of different games and their likelihood to support the development of particular understandings (Larkin 2015).

The goal of the current study was to contribute to our emergent understanding of whether and how particular types of digital games can support student learning and engagement. We focused on commercially available educational apps that were all classified as “games” in the app store, but that represent extremes in how open, exploratory and “game-like” they are. Our goal in focusing on commercially-available apps is to connect with the games that teachers and schools are most likely to access. We contrasted these apps to explore how students felt about the different digital experiences and how well each supported transfer of knowledge to paper-and-pencil assessments, with a goal of better understanding the educational benefits and drawbacks of different kinds of apps.

Theoretical framework

An important starting point for this research involves considering why design matters, and what, specifically, is impacted by different designs. It builds on research in mathematics education devoted to understanding how task design supports student reasoning (Henningsen and Stein 1997). In thinking about the design of tasks, it is tempting to consider three separate components: the specific disciplinary topics that are being targeted, the core features of the task that are intended to help a student think about those ideas, and the prior knowledge of the student. However, theories of learning that consider learning to be an interactive accomplishment challenge the notion that these components can be separated, and instead see the player, the tasks, and the disciplinary content as pieces of a complex system that cannot be pulled apart to be studied separately (Barab et al. 2010; Gee 2003; Greeno and MMAP 1998). From this perspective, a task that can be presented as a worksheet or embedded in an interactive game is not the same task; the way the task is framed and the motivation or attention the different tasks invite from the player work together to make those distinctly different learning experiences (Gee 2003).

The idea of “task” and “content” as interactive is a core consideration for this paper, in that we wished to determine whether games that, on their surface, claim to be about the same thing, actually are the same thing. To address this question, one could systematically vary elements of games and study student learning in relation to each varied element. However, such an approach forgets that games are an interactive whole, more than the sum of its parts. Changing one part might actually change the overall game experience, and thus the one thing that is being varied might not ultimately be the one thing that is actually being measured.

Another approach is to classify games based on their elements, and then to study a collection of games that share elements in an attempt to begin to understand how characteristics of games might impact learning. This approach does not undermine the gestalt of the game that has been designed, but does allow for contrasts to be made between characteristics of particular games. To be sure, this design cannot definitively claim to be measuring one particular thing. However, if more such studies are conducted, they can be aggregated, such as the recent meta-analysis conducted by (Clark et al. 2015). In order to do this, studies need to be explicit about the elements of the design that are included in the games being studied, so that we can begin to tag these elements and aggregate them over multiple studies. This is the approach taken in this study, as detailed below.

What do we know about games and learning?

Digital games have demonstrated potential for supporting student learning across disciplines (Kebritchi et al. 2010). Digital games have the potential to motivate and capture student attention (Dickey 2007; Garris et al. 2002), to situate disciplinary learning in realistic contexts (Barab et al. 2005; Clarke and Dede 2009) and to offer consistent and substantive feedback about reasoning (Gresalfi and Barnes 2015; Nelson 2007; Rieber 1996) and to support active exploration and experimentation (Olson 2010; Squire 2006).

Three recent meta-analyses on learning from digital games for k-12 students confirmed that digital games typically led to higher cognitive outcomes (such as learning and retention) than traditional instruction (Clark et al. 2015; Vogel et al. 2006; Wouters et al. 2013). However, all reviewers noted the paucity of studies that compared digital games to other games or learning environments (c.f. Young et al. 2012). Further, evaluation of digital games has focused almost exclusively on games that were developed as part of a research project and included educational expertise on the design team. This stands in contrast to many “educational” games that are available for purchase, which may not include expertise either in the targeted content or in learning theory more broadly. Thus, the games that were reviewed are possibly not representative of the games that are generally being played.

Vogel et al. (2006), claim that games need to satisfy three criteria: ‘‘has goals, is interactive, and is rewarding (gives feedback)’’ (p. 231). Based on this definition, a variety of digital activities can be classified as “games,” and indeed, a search in the app store yields a range of activities under that single category. Differences in game mechanics lead to very different game play experiences, however, and students can easily distinguish between a game and “just a worksheet on an ipad,” as one of our participants put it. Nevertheless, we do not yet know whether these differences make a difference in terms of student learning and motivation.

Many digital forms of instruction offer a host of potential advantages such as immediate feedback, personalized scaffolding, and an individualized pace of activities. Indeed computer-assisted learning in non-game environments also improves student learning relative to traditional instruction (Sosa et al. 2011; Tamim et al. 2011). Digital apps that more closely match traditional assessments allow children to practice target tasks at their own pace and receive individualized feedback. Because these more structured digital worksheets share many similarities with paper-and-pencil assessments, it seems likely that they should promote success on paper-and-pencil assessments.

However, such apps miss opportunities afforded by interactive technologies, such as games, which can involve exploring, applying, and personalizing. When a true “game” experience is designed, learning can appeal to motivational aspects common to game play, across ages (Ebner and Holzinger 2007; Nygren et al. 2012). However, students sometimes have difficulty transferring knowledge they learn in games to out-of-game contexts, especially traditional assessments (Girard et al. 2013). This may be due to the high contextualization in games (e.g., first person camera perspective, relevant and thick storyline, visual realism, and anthropomorphism), leading to distraction or cognitive overload and/or knowledge embedded in the game context that does not transfer to other contexts (Clark et al. 2015).

Taken together, it is clear that while the collection of activities that satisfy the criteria of “games” have significant potential for supporting learning, how these activities might be distinguished, and what forms of learning and engagement they support, is still unclear. Making this issue even more complex, the disciplinary learning that a game is targeting also contributes to the design and potential of the game, and thus the question is not “what kinds of designs support learning,” but rather, “what kinds of designs support learning of particular content?” (Mayer and Johnson 2010).

Digital games for fraction learning

In this study we focus on fraction concepts, given their broad importance to mathematics learning (Common Core State Standards 2010; Siegler et al. 2012). Understanding fractions is a major hurdle for students; for example, on the 2009 National Assessment of Educational Progress, only half of fourth-graders correctly identified equivalent fractions and a quarter correctly identified the fraction with the value closest to ½ (retrieved from: http://nces.ed.gov/nationsreportcard/). Perhaps for this reason, there are over 200 games that are available in the Apple app store classified as ‘fraction’ in the category ‘games.’

Both digital games and worksheets allow students to work with fraction models such as bars and number lines. Virtual models of fractions can be as effective as physical models while also being more efficient and easier for teachers to implement (Mendiburo and Hasselbring 2014; Reimer and Moyer 2005). Students can interact directly with virtual models, moving and partitioning them, and receive hints and feedback.

However, there is little published research on digital apps for learning fractions. The two empirical studies we were able to identify suggest that digital games can support learning of fraction concepts (Norton et al. 2012; Riconscente 2013). For example, Motion Math Fractions is a digital game in which students tilt a tablet to bounce a ball to place fractions (and decimals) on a number line. Fifth-grade students who played the app 20 min a day for 5 days developed greater knowledge of fraction and decimal magnitude, particularly placing fractions and decimals on a number line, and better attitudes and confidence towards fractions relative to students who were randomly assigned not to play the game (Riconscente 2013). Additionally, when asked about the game, all students reported they would play it again. Prior research on learning from digital fraction games has not compared the games to learning from alternative fraction activities.

Current study

In the current study, we contrasted commercially-available apps that were different in terms of design, but which focused on the same disciplinary content: order and equivalence of fractions. We posed three questions:

  1. 1.

    Learning: Does the design of an app (more game-like versus more worksheet-like) influence what students learn, when measured by pencil-and-paper assessments?

  2. 2.

    Attention: Does the design of an app influence what students notice about the app, in particular whether they can identify what it is supposed to be teaching them?

  3. 3.

    Enjoyment: Does the design of an app influence students’ enjoyment of the app and willingness to play more?

Research question 1 targets a central concern of all educators, specifically, whether the app is a useful instructional tool. Questions 2 and 3 are more nuanced in that they help us to better understand how students are connecting with the app both in terms of what they notice and what they enjoy.

Method

Participants

Participants were 95 students from seven third-grade classrooms from two public elementary schools in the southern United States. All students had been introduced to fractions concepts previously, and all students had experience using iPads. The first school served predominately White, middle-to-upper-class students in kindergarten through fourth grade (85% White, 7% African American, 3% Hispanic, 5% Asian; 8.5% qualified for free or reduced priced lunch; 50% female). In 2014, 89% of students at the school in third- and fourth-grade scored proficient or higher on the state’s standardized math assessment. The second school served ethnically diverse students in pre-kindergarten through fourth grade (41% White, 40% African American, 6% Hispanic, 13% Asian; 40% qualified for free or reduced priced lunch; 52.5% female). In 2014, 63% of students scored proficient or higher on the state math assessment.

Digital apps

We selected digital activities that were available in the app store, which appeared with a search of “fractions” and “games,” and which focused on introductory fraction concepts of fraction magnitude, equivalence, and comparison. We then reviewed the apps and chose two types that were highly contrasting. The first, which we call games, included game mechanics that were not, on their face, explicitly about learning fractions. The storyline or context of the game was the primary focus of attention and activity, and understanding or use of factions was a more implicit aspect of the game than in the digital worksheets. The second, which we call worksheets, included a set of activities that focused specifically on correctly identifying the order or equivalence of different fractions. The storyline of these games was either thin or non-existent, and the disciplinary content of the game was the explicit focus of hints and feedback.

Digital games

We selected two apps that focused on order and equivalence of fractions and that were “best examples” of games. This meant that the apps were endogenous (Squire 2006) in that they leveraged core game mechanics that were connected to the targeted ideas being learned. We also selected apps that had received positive reviews by educators, parents and children. There were few apps that satisfied all of these criteria; we selected two: Slice Fractions and Motion Math: Fractions!

Slice Fractions has garnered significant praise and attention. An unpublished study conducted by researchers of the University of Quebec at Montréal demonstrated that Slice Fractions significantly improved third-grade students’ performance after 3 h of game play (LeBlanc, December 8, 2015). The game has won a series of awards, including Best Original Digital Content 2015 from Youth Media Alliance, Best Family Friendly Game 2014 from the Indie Prize Showcase Awards, and Gold Medal Winner 2014 from International Serious Play Awards. The core mechanic of the game involves slicing pieces of ice and arranging them to fall onto fires that are blocking the path of a Mastodon. Clearing the path is the goal of every level, although how to clear the path changes from level to level (see Fig. 1). Verbal directions are not given; rather, students explore as they try to complete each activity, retrying and getting non-verbal hints until they accomplish it.

Fig. 1
figure 1

Screen shot from an early level of Motion Math (left), and a midpoint level of Slice Fractions (right)

Full size image

As noted above, Motion Math: Fractions! has been the subject of previous research demonstrating its effectiveness (Riconscente 2013). The game won the Children’s Technology Review Editor’s Choice Award for Excellence in Design in 2010 and a gold medal from the Serious Play Conference in 2011. The core mechanic of Motion Math involves tilting the tablet to bounce a ball to place it on the number line (see Fig. 1). The ball that drops sometimes takes different forms (ball, spaceship, etc.,) and the background of the game changes over time. Within each ball is symbolic or circle representations of a fraction or decimals. When their placement of the number is incorrect, students try again with scaffolding (e.g., an arrow appears where the student incorrectly placed the fraction pointing in the direction where the number needs to go on the number line). With more failure, the number line is partitioned, and eventually students are shown where exactly the ball should fall.

Digital worksheets

We also selected apps that seemed the “best examples” of their kind. We looked for apps that covered the same content as the selected games, that offered feedback to students, and that used clear representations of fractions and did not rely solely on symbolic manipulation. The apps that we selected were Tiny Fractions, Sort Fractions and Jungle Fractions. Sort Fractions and Jungle Fractions were both played in one of the sessions in order to fill 30 min of time.

Tiny Fractions was developed by “Tap to Learn,” a design company that emphasizes the potential of immediate feedback gained by “tapping” a screen to quickly engage ideas. Tiny Fractions is highly rated on the app store, receiving 4.5/5 stars. In Tiny Fractions, students identify, compare and order fractions. Fractions are represented using partially filled beakers (see Fig. 2). Each activity begins with a demonstration of how to complete the task with sample fractions. The first activity is to represent symbolic fractions using test-tube beakers. Students choose a number of partitions within a beaker and fill the beaker to the appropriate level. Then they compare and order symbolic fractions with different numerators or different denominators. Feedback instructs students on which mechanic to use next or by revealing a hidden visual representation of the quantity.

Fig. 2
figure 2

Screen shot of Tiny Fractions (top left), Sort Fractions (bottom left), and Jungle Fractions (right)

Full size image

In Sort Fractions, also developed by Tap to Learn, students order a set of symbolic fractions or decimals from least to greatest (see Fig. 2). Students could view circular representations of the numbers for help if requested. This game takes less time for students to play through, and thus we selected a second game for students to play that targeted the same ideas and used similar representations. This game, Jungle Fractions, was developed by Jungle Education, and was featured by Apple in “What’s Hot”, “New & Noteworthy” and “Apps for Learning Math.” In Jungle Fractions, students can engage a variety of fraction activities grounded in circular representations of fractions, from naming a shaded portion of a fraction to comparing two partially shaded representations and declaring which is larger (or if they are equal; see Fig. 2). Students get feedback about the accuracy of their answers with either pleased or sad animal sounds.

Procedure

Data collection took place in the spring of the school year. Day 1 included a pretest and a half hour of working on an app. Day 2 included a second half hour of working on a new app and a posttest. Day 3 included group or individual interviews. In total, approximately 3 h of data were collected over the 3 days; these 3 days spanned a single week.

Within each classroom, students were randomly assigned to one of two conditions (game vs. worksheet) and worked in a small group with 5–6 students assigned to the same condition. Students in the game condition played Motion Math and Slice Fractions, and students in the worksheet condition played Tiny Fractions, Sort Fractions, and Jungle Fractions. Within condition, the order of the apps was counterbalanced, and there were no effects for order on posttest performance. The procedure for playing the apps was the same each day and across conditions: After a 1–2 min introduction to the app that focused on the mechanics, students worked on the app for 30 min. This same procedure was followed on Day 2, with students working on the second app in their condition. Each student had his/her own tablet, and students were allowed to talk with each other and help each other, although they were asked not to touch each others’ tablets.

Measures

Assessment and coding

The assessment included fraction items from 1990 to 2009 NAEP mathematics released items as used in Fuchs et al. (2013). The assessment also included four fraction magnitude comparison problems adapted from the 2010 Fraction Battery (e.g., circle the fraction that is bigger; Schumacher et al. 2010). Items measured students’ understanding of symbolic fraction magnitudes (n = 6), magnitudes represented by shaded areas (n = 6), and magnitudes represented on a 0–1 number line (n = 2), and ability to reason about another students’ incorrect answer (worth 2 points). The maximum score was 16 points. Coefficient alpha on this sample was .88 at pretest and .85 at posttest.

Questionnaire

Students filled out a questionnaire twice during the study; once after working on the app on Day 1, and a second time after working on the app on Day 2. The questionnaire included 9 items that asked about their experience playing the game. Some questions targeted why they were playing (“I played this game because I was having fun”), others asked for their opinion about the experience (I would want to play this app again), and others asked about how it felt to play the game (I thought about how I couldn’t stand playing the game anymore). Students responded using a four-point rating scale (1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree). Using a four-point scale ensured that students would need to express an opinion in some way. Some items were reverse scored so that higher scores reflect more agreement/enjoyment. Students’ ratings were averaged across both days.

Conceptual interviews

We also conducted one-on-one interviews with a subset of students at one of the schools (n = 23; 11 from the game condition, 12 from the worksheet condition). Students were selected from each of the four small groups in each classroom (1–2 students per group per class). Interviews focused on three topics: Students’ understanding of the apps they had played, students’ enjoyment of the apps they played, and students’ understanding of a subset of fractions items that were included on the post test. Interviews lasted between 10 and 20 min, and were videotaped and audiorecorded. All interviews were double coded separately by two coders, who discussed any conflicting codes until agreement was reached.

Videos and observations

Finally, a subset of randomly-selected groups were videotaped while playing the game, with the goal of using these videos to explore or delve into findings from our measures. Videocameras were set up so that they could capture student interaction and facial expressions, and did not generally show detail of the screens that students were looking at.

Results

Research question 1: learning about fractions

Our first question considers whether the kind of app students played impacted how much they learned about fractions that they were able to transfer to a paper-and-pencil assessment. It would be reasonable to predict that digital worksheets, which explicitly targeted fractions and presented few distractions from that content, would be more effective at supporting transfer to a traditional paper-and-pencil assessment. In fact, students seemed to perform equally well on the assessment after playing the game apps as they did after using the worksheet apps (see Table 1).

Table 1 Percent correct paper-and-pencil assessment by time point and condition
Full size table

We examined changes in performance from pretest to posttest using a repeated measures ANOVA model with condition as the between-subject factor and time as the within-subjects factor. There was a significant effect of time, F(1, 94) = 15.35, p < .01, η 2p  = .14; indicating that students in both conditions learned from using the apps. However, there was no effect of condition and no condition by time interaction. Children in the game condition exhibited similar amounts of learning and transfer as children in the worksheet condition (see Table 1).

We also explored our assessments by grouping items according to the type of model used in the item (area model, number line, or symbolic only; one two-part item could not be classified). There was a significant effect of time for items including an area model, F(1, 93) = 10.02, p < .01, η 2p  = .10, and for items using symbols, F(1, 93) = 5.75, p = .02, η 2p  = .06, with students’ scores improving from pretest to posttest (see Table 1). For number line items, there was a marginal effect of time, F(1, 93) = 3.16, p = .08, η 2p  = .03; this marginal effect may reflect that there were only two number line items. There was no effect of condition or condition by time interaction in any of these analyses.

Research question 2: what students noticed

Our second research question delved into students’ experience of playing the game, specifically whether they were aware that they were using or learning mathematics as they were playing the app. We were also curious about how salient the math would be to students given the difference between the extent to which fractions was in the foreground (worksheets) or background (games) in the apps. To address this question, we examined the coded conceptual interviews conducted with a subset of students.

Three interview questions specifically targeted what students noticed about the apps they played. For each app, we showed an image from the games the student had played and asked students (a) what was going on in the image, and (b) if they thought the game was related to the assessment they had just completed, and if so, how. These questions were designed to create an opportunity to see if students understood the app and if they spontaneously offered any mathematical explanations to account for decisions a player might make in the midst of game play.

Across the conditions, students had no trouble explaining how the apps worked. Almost all students in the worksheet condition referenced mathematics when explaining how the game worked (see Fig. 3). For example, students describing Sort Fractions typically referenced the goal of the game being to order a set of cards with fractions written on them:

Fig. 3
figure 3

Counts of students’ explanation of games in terms of game mechanics and in terms of mathematics, reported by condition

Full size image

So I would flip them over, so I saw what the bar looked like, and then these two. So then I’d put it…so I’d press that. Then I’d put it [2/5] in the front because it’s less than 2/4.

This explanation accurately described how the app worked and focused on the mathematical ideas that were the central focus of the activity.

Students in the game condition also referenced mathematics when describing how apps worked. As one example, one student offers a mathematical explanation for how he knows what to do in Motion Math:

St: Um, you have to estimate. Imagine in your brain, pretend this is equal, dividing this up, that’s 1,2,3,4,5,6. Let’s imagine this is equal okay? So you’re going to try…imagine in you brain, this is divided into 1/6. You…six parts.

Int: Six parts, okay.

St: You don’t want to get it there, because that’s 6/6. And you don’t want to get it anywhere in between, because those are 2/6, 3/6, 4/6, 5/6. You want to get it at the first line. 1 out of 6.

However, students in the game condition were equally likely to describe the game mechanics or to connect the game and mathematics together (see Fig. 3), particularly when describing Slice Fractions:

So on this….you’re supposed…you’re supposed to make them into pieces where it’s like, its supposed to be where it goes down and it makes the volcano things go away. And on this one [right side of screen] it needs to be 3/6, and that’s the same size, so you just pop it and it will go down. And on this [left side] you need to cut it so it’s like a triangle, so it will fall down on 1/6.

We also asked students whether they thought that the apps they had played were connected to the assessment they had completed. Across both conditions, students thought that the games were related to the assessment, although more students in the worksheet condition believed this to be true (see Fig. 4).

Fig. 4
figure 4

Percent of student responses in interviews in each condition that perceived the app to be connected to the post-assessment

Full size image

Research question 3: enjoyment and interest

We used several sources of evidence to investigate whether there was a difference in students’ enjoyment of the game and desire to play. First, we examined differences in children’s self-reported enjoyment of the games, as measured by a questionnaire. On the questionnaire, children who played the games rated their enjoyment higher (M = 3.06, SD = .50) than children who worked on the digital worksheets (M = 2.57 SD = .86), t(23.2) = 1.98, p = .06, d = .69 (test adjusted for violation of homogeneity of variance).

Our observations of student play during the study were consistent with these questionnaire responses. There was a clear difference in students’ enthusiasm and enjoyment when they were in the game condition versus the worksheet condition, with researchers having to do much more encouraging of students to stay on task in the worksheet condition. In two of the classes, the teachers allowed students free play time on the ipads following the completion of data collection; in both classes, all students in the classroom chose to play Slice Fractions.

Our analyses of student interviews delved into more nuanced explanations of interest and enjoyment. If it did not come up spontaneously, we asked students if they enjoyed the game. Overall, more students in the game condition reported enjoying playing the apps than did students in the worksheet condition, and more students in the worksheet condition claimed that they did not enjoy the apps than students in the game condition (See Fig. 5).

Fig. 5
figure 5

Number of student responses in interviews in each condition that perceived the app to be enjoyable

Full size image

Students were very clear about the reasons they enjoyed or disliked the games. All the students in the worksheet condition who did not enjoy the apps claimed that they were boring, as seen in the following exchanges:

Int: Would you ever play these games at home, you know in your free time?

B: No, I would play some fun games.

Int: So did you have fun playing the games?

G: Sort of, the games were kind of boring, but they taught you fractions, and I like fractions.

Students in the worksheet condition who enjoyed the games discussed enjoying getting feedback and practice on skills that they knew they needed help with. For example, one student explained why she liked Sort Fractions:

Yeah. I actually liked it because I wasn’t that good at putting fractions in order.

Students in the game condition expressed much more enthusiasm for the apps they played, particularly for Slice Fractions. Students primarily claimed that the game was fun, and sometimes went into some detail about what made the game fun:

I just liked how they sometimes put in fun stuff. Like for the game where you bounce it on the number line [Motion Math], you go into space and got to see all this fun stuff, and it kind of impressed you. And for this game [Slice] it kind of like what hats you could get and stuff, and how there was that whale when you got far enough.

Overall, students in the game condition had much more favorable opinions about the apps that they played and were clearly aware both of the difference in how enjoyable the apps were to play and how likely there were, consequently, to continue to play. Somewhat problematically, some students were clear that when something is fun it isn’t teaching you as much…and the things that are more boring are probably more helpful. As one student clearly expressed:

[Slice Fractions] is a little bit low on the teaching side. It does…it did teach me a little, but it’s fun. Usually the funner the game, the less teach-y it’s going to be, but if it’s gonna amount to more. If people like this one less, they’re going to play it less, even though it teaches you more. So if people like this one more, it’s going to add up to more.

Discussion and conclusions

The goal of this study was to better understand the influences of particular types of designs on student learning and enjoyment. We focused on two very different kinds of apps that targeted order and equivalence of fractions. The decision to focus on such differences was based in part on our understanding of how these different environments should afford very different kinds of experiences for students, even when they ostensibly target the same content.

In designing this study, we were challenged to converge on a central hypothesis for our first research question on learning. On the one hand, research on transfer would suggest that learning environments that are more similar to the transfer tasks would be likely to lead to increased transfer when compared with learning environments that look more dissimilar (deWinstanley et al. 1996; Morris et al. 1977). Consistent with this argument, past studies have suggested that students sometimes have difficulty transferring knowledge they learn in games to out-of-game contexts, especially traditional assessments (Girard et al. 2013). This would lead to a hypothesis that apps that clearly focus on disciplinary content and offer students opportunities to practice those core ideas, as is common in digital worksheets, would be support better transfer to a paper-and-pencil assessment than apps that embed the same content in interactive and distracting game mechanics. On the other hand, research that has considered how the learner is positioned in relation to content might suggest otherwise; that apps that position the player as a problem solver, making disciplinary knowledge a tool to be used to engage those problems, might be both more motivating and lead to higher rates of learning and transfer for the game apps (Barab et al. 2010; Gresalfi 2009).

Rather than finding direct support for either perspective, we found that the game-like apps supported similar transfer as the worksheet-like apps. This occurred even though game-play was relatively brief (1 h) and teachers did not help students bridge from the game context to traditional assessment tasks. Although this study cannot isolate why this might be so, student interview data and questionnaire data suggests two potential factors to pursue in future research. First, students in the game condition often were not focused solely on game mechanics; a majority of students noticed the mathematical content and some were integrating ideas about fractions with their engagement with game mechanics (see Fig. 3). Many also noticed the relevance of the game for the paper-and-pencil assessment (see Fig. 4).

Second, as expected, students found the games to be more enjoyable and engaging than the worksheets. Students in the game condition rated their enjoyment more highly and were more likely to state that they enjoyed the apps, and less likely to state that they did not enjoy the apps. Further, when free play time on the tablets was made available in two of the classes, all students in the classroom chose to play Slice Fractions. It may be that the increased engagement in the digital games translated into more attention, commitment, or persistence relative to the digital worksheets, which contributed to equivalent transfer even though the games were less tightly linked to the paper-and-pencil assessment. In contexts where students decide how long to play the apps, they would likely play the games longer than they would complete the worksheet apps. In turn, the games could support greater transfer to a paper-and-pencil assessment than the digital worksheets under free choice conditions. Future research would need to explore this possibility.

Potential limitations

As noted, this study examined an intervention that was of short duration—one hour. There are reasons to be skeptical of such an intervention, ranging from concerns about novelty effects to concerns about ecological validity. As a field, we know that learning takes time, and that robust learning, in particular, takes even more time. Thus, we tend to prefer to study interventions of longer duration. However, there are both methodologically and ecologically valid reasons for studying an intervention of such short duration. With respect to the former, there are numerous examples of “one shot experimental studies” laying important groundwork for future longer-term classroom research and for providing experimental evidence, in combination with other research, which has very important implications for classroom practice. The U.S. Department of Education’s Educators Practice Guides (http://ies.ed.gov/ncee/wwc/Publication#/ContentTypeId:3) are prime examples of how evidence from a variety of research designs is important to provide converging support needed to identify evidence-based instructional strategies and practices.

Of course, with respect to motivation and engagement, it is possible that some results were driven by the novelty effect. However, we believe this is unlikely, because all apps were equally novel to students—none of the students in our study had used any of the apps before. Thus, the difference in interest is notable, even if overall interest was driven by novelty. Further, the notion of novelty driving interest and engagement in games has not been borne out in the literature, which has documented that games support sustained motivation and engagement (Ryan et al. 2006)—sometimes to a worrying degree. Finally, one critical finding of the study cannot be attributed to novelty—children gained content knowledge that transferred to a paper-and-pencil assessment from using the apps for only 1 h.

With respect to ecological validity, a drop-in experimental study that interferes with everyday classroom practice is often faced with skepticism about whether the findings are relevant to what actually happens in schools (Lave 1980). However, in this case, our study design more closely resembles classroom practice than might be expected. With respect to duration, the idea that students would play games in small groups for approximately 30 min at a time is precisely the way that ipads and similar devices are often used in classrooms, particularly when teachers do not have a full classroom set of devices. An hour (total) of play on an app is not unusual, particularly when there are multiple apps that students have available to them. With respect to game type, the use of a commercially-available game which can easily be accessed and downloaded from the app store is again a typical use of games in the classroom.

Implications and next steps

This paper took a different approach than is common to investigating digital games. Rather than comparing a game to a non-game learning environment, or systematically varying a component of a game between conditions, we compared genres of games with an eye towards learning and enjoyment. Our rationale for this approach was driven both by theory and pragmatism. Pragmatically, the majority of evaluation research on digital games has focused almost exclusively on games that were developed as part of a research project and included educational expertise on the design team. The current study more closely aligns with the choices teachers have; they select among commercially-available apps that target the desired content that they are required to cover. Of the games most readily available, a key choice teachers have to make is between apps that implicitly support target content embedded in contexts that support exploration, or those that provide practice completely commonly-assessed tasks, with added support. Both options allow for immediate feedback, personalized scaffolding and the individualized pace of activities, all features of computer-based activities that are thought to support learning (Sosa et al. 2011; Tamim et al. 2011). There is little available research that would help teachers to determine which kind of app might be best, and for which purposes. The current study suggests that commercially available apps that are more game-like and that are more worksheet-like both have the potential to improve learning and transfer on paper-and-pencil assessments. However, digital worksheets miss out on opportunities afforded by interactive technologies, which can involve exploring, applying, and personalizing the kinds of play that legitimate games afford. Current findings converge with past claims that digital games motivate and engage students (Dickey 2007; Garris et al. 2002; Lepper and Malone 1987; Malone and Lepper 1987). Improved motivation and engagement may be particularly useful when students have choice of whether and for how long to use the apps. This is particularly likely to be the case when students are left to their own devices or are able to decide for themselves how long to play.

Theoretically, our rationale for the design of this study stems from the perspective that games are interactive systems (Barab et al. 2010; Gee 2003), which cannot be taken apart to understand their constitutive parts. This is a claim that draws on situative theories of learning and interaction (Greeno and MMAP 1998; Schoenfeld 1998; Wertsch 1991). Thus, we sought not only to understand whether one kind of design was more effective than another, but also, why. By examining what students pay attention to while playing games, we were able to begin to better understand what makes particular games effective. Although it might be reasonable to assume that apps that include significant game mechanics would distract students from engaging with targeted content, in this study we found that students were attuned both to the mathematical decision making implicit in the game, as well as the ways that game mechanics intersected with those mathematical ideas. We found this to be the case across games that offered exploration as a game mechanic, suggesting that this was not a characteristic of one game in particular, but of a class of games. Students who played worksheet-like apps were more focused on the mathematics of the experience, but surprisingly, these students did not learn more about fractions as measured by a paper-and-pencil transfer task. The question of how game mechanics were supporting students’ understanding and learning is one that clearly deserves continued study.

With respect to future research, this study begins to tell us something about the design of games to support learning, but tells us much less about how these games influence learning and enjoyment when part of ongoing classroom activity. The educational value of digital games can be enhanced if game play is followed by instruction that builds on the ideas explored within the game. For example, community-college students who played a specially designed game to develop statistical intuitions followed by a lecture designed to formalize these intuitions had better transfer to a paper-and-pencil assessment than students who only played the game or only received the lecture (Arena and Schwartz 2013). Likewise, when games are integrated into classrooms, teachers’ talk and supportive instruction influences student learning (Bell and Gresalfi 2017; Erhel and Jamet 2013). Thus, another important next step is to add the learning ecology back into the framework when investigating the potential of different games to support learning.