1 Introduction

The market share and approaching ubiquity of the video game industry are ever increasing. The six billion dollars recorded for video game sales in the United States in 2001 grew to 16 billion dollars in 2010 (ESA 2011). Confirming this trend, the majority of youth own game devices and spend considerable amounts of time playing video games. Approximately 66 % of youth aged 8 to 18 years have cell phones and 76 % have iPods or MP3 players (Kaiser Family Foundation 2002). Approximately one hour is spent playing video games each day (Rideout et al. 2010). There is little wonder that over the past decade there has been an extended interest in video games, learning, and engagement among educational researchers and learning scientists (Kim and Chang 2010a, b; Young et al. 2012).

Despite deep-rooted concerns about the emotional and cognitive effects of video games on students’ academic performance (Maass et al. 2011), and questions about potential differences in the use and applicability of gaming technology for boys and girls (Lowrie and Jorgensen 2011), many educational researchers and learning scientists believe that video games can be used beneficially for students’ academic engagement and motivation. Oblinger (2006) exhorts the use of video games to foster environments in which students are actively involved and engaged in deeper learning. As an example, Barab et al. (2005) claim that educational video games have the potential to stimulate students to use complex cognitive processes in exciting ways. This is because the gaming context can be used to effectively motivate students using positive emotion and mild frustration to apply attention, memory, and motor skills to process information in a goal-directed manner. The authors, along with our research and development team, have made evidence-based efforts to develop and use video games to promote students’ academic engagement by including features of entertainment games that enhance challenge, interest, play and mastery. These game features have been identified as ones that can be appropriated by educationists and learning scientists (Gee 2010; Schell 2008).

Leveraging the groundwork established by colleagues in the academic and entertainment sectors, we explored the potential of educational video games to promote students’ mathematical engagement, which is known to be an important contributor to academic achievement (Kebritchi et al. 2010). Importantly, the reported study contributes to the literature base by examining educational video game effects using a detailed description of the components of engagement, categorizing the construct into three sub-domains: behavioral, emotional, and cognitive. One step further, the study pays special attention to the differential effects of video games for male and female students on these three sub-domains of engagement, examining gender differences noted in the previous studies (Lowrie and Jorgensen 2011; Mandinach and Corno 1985).

To implement the study, we recruited 107 fifth grade students to investigate the effects of an educational video game [APP] designed to foster pre-algebraic fractions concepts by eliciting mental actions of splitting, partitioning, and iterating. While mainly analyzing video game effects on student’s mathematical engagement, this study also explored the effects of student gender and prior mathematics achievement. The following overarching research questions guided the study:

  1. 1.

    Does the [APP] improve student’s mathematical engagement measured by an overall engagement score as well as scores for the sub-domains: behavioral, emotional, and cognitive?

  2. 2.

    Are there any differential effects of the [APP] for male and female students, controlling for prior mathematics achievement?

2 Literature review

2.1 Measuring three domains of engagement

Fredricks et al. (2004) emphasized the importance of conceptualization of three components of mathematical engagement in addition to overall engagement: behavioral, emotional and cognitive. Behavioral engagement refers to participating in work, doing required work, and following the rules; emotional engagement, having negative and positive poles, covers interest, happiness, anxiety, and belonging; and cognitive engagement reflects mindfulness and willingness to exercise effort to understand complicated ideas and master high-level skills. As defined by Barkatsasa et al. (2009), behavioral engagement is students’ demonstrations of concentrating and showing persistence for mathematics learning, adopting different strategies to solve mathematics problems, and trying to answer mathematics questions. Emotional engagement is students’ feelings about learning mathematics, such as joy, interest and satisfaction. Cognitive engagement is essentially defined as motivation, effort, and strategy use (Fredricks et al. 2004); this includes a psychological investment in learning, a desire to go beyond the requirements, and a preference for challenge (Newmann 1992). Fredricks and McColskey (2012) particularly highlighted the importance of empirically and theoretically supported studies analyzing differences and interactions of the three domains of engagement. The multifaceted concept of engagement of Fredricks et al. (2004) has been shared by other researchers such as Kong et al. (2003) and Darr (2012), all of whom adopted three sub-domains to define mathematical engagement.

2.2 Video games and engagement

Despite the ongoing effort of educational researchers and learning scientists to measure the effects of video games on academic engagement, the community has not reached consensus on the circumstances under which positive or negative effects result from the use of video games. For example, in a study done by Brom et al. (2011) with a video game that they developed in which students trained virtual animals to perform certain actions (i.e., dogs waving and parrots speaking), emotional engagement was examined by measuring students’ enjoyment in relation to a lesson on animal learning. Results did not demonstrate any differences in emotional engagement between students in the game and non-game groups.

In the midst of this argument, groups of researchers continue to report positive findings from the use of educational video games in the classroom. Annetta et al. (2009) reported a positive association between video games and student engagement using a 3-D Multiplayer Educational Gaming Application and virtual environments for students to learn science concepts by solving a problem-based mystery. The authors found that the students in the experimental group demonstrated significantly higher levels of engagement compared with those in the control group. Similarly, Arici (2008) employed an inquiry-based 3D Multi-User Virtual Environment game in a class in which students solved water quality issues in a virtual community and made scientific decisions reflecting the differing interests of various stakeholders. After a 2-week experimental period, Arici found that the engagement level of the video game group was significantly higher than the traditional instruction classroom group. Clark et al. (2011), who developed and implemented a game called Scaffolding Understanding by Redesigning Games for Education (SURGE) to teach physics to students in grades 7 to 9, found that those who played SURGE displayed high levels of engagement and enjoyment in class.

2.3 Video games and gender

There have been several studies reporting gender differences associated with video game play. Lowrie and Jorgensen (2011) collected data from 428 primary students aged 10 to 12 years and revealed that boys and girls were different in the types and duration of video games played. As hypothesized, boys spent more time playing video games than girls: almost 50 % of boys played video games more than three days per week while only 32 % of girls did so; in addition, approximately 34 % of boys played video games for more than 3 h per non-school day as compared to only 16 % of girls. On average, 25 % of boys preferred action types of games in contrast to 4 % of girls. The largest percentage of girls (approximately 29 %) preferred games that feature adventure, simulation, strategy, and role-playing. In video games that embedded mathematical concepts and ideas, boys and girls indicated different preferences. Boys preferred video games requiring visual and spatial skills, particularly interactions with maps that provided layout and perspective, while girls preferred problem-solving video games that involved logic and accurate computation.

Research also indicates discrepancies in gender differences in engagement toward learning based on video games. In Mandinach and Corno (1985)’s research on the differential effect of a video game on cognitive engagement of students, male students tended to indicate higher levels of cognitive engagement such as self-regulation, task-focused learning, and resource management. In contrast, the research results reported by Brom et al. (2011) showed no significant gender differences in emotional engagement. When students were asked to respond regarding their preferences for science learning through a video game, both girls and boys demonstrated high engagement, although boys’ engagement levels were slightly higher than girls.

2.4 Video games and mathematics achievement

Many studies have shown that student academic ability is closely associated with academic engagement. In a study conducted by Marks (2000), academic engagement was operationalized as a combination of a student’s effort, attentiveness, interest, and the completion of in-class assignments. In Marks’ study, a school student’s performance level was a significant predictor of academic engagement in mathematics and social studies classrooms. Similarly, Barkatsasa et al. (2009) found a significant positive relationship between math achievement and math engagement. High achieving students demonstrated higher levels of affective and behavioral engagement. This research also considered each student’s prior mathematical achievement level as an important factor of mathematical engagement and adopted it as a covariate in analytical models. Thus, the video game effects were examined after controlling for mathematics achievement effects on mathematical engagement.

3 Methodology

3.1 Participants and treatment

For this study, 107 fifth grade students from five classes from low performing schools in a rural school district in Virginia participated in the 10-day project. One class with 28 students was assigned as a control group in which students learned fractions using paper-and-pencil drills. The other four classes, totaling 79 students, were assigned to learn fractions by playing the [APP] on iPod Touches as a treatment group. On day one, the students completed the math engagement survey and solved fractions problems as pretests. For the next eight consecutive school days, the students spent 20 min per day learning fractions playing the [APP] or working on paper- and-pencil drills. On day 10, students completed the math engagement survey as posttests (See the instrument section below for details).

3.2 The [APP]

The [APP] is an educational video game developed by our research team to promote fractions learning for 5th grade students using iOS mobile devices (i.e., iPod Touch, iPad, and iPhone). To develop the educational game software application [APP]app), we followed the Incremental Development (iterative refinement and progressive elaboration) software development paradigm under a software engineering life cycle consisting of problem formulation, requirements engineering, architecting, design, programming, integration, and delivery/deployment. We employed the Xcode integrated development environment, Objective-C programming language, and a rich set of class libraries provided by Apple, Inc. under the object-oriented paradigm.

The game prompts students to produce candy bars of specified sizes to complete customer orders. Given a customer order, students select a whole candy bar of the same color and attempt to produce the customer order from that whole by partitioning (breaking) the whole into pieces and iterating (copying) one of those pieces a certain number of times. Research on students’ fractions learning indicates that engaging in the mental activities of partitioning and iterating can provoke development from part-whole conceptions of fractions toward measurement conceptions of fractions (Olive and Vomvoridi 2006). This and similar developments supported by the game fit a hypothetical learning trajectory (Simon and Tzur 2004), which informs ways to support students’ progress from basic fractions concepts toward more sophisticated fractions concepts and algebraic reasoning (Steffe and Olive 2010).

3.3 Instruments for mathematical engagement and mathematics achievement

This study used a mathematical engagement instrument that was composed of 33 items to measure before and after treatment overall math engagement levels and the three sub-domains of engagement: behavioral (11 items), emotional (11 items) and cognitive (11 items). Students were asked to choose one option (1 = strongly agree; 4 = strongly disagree) in response to each statement.

The study conducted several tests for reliability statistics from piloting using 151 fifth grade students and found defensible internal consistency for overall (α = 0.89), behavioral (α = 0.68), emotional (α = 0.84), and cognitive (α = 0.79) engagement. The reliability test results at the posttest demonstrated higher internal consistency than at the pretest (overall α = 0.91, behavioral α = 0.79, emotional α = 0.86, and cognitive α = 0.81).

3.4 Variables

The outcome measures of the study were the score changes in mathematical engagement from the pretest to the posttest. Therefore, positive values indicated improved mathematical engagement level of students and vice versa. Because the study also analyzed three sub-domains of engagement in addition to overall engagement, the four outcome measures were analyzed.

As for main predictor variables, this study used two variables: treatment (paper-and- pencil drill group = 0; [APP] group = 1) and gender (Male = 0; Female = 1). The study also included each student’s pre-test mathematics score in an effort to control for the effect of student’s prior ability with fractions for the analysis of mathematical engagement.

3.5 Analysis

The study conducted several preliminary analyses using descriptive statistics, correlation analyses, and independent samples T-tests to explore students’ engagement levels before examining the treatment effects. Additionally, a set of two-way ANOVA using Generalized Linear Modeling (GLM) was conducted to detect pre-existing differences of engagement.

As for main analyses, the study adopted a two-way ANCOVA to compare two treatment groups ([APP] vs. paper-and-pencil drills) and two gender groups (males vs. females) on mathematics engagement after controlling for preexisting ability in mathematics. In the analysis, four groups (males in [APP], females in [APP]; males in paper-and pencil-drills and females in paper-and-pencil drills) were compared. Four ANCOVA analyses using GLM were conducted to examine overall engagement and three sub-domains of engagement.

4 Results

4.1 Descriptive statistics and independent samples T-test

As shown in Table 1, the descriptive statistics revealed that students in the [APP] showed greater increases in overall, behavioral, and emotional engagement as compared with those in the paper-and-pencil drills.

Table 1 Descriptive statistics of pre- and post-math engagement scores
Full size table

The study conducted independent samples T-tests at the pretest to check the pre-existing engagement levels by examining the mean difference between the paper-and-pencil group and the [APP] group in four engagement outcomes. The results of T-tests were interpreted without reservation because the assumption of homogeneity of variances was not violated. Before the treatment, the overall engagement of the [APP] group (M = 94.13) was higher than that of the paper-and-pencil drill group (M = 89.96) although the change of the [APP] group from the pretest to posttest was small. Among the three sub-domains of engagement, the cognitive engagement of the [APP] group (M = 30.93) was significantly higher (T = −2. 07, p < .05) than that of the paper-and-pencil drill group (M = 27.89). Again the change of the [APP] group from the pretest to posttest was small. The other two engagement levels (emotional and behavioral) in the [APP] group were slightly higher than those of the paper-and-pencil drill group, but they were not statistically significant (see Table 2).

Table 2 Independent samples T-tests comparing pre-math engagement scores of [APP] and paper & pencil groups
Full size table

The second sets of T-tests were conducted to examine differences in engagement between male and female students at the pretest. There were no significant differences between two groups although male students displayed slightly higher overall and cognitive engagement levels (Table 3).

Table 3 Independent samples T-tests comparing pre-math engagement scores of male and female
Full size table

4.2 Two-way ANCOVA

The homogeneity assumption check was done to avoid extreme cases of statistical findings and reach valid interpretations and conclusions from the ANCOVA analyses. As shown in Table 4, the overall engagement model revealed that the assumption of homogeneity variance was not violated (Levene’s F(3, 85) = 2.57, p > .05). The main analysis results showed a significant interaction effect of treatment and gender (F(1,84) = 5.21, p < .05), indicating that overall engagement changes were different for males and females depending on the treatment group. Figure 1 shows that both male (Pretest = 96.26; Posttest = 97.70) and female students (Pretest = 92.06; Posttest = 93.71) in the [APP] demonstrated slightly increased overall engagement. However, in the paper-and-pencil group, male students displayed a sharp decrease in overall engagement (Pretest = 88.67; Posttest = 75.25) while female students had a comparatively small decrease (Pretest = 91.00; Posttest = 89.00).

Table 4 Two-way ANCOVA for math engagement scores
Full size table
Fig. 1
figure 1

Overall engagement change across gender and [APP] groups

Full size image

The main effect of [APP] which compared the overall engagement of the [APP] group and the paper-and-pencil drill group revealed a significant result with (F(1,84) = 9.80, p < .01), indicating that the two groups’ overall engagement levels were significantly different. The significant difference was due to the changes of both groups rather than the improvement of the [APP] group as shown in Fig. 1. While the [APP] group showed a slight increase of engagement, the paper-and-pencil group displayed decreased engagement.

The main effect of gender was not significant with (F(1,84) = 3.52, p > .05), indicating that there was no significant difference in overall engagement when compared male and female regardless of the treatment and control groups. Mathematics achievement did not reveal a significant effect on overall engagement (F(1,84) = 0.43, p > .05), indicating that the changes in mathematical engagement were not affected by prior achievement levels.

Similar results were noted in the behavioral engagement model. Although the assumption of homogeneity of variance was violated showing Levene’s value (F(3, 83) = 3.03, p < .05), the results were interpreted because ANCOVA is a robust statistic which allows a minor violation of the assumption. This model showed a significant interaction effect of treatments with gender (F(1,82) = 6.44, p < .05). Specifically, males (Pretest = 35.74; Posttest = 36.06) and females (Pretest = 34.89; Posttest = 35.33) in the [APP] group demonstrated slight increases in behavioral engagement from the pretest to the posttest. However, both males and females in the paper-and-pencil group displayed decreases in behavioral engagement. Again, males in the paper-and-pencil group displayed a large amount of decrease (Pretest = 33.75; Posttest = 35.33) while females in the paper-and-pencil group demonstrated a slight decrease (Pretest = 33.87; Posttest = 33.40).

The main effect of [APP] was also significant with ((F(1,82) = 11.85, p < .01), showing a significant difference between the [APP] and the paper-and pencil-drill groups in behavioral engagement. Again, the significant results were due to the changes of two groups: the [APP] group showed a slight increase and the paper-and-pencil group a decrease as shown in Fig. 2 and Table 4.

Fig. 2
figure 2

Behavioral engagement change across gender and [APP] groups

Full size image

The main effect of gender was significant with (F(1,84) = 6.45, p < .05), indicating that there was a significant difference in behavioral engagement when compared male and female from both groups. Mathematics achievement score did not have a significant effect on behavioral engagement (F(1, 82) = .01, p > .05), either.

The emotional engagement model violated the homogeneity assumption (F(3, 85) = 3.77, p < .05), though as noted above, the model is robust to violations of this assumption. As the interaction effect of treatment and gender was not significant in the emotional engagement model (F(1,84) = 0.86, p > .05), treatment and gender effects were analyzed separately. The change in emotional engagement of the [APP] group was significantly higher than that of the paper-and -pencil group (F(1,84) = 6.83, p < .05). Similar to behavioral engagement, students in the [APP] group had slight increases in emotional engagement (Male = .5; Female = .10) while the students in the paper-and-pencil group had decreases (Male = −3.83; Female = −2.00). This model showed neither a significant gender effect nor mathematics achievement effect on emotional engagement.

The cognitive engagement model did not show any significant treatment and gender effects and interaction effects with no violation of homogeneity assumption (F(3,84) = 1.80, p > .05). Despite no significant effects, only male students in the paper-and-pencil drill group displayed a decrease (M = −4.27). However, the other students had increases: female students in the paper-and-pencil group (M = .47); male students in the [APP] group (M = .69); and female students in the [APP] group (M = .13). The main effect of [APP], gender, and mathematics achievement did not reveal significant effects on cognitive engagement.

5 Discussion

With an aim to examine the educational effect of video games in 5th grade classrooms, this study empirically explored the effect of video games on math engagement compared to a control group with paper-and- pencil drills for the same amount of time (20 min of ten class sessions). The video game used in this study was the [APP], which was developed to improve fractional understanding of 5th grade students. The two-way ANCOVA produced the significant differences between the [APP] group and the paper-and-pencil drill group in overall engagement, behavioral engagement, and emotional engagement. The results were interpreted the significant results were due to the changes of both groups: the [APP] group had small increases while the paper-and-pencil drill group displayed big decreases in all engagement levels. This study adopted the paper-and-pencil drill group as a control group to emulate the regular math classroom condition in which teachers depend on paper-and-pencil drills to instruct pre-algebra concepts. As expected, the study noted big decreases of all engagement levels in the paper-and-pencil drill group. Moreover, the study expected the diminishing engagement levels of students in regular math classrooms considering the students who participated in this study were from low performing schools located in a rural area. As compared to the large drops of the engagement levels of paper-and-pencil drill groups, the [APP] group was able to make an increase, although it was insignificant improvement.

While exploring its effect, this study paid special attention to student gender and found a differential effect of the video game for male and female students. When compared to males in the paper-and-pencil drill group who displayed a drastic decrease in engagement, males in the [APP] group displayed a slight improvement in mathematical engagement, particularly in overall and behavioral engagement. On the other hand, females in the two groups did not demonstrate significant differences in any engagement domains.

A major contribution of this study is the examination of the effects of video games on overall mathematical engagement as well as the three sub-domains of mathematical engagement (behavioral, emotional, and cognitive engagement). Despite the fact that detailed attention to the specific domains of mathematical engagement is particularly important for intervention strategies using video games, little research has been done in this area. The study found some variations in the effects of the [APP] across the three sub-domains of mathematics engagement. Among the three sub-domains, there was a slight change in student’s behavioral engagement after treatment, and we found a differential effect for male and female students. However, the changes were not salient in either emotional or cognitive engagement. In behavioral engagement, both male and female students in the [APP] group showed small increases. In contrast, both male and female students in the paper-and-pencil drill groups displayed decreases in behavioral engagement, with male students exhibiting a sharp decrease. In emotional engagement, there was a noticeable difference between the [APP] group and the paper-and-pencil drill group, but there was no interaction effect. In other words, the change patterns of male and female students were not different in the [APP] and the paper-and-pencil groups. In the cognitive engagement domain, there were no significant effects of the factors noted.

These results evince the importance of considering not only overall engagement but also sub-domains of mathematical engagement in exploring the effect of video games. This study filled the research gap in the existing research, which has been focusing on only one component (Annetta et al. 2009; Arici 2008). The study findings are expected to make meaningful contributions to the field with comprehensive and rich discussion regarding the effect of video games on overall engagement and its subcomponents.

The study findings also highlight the importance of consideration of gender when exploring strategies to implement video games into classrooms. In this sense, this study provided the empirical evidence of a gender difference in the use of video games and contributes to the field, which does not present clear suggestions for effective uses of video games in school for boys and girls (Arici 2008; Brom et al. 2011).

Aligned with previous studies (Brom et al. 2011; Mandinach and Corno 1985), this study’s findings supported the potential value of the [APP] for male students. On the other hand, video games showed minimal improvement of mathematical engagement for female students. To make video games effective for female students, future studies should consider including game features which can lead girls to engagement. Researchers such as Lowrie and Jorgensen (2011) recommend that video games should include female-preferring features such as problem-solving components using logic and accurate computation as well as male-preferring features such as components using visual and spatial skills.

This study did not find a significant effect of math achievement on math engagement. This result does not agree with previous studies showing its significant effect (Barkatsasa et al. 2009; Marks 2000). This contrast may be caused by potentially differential effect of a video game on mathematical engagement depending on different math abilities, as found by Mandinach and Corno (1985). This study recommends an experimental study to explore the effect of differential math abilities on overall mathematical engagement and three subcomponents.

In contrast to the improved math engagement of students in the [APP] group, those in the paper-and-pencil groups experienced drastic decreases after the treatment. Especially, boys in the control group demonstrated a sharp drop in behavioral engagement. This may be due to the disappointment of the students who were not able to participate in the treatment, which involved playing a video game on an iPod Touch, particularly as the students were from rural areas with limited access to iOS mobile devices. Future studies should compare the changes of math engagement of these students when the same students learn fractions by playing the [APP] on iPod Touch. More importantly, the results of the study emphasize the importance of instructional activities. The paper-and-pencil drills in the control group can be a demoralizing factor of classroom learning, exerting a negative effect on student’s mathematical engagement. This study also suggests that future studies should explore other control conditions rather than paper-and-pencil drills which may have caused a confounding effect in the study. Using a control group condition which shows stable engagement levels from the beginning to the end of the study will serve more reliable baseline measures to examine treatment effects. In closing, we strongly recommend further studies exploring the effects of various video games and developing further instructional activities using video games that will promote students’ engagement and motivation of mathematics classrooms.