Keywords

Educational games have become a hot issue in the educational technology domain and are considered as a potential learning tool. Positive outcomes and effects have been claimed and educational effectiveness is expected from the use of games. Amongst others, educational games are expected to evoke intense engagement and motivation in the learning process (e.g., O’Neil, Wainess, & Baker, 2005; Vogel et al., 2006), to actively involve students in challenging situated problem solving (e.g., Becker, 2007; Garris, Ahlers, & Driskell, 2002), to enhance learning and understanding (Hayes & Games, 2008), and to improve student’s performance (Liu & Chu, 2010). Notwithstanding the popularity of educational games in education and the optimistic stance that is taken towards the potentials of games in education, empirical research, and evidence for the claims and expectations remain limited (Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012; Girard, Ecalle, & Magnan, 2013; Hays, 2005; O’Neil et al., 2005; Randel, Morris, Wetzle, & Whitehead, 1992; Sitzman, 2011; Tobias, Fletcher, Dai, & Wind, 2011; Vandercruysse, Vandewaetere, & Clarebout, 2012; Wouters, van der Spek, & van Oostendorp, 2009). There is lack of scientifically rigorous studies that pinpoint instructional design features that improve the instructional effectiveness of games (DeLeeuw & Mayer, 2011). This hampers drawing conclusions on the effectiveness of educational games but also results in insufficient guidance for game designers on how to develop effective games. In order to make a step forward with respect to this guidance, there is a need for studying specific characteristics of game-based learning environments (GBLEs)—rather than games as such (Aldrich, 2005)—as well as interactions between these characteristics and learner-related variables. Therefore, in this study we attempted to get more evidence on the assumed benefits of games by focusing on the effects of one specific characteristic related to the game (i.e., the type of game integration in the curriculum) on students’ motivation, learning processes, and acquired knowledge.

Integration in the Curriculum

Multiple claims are made about the added value of gaming in the math curriculum and various factors have been mentioned to affect math-game effectiveness. Game integration is one of these factors. The notion of game integration, however, is multidimensional. On the one hand, game integration can be described as the integration of the learning content into the (story line of the) game. Habgood and Ainsworth (2011) define this integration as intrinsic integration or the more productive relationship between educational games and their learning content. Clark and colleagues (2011, p. 2180) distinguish between “conceptually integrated” and “conceptually embedded” games. In the former type of games, the learning goals are integrated into the actual movement and gameplay mechanics which has the potential advantage of engaging the player with the learning content in the game during a longer amount of time. In the latter type of games, this is not the case and also other interactions (with no referral to the learning content) are involved in the game.

On the other hand, game integration can refer to the use of the game (and thus its integration) in the classroom. In this study, we focus on this latter meaning of game integration, which is also multidimensional. For instance, Demirbilek and Tamer (2010) found in their study that teachers utilize computer games during math lessons in different ways, for example, as evaluation purpose, as remediation stage, as reinforcement, distractor, or bonus. Various studies have investigated concrete ways of game integration. Most of these studies focus on cooperative and competitive gaming (Ke, 2008a, 2008b; Ke & Grabowksi, 2007; Vandercruysse, Vandewaetere, Cornillie, & Clarebout, 2013) and the degree of which free choice to play a game influenced learning results (Barendregt & Bekker, 2011). The results are interesting but do not pertain to the essence of this study. In this study, we investigate the effect of the integration of the learning content in the game in the curriculum of the students, and more specifically the effect of the absence or presence of an explicit link between the learning content in the game and the curriculum (i.e., the learning content in the classroom) of the students. In this study, the curriculum is interpreted based on the definition of Walker and Soltis (1997):

The curriculum as we use the term, refers not only to the official list of courses offered by the school—we call that the ‘official curriculum’—but also to the purposes, content, activities and organization of the educational program actually created in the school by teachers, students and administrators (p. 1).

The integration of the game in the curriculum refers to the integration of the game in the classroom activities. The literature reviews of Hays (2005) and Tobias and colleagues (2011) indicated that a stronger integration of games in the instruction program (or curriculum) promotes the learning process. Tobias and colleagues (2011) pointed to the fact that games that are not related to the instruction program might be fun, but probably do not promote the cognitive possibilities of the learners. This might be due to the fact that games don’t appear to help students to make the leap from tacit understanding during gameplay to more formalized knowledge in the classroom (Clark et al., 2011). In order for students to make the connections between the game and the more formalized knowledge demanded in a school-based context, aids (or scaffolds) are required (Clark et al., 2011). In this study, aid is provided by linking the learning content in the game with the math content during math class. By linking the learning content in the game explicitly with the learning content in the classroom, the (learning) goal of the GBLE becomes clearer to the students. This might be beneficial for the students because clear goals are supposed to stimulate engagement and engage players’ self-esteem (Akilli, 2007; Bergeron, 2006; Garris et al., 2002; Hays, 2005; Malone, 1980; Prensky, 2001). Hence, we might assume there is a relationship between the degree of integration of a game in the curriculum and the learning effect. Din and Calao (2001) already investigated the two extremes; being the difference between games that were integrated vs. games that were not integrated in the curriculum. In this study, we do not investigate the extremes, but more the continuum of integration: a strong integration vs. a weak integration in the curriculum. The effect of different degrees of integration (strong vs. weak integration) on mathematical performance and learners’ motivation will be investigated.

Learners’ Perception: Moderating the Influence of the Teachers’ Instruction

“Learners are active actors in learning environments and not mere consumers of instructional designers’ products” (Lowyck, Elen, & Clarebout, 2004, p. 429). The so-called mediational paradigm (Winne, 1982, 1987) is based on this thought and emphasizes the crucial impact of students’ cognitive processes. This contrasts with the process-product paradigm from earlier days (but also nowadays in some research studies, cf. Vandercruysse et al., 2012), in which it was assumed that an instruction method (process) directly influences learning outcomes (product) of students. Now, researchers are more and more convinced that learners actively construct their own knowledge and interpret the teachers’ instructions. The way students interpret the instruction evokes different cognitive processes (Lowyck et al., 2004) which then lead to different learning outcomes (Winne, 1987). Unintended interpretations of the instruction by students might lead to unintended learning results (Lowyck et al., 2004). Entwistle (1991), Salomon (1984), and Shuell and Farber (2001) share this thought of the moderating role of students’ perception. More specific, Salomon (1984) demonstrated that students’ differential learning may depend on what they perceive the learning material to be. If students perceive the material as “easy” leisure time activities, they invest less mental effort compared to students who perceive the material as more instructional (Salomon, 1984). Hence, although a teacher may decide to implement a game in the classroom, the perception of the students will determine to what extent and how this implementation will influence their learning.

From this point of view, we claim the importance of taking students’ perception into account. In this study, students’ perception is defined as (1) students’ expectations about the goals of the environment and more specific whether the players think of the game as a leisure time activity (something fun) or an educational one (something more akin to work, perceived playfulness) and (2) the degree to which students believe that using GBLEs will enhance their performance on what the GBLE focuses on (perceived usefulness) (Vandercruysse et al., 2015).

Students’ perception about learning environments is not only related to the instructional method (i.e., the way the educational game is introduced to the student and integrated in the curriculum) and performance, but also their intrinsic motivation (Lowyck et al., 2004). Intrinsic motivation gets stimulated when students perceive instruction as important and relevant (Kinzie, 1990; Ryan & Deci, 2000). The study of Herndon (1987) also concludes that students’ intrinsic motivation is higher when students are confronted with relevant and interesting instruction for the students compared to instruction that does not take students’ interest into account. Hence, we may assume that students, who perceive the game environment as more useful and effective, will show higher intrinsic motivation than students who perceive the environment as less useful and effective.

The Present Study

In this study, we investigate the impact of the integration of the game in the curriculum, by which we focus on the way this integration takes place. Two experimental conditions are set up. In the weak integration condition, students get the chance to play with the GBLE during 2 h as a reward for their efforts in the last lessons; during instruction time only the fun-part of the GBLE is mentioned. In the strong integration condition, students are told they need some extra exercises on the content they had during math class. For a change, the exercises are implemented in a GBLE and will help to improve their math skills. In the latter condition, the link between the learning content in the GBLE and the curriculum of the students is made explicit, while in the former condition this is not the case. The research focus of this study is the relation between on the one hand the explicitness of the link between GBLE learning content and curricular/classroom learning content and on the other hand students’ performances, motivation, and perception.

Based on the literature, we suppose that, because of the explicit link between the learning content in the game and the learning content in the classroom, in the strong integration condition, students have a better idea about the goal of the GBLE. This might lead to greater (intrinsic) motivation for the strong integration condition (hypothesis 1) because clear goals are supposed to stimulate engagement and engage players’ self-esteem (Akilli, 2007; Bergeron, 2006; Garris et al., 2002; Hays, 2005; Malone, 1980; Prensky, 2001). Additionally, students in the strong integration condition are supposed to perform better than students in the weak integration condition (hypothesis 2) because a stronger integration of games in the instruction program (or curriculum) is assumed to promote the learning process (Hays, 2005; Tobias et al., 2011) due to the fact that in the strong integration condition students will be more able to make the leap from tacit understanding during gameplay to the more formalized knowledge in the classroom (Clark et al., 2011). Additionally, a higher perceived usefulness and perceived playfulness (hypothesis 3) is assumed for the strong integration condition. This assumption is based on the mediational paradigm which assumes that the effect of game integration in the curriculum on students’ motivation and performances is influenced by students’ perception (Entwistle, 1991; Lowyck et al., 2004; Salomon, 1984; Shuell & Farber, 2001). In the strong integration condition, the GBLE might be more perceived as a useful means to learn math; while in the weak integration condition students might perceive the environment more as a leisure time activity. Additionally, we assume that this difference in perception will result in a difference in motivation and performance. We expect that students who perceive the environment as a useful means to learn math will be more intrinsically motivated (hypothesis 4) and perform better on the mathematical exercises (i.e., solve them more correctly) during the gameplay and afterwards (hypothesis 5) than students who perceive the environment as a pastime (Lowyck et al., 2004; Salomon, 1984). Finally, students that perceive the game environment as a useful tool to learn math before the gameplay, will keep this perception after the gameplay (hypothesis 6).

Method

Participants

The sample of this study consisted of 49 vocational track students. Participants were selected from the third and fourth year of secondary vocational education in Flanders (Belgium). Five classes from one secondary school were selected. The students all followed hairdressing courses which resulted in an unbalanced gender division (only two male students). In Table 1, an overview is given of the conditions and the number of students that participated in the study. For all involved students, this research was organized during the course Project General Subjects (PGS).Footnote 1 The participants formed a homogeneous group with respect to cultural background; they lived in the same region and had similar educational background, computer access, and ICT knowledge. The age range varied between 15 and 18 years old (M = 16.43; SD = .83).

Table 1 Conditions with number of students who initially participated (and the amount of students who actually participated in the whole study)
Full size table

Because students who did not complete the whole study were discarded from the analyses, one student was removed from the dataset. This resulted in 48 participants for whom data on all measured variables were available.

Design

A prepost between subject design with experimental condition (weak integration vs. strong integration) as a between subjects variable was used. Two experimental conditions were defined. In the weak integration condition, students were told that as a reward for their intensive work during the math class, without specifically referring to the mathematical content, they got playtime. During the instruction, the fun and leisure component of the game instead of the learning goals were emphasized (i.e., “We organized a gameplay session because you did your best during the previous courses and you really deserve it. It’s supposed to be fun, so we can get a fresh start afterwards with a new topic”). Hence, playing the game in the weak integration condition was not introduced as being part of the curriculum. In the strong integration condition, students were told they got some additional exercise time for practicing fractions. So students got the chance to practice their calculations with fractions in line with their math course, by playing the game. During the instruction, an explicit link between the math course and the exercises in the game was made and the learning goal and the opportunity for the students to have some extra exercises were emphasized (i.e., “We will practice a bit further on fractions similar to what we did in other mathematics lessons but now by playing a math game. Try to do your best because the exercises in the game will help to improve your fraction calculating skills”). Hence, playing the game in the strong integration condition was part of the curriculum.

Materials

GBLE: Monkey Tales. An existing 3D game was used as GBLE, namely the museum gameFootnote 2 from the Monkey Tales series (LarianStudios).Footnote 3 In the game, players have to beat Carmine Pranquill, a huge dinosaur, which has conquered the museum. This can only succeed by passing through all the rooms in the museum. Every room contains two challenges: (1) solving a 3D puzzle-game and (2) winning a mathematical mini-game. A player can only win the mini-games by showing better math skills as compared to the opponent (a monkey).

Four different kinds of mini-games are implemented in the museum game. See Fig. 1 for an example of a challenge in the mini-game “balloons pop-up” which is a shooting gallery. The math-assignment appears at the bottom of the screen (i.e., “Shoot on the fractions that equal 1/5”) and on the treadmill, cards with possible answers pop-up on the screen (i.e., 3/15 and a bonus card). By using the mouse to aim and throw a ball towards the cards (left click) with the correct answer, they gain points (blue/left score). By choosing—as fast as they can—all the right answers, they can beat the monkey (their opponent—red/right score).

Fig. 1
figure 1

Example comparison task: Which fractions equal 1/5?—Monkey Tales from Larian Studios

Full size image

The museum game is originally intended for third grade primary school children as rehearsal and additional practice of math content learned during math courses. For this experiment, the content was adapted to our target group and their curriculum (i.e., second grade vocational track students). All mini-games in the environment are related to comparing, adding, and multiplying fractions. Different difficulty levels concerning fractions are implemented (see Table 2) based on (1) the range of numbers of the denominator and nominator in the fractions and (2) the operations students have to conduct with the fractions.

Table 2 Overview of different types of exercises implemented in the game environment according to the different mini-games
Full size table

Measurements. The measurements in this study are threefold: We measured students’ motivation, their performances, and their game perception.

Motivation. Students’ premotivation (before the intervention started) was measured with subscales of the Dutch version of the motivated strategies for learning questionnaire (MSLQ; Pintrich, Smith, Garcia, & McKeachie, 1993). This self-report instrument assesses students’ motivational orientations and their different learning strategies on a 6-point Likert scale. For this study, the subscales for intrinsic goal orientation (four items, e.g., “In class, I prefer course material that arouses my curiosity, even if it is difficult to learn,” α = 0.78), extrinsic goal orientation (four items, e.g., “Getting a good grade in this class is the most satisfying thing for me right now,” α = 0.85), and task value (four items, e.g., “I am very interested in solving fractions,” α = 0.88) were administered. The higher a student scores on these subscales, the higher his/her premotivation. Correlations between the three subscales are positively significant (r Task Value−Intrinsic Goal = .80, p < .001; r Task Value−Extrinsic Goal = .81, p < .001; r Intrinsic Goal−Extrinsic Goal = .78, p < .001). Reliability of the MSLQ, measured by three subscales, is α = 0.94. Students’ premotivation was operationalized as the sum of the scores on the three subscales.

To measure students’ intrinsic motivation during completion of the tasks (i.e., playing the educational game), a post-assessment of students’ motivation was done wherein students were instigated to reflect on their motivation during task completion. Students filled in the Dutch version of the intrinsic motivation inventory (IMI; McAuley, Duncan, & Tammen, 1987; Plant & Ryan, 1985). Students completed two IMI subscales: the interest/enjoyment subscale (seven items, e.g., “I enjoyed playing this game very much,” α = 0.91) and the perceived competence subscale (six items, e.g., “I think I am pretty good at playing this game,” α = 0.87). The correlation between the interest/enjoyment subscale and perceived competence subscale was positively significant (r = .57, p < .001). Again—and in line with the MSLQ—both subscales are taken together (α = 0.91) and the sum of both subscales is used for analyses.

Performance. In a self-developed pre- and posttest students’ math performance concerning calculating factions was measured. Both tests, with comparable difficulty level, contained 30 questions (30 items, α pretest = .83 and α posttest = .87) with only one possible correct answer. There was no time-limit. In Table 3, an overview is given of the test-items. As Table 3 shows, there is a considerable overlap between the questions in the tests and the exercise types in the mini-games (i.e., questions concerning comparing, adding, subtracting, and multiplying fractions). Additionally, three transfer questions are presented to the pupils, more specific, these questions concern proportional reasoning problems.

Table 3 Overview of different types of questions presented in pre- and posttest
Full size table

Next to this pen-and-paper math performance during the pre- and postphase, also in-game math performance is taken into account. Therefore, the score students received in the mini-games and the amount of mini-games students were able to win were used as indicators. These in-game score parameters, however, are possibly an underestimation of students’ math ability because of the difficulty of the mini-games which also require gaming- and puzzle-solving skills of the students.

Game perception. The game perception scale (GPS; Vandercruysse, Vandewaetere et al., 2015) was used to measure students’ perception of the GBLE. This questionnaire measures (1) students’ expectations about the playfulness of the GBLE and (2) the degree to which a student believes that using a GBLE will enhance his or her performance. Both aspects are represented in a subscale of the GPS: the perceived playfulness subscale (three items, as suggested by Vandercruysse et al., 2015; for example, “I was playing the game rather than working/learning,” α pre = 74 and α post = .85) and the perceived usefulness subscale (five items, e.g., “I think that playing this game is useful for learning fractions,” α pre = .77 and α post = .91).

Procedure

The study started with a pretest session of 1 h. During this session, a short refresher course on math was given as introduction. Although calculating fractions is part of the curriculum, this activated their prior knowledge (Merrill, 2002). In line with the mathematical content that was implemented in the game for this experiment (see Table 2), some general information concerning fractions and instruction related to comparing, adding, subtracting, and multiplying fractions was focused on . This introduction was followed by the questionnaire which measured students’ premotivation (MSLQ) and perception of the GBLE (GPS). Also the pretest, which students had to fill in individually and without using a calculator, was presented to the students.

After this pretest session, students received their instructions which varied depending on the condition they were assigned to (see design). This was followed by a playtime session which lasted for 2 h.

After the playtime session, students received the 30-item posttest and postquestionnaire which measured postexperimental motivation (IMI) and game perception (GPS). Again students were stimulated to work individually and without the aid of a calculator (for the posttest). This session took approximately 1 h.

As previously mentioned, the experiment organized during the PGS course of the participants. Since students in vocational education weekly have 6 h of PGS, we strived for a maximum time interval between the pretest, intervention, and posttest of 1 week.

Results

For all analyses, a significance level of α = 0.05 was set. After detecting for outliers, one participant was excluded, which resulted in 47 participants. To investigate possible significant differences between the classes, multilevel analyses were conducted for all the dependent variables of the analyses. None of the analyses revealed a significant difference between the classes. Hence, the differences between classes were not taken into account in the following analyses.

Initial Differences Between the Two Experimental Conditions

To identify possible initial differences between both conditions, two ANOVAs were conducted with condition as independent variable and score on the pretest and premotivation questionnaire (MSLQ) as dependent variables. Additionally a MANOVA, with the two subscales of the GPS as dependent variables and condition as independent variable was done. Concerning the pretest, the mean scores were 61.47 % (SD = 19.32 %; with a minimum of 26.67 % and maximum score of 90.00 %) for the weak integration condition and 65 % (SD = 13.28 %; with a minimum of 36.67 % and a maximum of 90.00 %) for the strong integration condition, which is quite high for our target group. There was no significant difference between both conditions (F(1, 45) = .52, p = .48) with respect to students’ score on the pre-test. Also for students’ premotivation as measured by the sum of the score on the subscales of the MSLQ (M weak integration condition = 38.09; M strong integration condition = 30.05) with a minimum of 12 and a maximum score of 72, no significant difference was found (F(1, 39) = 3.52, p = .07). The MANOVA, using Wilks’s statistics, showed a significant initial difference between both conditions (Wilks’s λ = .80; F (2, 40) = 4.98; p = .012, η 2 = .20) concerning their GPS-subscale scores. Separate univariate ANOVAs on the outcome variables revealed no significant difference for their perceived playfulness of the GBLE (F(1, 41) = 2.59, p = .12), but we did find a significant difference between both conditions for their perceived usefulness (F(1, 41) = 8.46, p = .006). The weak integration condition (M = 18.65, SD = 4.99; with a minimum of 8 and a maximum score of 48) scored higher for their perceived usefulness than the students in the strong integration condition (M = 14.55, SD = 4.14) and perceived the GBLE as more useful before the intervention took place. Therefore, and because we assumed that students’ game perception would influence students’ motivation, performance, and perception, we corrected for students’ pregame perception (more specific their perceived usefulness and perceived playfulness) in the following analyses.

Effect of Curriculum Integration on Students’ Motivation (Hypothesis 1 and 4)

Because we supposed that students in the strong integration condition would show greater (intrinsic) motivation (hypothesis 1) the relation between curriculum integration and students’ intrinsic motivation was investigated with an ANCOVA. Condition was used as factor and motivation, measured by the sum of the scores on the interest/enjoyment subscale and perceived competence subscale, as dependent variable. Students’ GPS score on the perceived usability subscale and perceived playfulness subscale were used as covariates.

Results show that curriculum integration, controlled for both GPS subscales, is not significantly related to students’ postexperimental intrinsic motivation (F(1, 35) = .45, p = .51). The weak integration condition (M = 45.42, SE = 2.47) is not significantly different from the strong integration condition (M = 42.92, SE = 2.54) regarding their intrinsic motivation. Hypothesis 1, which expected a relation between the degree of curriculum integration and students’ (intrinsic) motivation during gameplay (and more specific that the strong integration condition would show greater intrinsic motivation), was not confirmed.

Hypothesis 4, in which we expected that students who perceived the environment as a useful means to learn math would be more intrinsically motivated, was partly confirmed since a significant effect was found between students’ perceived usability score and their postexperimental intrinsic motivation (F(1, 35) = 14.12, p = .001, η 2 = 0.29). More specifically, students who perceived the game as more useful for their learning, prior to the gameplay showed higher scores on self-reported intrinsic motivation as measured after gameplay. However, no significant relation was found between students’ perceived playfulness and their postexperimental intrinsic motivation (F(1, 35) = .002, p = .96).

Effect of Curriculum Integration on Students’ Performance (Hypothesis 2 and 5)

Math performance. An ANCOVA with condition as factor, the posttest score as dependent variable and perceived usefulness and perceived playfulness as covariates was conducted because a stronger integration of games in the instruction program (or curriculum) was assumed to promote the learning process (hypothesis 2). The ANCOVA revealed no significant effect of curriculum integration on students’ performance on the posttest after controlling for their perceived usefulness and perceived playfulness (F(1, 39) = .34, p = .56). After playing the game with a different instruction, and thus a different integration in the curriculum, students in the strong integration condition (M = 67.03 %, SE = 4.43 %) scored not significantly higher than students in the weak integration condition (M = 63.31 %, SE = 4.10 %). Hypothesis 2, in which a difference was expected, was not confirmed. As was the case with the pretest, a large variation in scores was found which indicated that some students scored very low on the posttest (minimum = 33.33 %) and other students scored very high (maximum = 100 %).

The relation between the perceived usefulness and the posttest score (hypothesis 5) of the students was not significant (F(1, 39) = .16, p = .69). Also the relation between the perceived playfulness and the posttest score of the students was not significant (F(1, 39) = .95, p = .34). The way students perceived the game (concerning its usefulness and playfulness) before the intervention was not related to their performance on the posttest after playing the game. Hypothesis 5, in which we assumed that the game perception would be positively related to students’ performance, was not confirmed.

Game performance. To investigate the relation between the curriculum integration of the game and students’ game performance (hypothesis 2), two ANCOVAs were conducted with condition as factor, perceived playfulness and perceived usefulness as covariates and the performance (measured with the total game-score and amount of mini-games won) as dependent variables.

The first ANCOVA with the amount of mini-games won as dependent variable, revealed no significant effect of curriculum integration, controlled for perceived usefulness and perceived playfulness, on students’ game performance (F(1, 36) = 3.74, p = .06). Students in the strong integration condition (M = 10.30, SE = 0.76) finished not significantly more mini-games than students in the weak integration condition (M = 8.16, SE = 0.72) and thus progressed not significantly further in the game.

The results of the second ANCOVA in which the total game-score is the dependent variable revealed a significant effect of the curriculum integration, controlled for pregame perception, on students’ performance in the game (F(1, 35) = 6.43, p = .02, η 2 = .16). The strong integration condition (M = 25 050.76, SE = 2049.22) scored significantly higher in the game than the weak integration condition (M = 17 653.16, SE = 1882.83). Hence, hypothesis 2 was only partly confirmed for the in-game performance of the students.

Further, the relation between the perceived playfulness and the amount of mini-games won (F(1, 36) = .12, p = .73) and the relation between the perceived usefulness and the amount of mini-games won (F(1, 36) = .49, p = .49) were both not significant. Also the relation between the perceived playfulness and the total game-score (F(1, 35) = .99, p = .33) and the relation between the perceived usefulness and the total game-score (F(1, 35) = .01, p = .93) was not significant. The way students perceived the game (concerning its usefulness and playfulness) before the intervention was not related to their in-game performance. Hypothesis 5, in which we assumed that game perception would be positively related to students’ performance, was not confirmed.

Effect of Curriculum Integration on Students’ Game Perception (Hypothesis 3 and 6)

Finally, the relation between curriculum integration and students’ perception of the environment after gameplay was investigated (hypothesis 3). A MANCOVA with condition as factor and students’ perceived usefulness (post) and perceived playfulness (post) as dependent variables was conducted. The scores on the two GPS subscales measured before the game-play were used as two covariates. The results showed no significant difference between both conditions related to their score on the two post GPS subscales (Wilks’s λ = .97; F(2, 35) = .50; p = .61). Hence, hypothesis 3 was not confirmed.

Hypothesis 6 instead was partly confirmed because a significant relation was found between the preperceived usefulness score and the postperceived usefulness score (F(1, 36) = 11.57, p = .002, η 2 = 0.24). As was expected, the more students perceived the environment as a useful game environment before gameplay (b = 0.62), the more they perceived the environment as a useful game environment after gameplay. No significant relation was found between the preperceived usefulness and the postperceived playfulness (F(1, 36) = .05, p = .83), between the preperceived playfulness and postperceived usefulness (F(1, 36) = .40, p = .53) and between the preperceived playfulness and postperceived playfulness (F(1, 36) = 2.68, p = .11).

Discussion

This study investigated the influence of the (weak or strong) integration of an educational game in the curriculum, and more specific the absence or presence of an explicit link between the learning content in the game and the learning content in classroom/curriculum, on students’ motivation, (in-game) math performance, and perception. Based on the literature, we expected that the condition in which the game was strongly integrated in the curriculum would show greater intrinsic motivation and better performances (Hays, 2005; Tobias et al., 2011). Additionally, the mediational paradigm (Winne, 1982, 1987) assumed that learners’ perception of the environment influenced the effect of curriculum integration on motivation and performance. The results showed that the strong integration condition indeed outperformed the weak integration condition for the in-game performances, more specific for the game-score (but not for the progress through the game, i.e., the amount of mini-games won). The other hypotheses however could not be confirmed.

The first finding confirms the assumption that students in the condition in which the learning content in the game (i.e., mathematics; operations with fractions) was strongly integrated (i.e., explicitly linked) in the curriculum, scored better during gameplay than students in the condition in which this integration was more weakly present. More concrete, the students from the strong integration condition obtained a higher game-score than the students in the weak integration condition. This is in line with the expectations (Hays, 2005; Tobias et al., 2011) that game integration in the curriculum enhances students’ game performances (which is a reflection of students’ math performances and their puzzle solving and gaming skills). However, one of the striking findings of this study is that, although learners in the strong integration condition were significantly more successful in the game (as indicated by their high game-score), their ability to solve fraction exercises measured with the posttest, which related to the game content, was not significantly different from the students in the weak integration condition. The scores on the pretest were already quite high for this target group, and only a slight progression was found for both conditions after the gameplay (i.e., approximately 2 %). A possible explanation is that the students did not make a connection between the content in the game and the content that was presented in the tests (Barzilai & Blau, 2014) and that the operationalization of the integration in the curriculum could be more explicitly elaborated (see further). Additionally, the time interval of the experiment was limited to 1 week. It might be interesting in future research to implement a long-term measurement, with multiple measurement moments, to investigate whether the students in the strong integration condition would continue to outperform the students in the weak integration condition after a longer period of time. Furthermore, it might also be interesting to investigate if a longer implementation of the GBLE in the classroom has an(other) effect on students’ (in-game) performances.

In contrast to our expectations, the strong or weak integration of the game environment in the classroom was not related to students’ motivation. A possible explanation for these findings is the low premotivation of the participants. Students in the weak integration condition had a mean premotivation score of 38.09 and students in the strong integration condition of 30.05 while the maximum score they could reach was 72 (and a minimum of 12). According to Winne (1987) insufficient (pre-)motivation can lead to ineffective instructional methods. Another possible explanation is that in both conditions different processes were influencing students’ intrinsic motivation. In the literature, it seems that students’ intrinsic motivation is influenced (and stimulated) when they perceive the game as more relevant (Herndon, 1987; Kinzie, 1990; Ryan & Deci, 2000). Students who belonged to the strong integration condition, perceived the game as more relevant and were more intrinsically motivated. In the weak integration condition however, students might also be intrinsically motivated, not because of the perceived relevance of the game, but because of the reward they received, that is, the game was introduced as a reward for their hard work during previous lessons (Deci, Koestner, & Ryan, 1999). So in both conditions, intrinsic motivation might have been stimulated, although only slightly, but as a result of two different processes.

Again unlike the expectations (Winne, 1982, 1987), no significant difference between students’ perception was found. More specific, integrating the game in the curriculum did not reveal any significant differences in students’ perception about the usefulness and playfulness of the educational game. A possible explanation here is that students’ perception were influenced by other factors than we intended to. Possibly the fact that students knew they participated in scientific research influenced their expectations and consequently their results (Grabinger, 2009; Vandercruysse et al., 2013). Because students participated in our study, they might not perceive the game as a part of the curriculum (in the strong integration condition) which may explain the lack of difference in perception. An additional possible explanation might be the study procedure, more specific the moment of measurement. Students needed to fill in the premotivation and preperception questionnaire before they received the introduction and instruction. This might have led to unintended misunderstanding of the students about the GBLE used in the study because they only saw the environment after filling in the questionnaires. Previous experiences (or the lack of such experiences) with other GBLEs might have influenced their responses.

However, the study revealed support for the supposed relation between the pre- and postperceptions of the students. Additionally, students who perceived the GBLE as an environment that was useful to learn solving fractions (i.e., their perceived usefulness) were more intrinsically motivated and perceived the environment as more useful after the gameplay. These findings emphasize the importance of students’ perception in game-based learning processes. Unfortunately, this effect was not found for students’ (in-game and posttest) performances.

A limitation of this study might be the operationalization of the integration in the curriculum. The limited link between the learning content in the game and the curriculum (even in the strong integration condition) might explain the lack of significant differences between both conditions. Therefore in a subsequent study, the operationalization of the integration of the GBLE will be based on the suggestions of Felicia (2011) who suggests based on Gagné’s “nine events of instruction” that game integration in the curriculum contains three steps (Felicia, 2011). Before the students start to play the game, teachers need to identify learning objectives, explain the objectives, demonstrate the game, and explain how common tasks are performed. A second step is the gameplay session. During this gameplay, teachers explain or clarify possible confusions and intervene shortly during “mini-teaching moments” to have an input that is essential for the understanding of the curriculum and to progress in the game. In the third and last step, a debriefing is organized. During this session, a connection is made between the curriculum and the game after play. This operationalization is more explicit than the operationalization in this study which only contained the first step. A link between learning content in the game and curriculum was only made explicit before the gameplay. The link during and after gameplay was lacking. Also Watson, Mong, and Harris (2011) and Charsky and Mims (2008) emphasized the importance of a short debriefing after gameplay during which students are learning to comprehend their mistakes and are stimulated to reflect which heightens the chance on transfer (Watson, Mong, & Harris, 2011).

Another limitation is the limited amount of participants. Although 51 participants were recruited, some analyses were only conducted with 36 participants because of incompletely filled-in questionnaires. This might be a possible additional explanation for a substantial decrease of the power of the study which reduces the chance for finding significant effects. Furthermore, the small group of participants seemed to be a heterogeneous group because of the high standard deviations and big range of the scores. It might be that, because of this heterogeneity, we did not find significant effects with an F-test because the denominator was very high. Another possible disadvantage of the participants in this study is the over-representation of girls. It is argued that girls have less initial computer and game knowledge, possibly resulting in a greater difficulty in using a game application (Vandercruysse et al., 2012). Also the learning time may have been too short in order to support deep learning (i.e., one-shot). Mean playtime was 80 min which might be too short for finding learning and motivational effects. In the next study, we will try to take into account these limitations.

In sum, this study only partly answers the question how the integration of an educational game is related to students’ motivation, performance, and perception. We only found a significant difference in students’ in-game performance. Although teachers are often convinced that using games with a stronger integration is advisable (Demirbilek & Tamer, 2010; Kebritchi, 2010; Koh, Kin, Wadhwa, & Lim, 2012), this study indicates that the integration of the game in the curriculum is only significantly related to students’ in-game performance and thus yields no influence on their score on a regular paper-and-pencil test, their intrinsic motivation and their game perception. Obviously, further research is warranted in which a more thorough operationalization of the game integration is used. Additionally, in this study only one operationalization of game integration was investigated. As mentioned in the introduction, game integration might also be operationalized in a completely different way. Further research could also focus on intrinsically integrated games and the effect this type of integration has on students’ performance, motivation, and perception.