Keywords

1 Introduction

1.1 Serious Games for Mathematics Learning

Children nowadays are born in a world that shows the rapid growth of various multimedia technologies, enjoying and spending more time playing digital games than their previous generations [10]. Therefore, digital games with learning purposes, known as serious games, have become an increasingly important educational method to keep children motivated [10, 31]. Compared to traditional instructional material, such as textbooks, serious games are hypothesized to have great advantages for children in terms of more motivation [5, 12, 32], greater learning achievements [5, 14, 29, 31], providing engaging and entertaining experiences [3, 12, 32], and customization to different learning abilities [14, 15]. Children express that they are more willing to spend time learning with games, which have been part of their daily life since a young age and are more enjoyable to engage with than traditional instruction [10].

Mathematics learning has been a primary concern in the educational system around the world, as children frequently experience mathematics as a difficult subject during their primary school years [28]. Learning motivation and interests are suggested to play an important role in children’s mathematics performance at school [1], where low feelings of competence and engagement for mathematics predict poorer mathematics performance [1, 8]. What’s more, children have different abilities in learning mathematics and need to prioritize their goals according to their abilities that best match their personal needs [1]. Therefore, recently some serious games for mathematics learning have emerged and reported to effectively enhance the motivation and enjoyment of children in mathematics learning [17, 19, 30].

1.2 Augmented Reality Games for Learning

Although the above serious games were effective for motivating students in learning and improving their learning performance, overall there is little evidence that serious games are considered more motivating than traditional instruction [31]. Therefore, more research needs to be done on how serious games should be designed to be engaging. In addition, problems have been reported with successfully integrating serious games in the classroom. The computers to play the game on are regularly located in another room and games are not designed to fit into standard classroom hours, leading to scheduling problems [27]. This physical separation also makes it more difficult to integrate games with existing instructional materials such as textbooks and blackboards, even though games are more stressful in reaching their learning goals when they supplement existing instruction [31], and they are more likely to be adopted by teachers when they blend into the curriculum [6]. Lastly, the tangibility, possession, feeling of turning pages, and better information comprehension of the physical textbook [11, 13, 33] are often preferred by students.

Augmented reality (AR) technology is able to combine the advantages of serious games and physical objects, allowing children to interact with and explore virtual objects on the top of real-world objects, completing tasks, learning concepts, and practicing knowledge in both the real and virtual world [15]. To be more specific, AR technology can improve the immersion of children in the learning content [24]. The appearing of AR elements in the real world, such as 3D objects or animations, can put children inside the magic circle and fostering an illusion of being inside the game world, where they will concentrate and engage more at a constant level [24]. Secondly, AR integrates both the sight of virtual objects and the feeling of physical objects, so children can view the previously static images from different perspectives and interact with the virtual content and physical objects more naturally and directly [2, 24]. In addition, one important feature of AR is that it emphasizes the contextual relationships between real and virtual objects [2], offerings meaningful and rich information to help construct an elaborate network of learning content [24]. Last but not least, AR technology can facilitate collaborative learning among children, allowing them to collaborate with classmates, receive support from teachers, and communicate with their parents [7, 16].

1.3 Self-Determination Theory

Self-Determination Theory (SDT) explains why people are motivated to engage and put effort in an activity for pleasure [18]. SDT has been applied to educational research and can improve children’s interests in learning and their confidence in their own abilities [18]. Due to the difficulty of mathematics and the higher effort demand required from children, it requires a strong degree of motivation, positive attitudes, and interest towards mathematics to achieve high performance [14]. It has been suggested that the potential of games to satisfy basic psychologic needs for competence, autonomy, and relatedness can lead to increased enjoyment [18, 20, 23, 25], desire for future play [18, 20, 23], recommendation to others [18, 20], and more positive ratings of the game [18]. Therefore, we are designing a textbook-based AR learning game for primary school children, called See Me Roar [16], which aims to provide children a motivating learning environment in doing mathematics exercises. Hypothesis 1 is proposed:

  • H1: See Me Roar will improve children’s (a) enjoyment, (b) desire to do the exercise in free time, (c) recommendation to others, (d) perceived fun of doing math, (e) likability of the experience over a paper exercise.

2 Schoolwork in Different Cultural Background

Cultural issues are important and complex in the design of AR games for learning, especially for mathematics schoolwork in primary school. Children often complain about schoolwork taking away their time for more enjoyable activities [26]. The learning environment is different between different cultures. In countries like China, students get used to having a lot of homework after school. According to a report [4], Chinese students from primary and secondary school spend three hours on average on homework every day, which is three times as much time or even more compared with their counterparts in other countries [4]. What’s more, mathematics is also considered as the most difficult subject by students, with 71.9% stating that they spend the most of time on mathematics homework [4]. The overwhelming homework can make students feel frustrated and stressed, resulting in the negative attitudes towards homework as well as the learning experience [4]. In addition, in the home environment in China, parents are highly involved and controlling in children’s schoolwork. Parents are asked by teachers to supervise their children in finishing their homework. According to the same report [4], over 80% of the parents feel exhausted from the homework of their children. While children from Western cultures spend fewer hours in school and devote less time after school to academic activities compared to Chinese children [9]. Hypothesis 2 is proposed:

  • H2: There are measurable differences in perceiving See Me Roar in (a) enjoyment, (b) desire to do the exercise in free time, (c) recommendation to others, (d) perceived fun of doing math, (e) likability of the experience over a paper exercise between children in China and children in the Netherlands.

3 Concept Design of See Me Roar

Based on SDT, we are designing an AR game for primary school students called See Me Roar. The current version of See Me Roar is the base game with basic functions, aiming to provide children a motivating learning environment in doing their math exercises. The game concepts were designed and developed together with two Dutch primary school students.

In the beginning of the gameplay, children are told that there are animals in their textbook waiting for their help to solve math problems. Then, children start to scan the textbook and find animals. When the animal shows up (Fig. 1 up-left), children can interact with the animal by touch-input, leading to a number of different actions, such as lying down, jumping, or flying. Children can control animals to move around (round button in Fig. 1). A relationship bar with the animal shows up on the right corner of the screen (See Fig. 1), starting from 0 point. Children have to find ways to build a relationship with the animal. They can open their bag that contains some food for the animals (Fig. 1 up-middle). For each food item there is a description of animal preferences (Fig. 1 up-right). Children can feed the animals based on their own choice. Once the relationship bar achieves 100 points, an exercise interface will appear and children can write their answer to the displayed exercise (Fig. 1 down-left) (the exercises match their learning progress in their textbook). Upon completion, children will get immediate feedback showing right or wrong answered questions accompanied by either a gift as reward from the animal (Fig. 1 down-middle), or an encouraging message for them to keep on going (Fig. 1 down-right). Different animals carry exercises with different difficulty levels based on the rarity of encountering them.

Fig. 1.
figure 1

Screenshot of See Me Roar (up-left: animal shows up; up-middle: interact with animal; up-right: description of food; down-left: exercise interface; down-middle: reward; down-right: encouraging message).

Full size image

4 Method

4.1 Participants

Two user studies have been done, including 38 children in total from China and an international school in the Netherlands.

China.

20 Chinese participants (10 Males and 10 Females; M = 8.2 years, SD = 0.62 years) were randomly selected from grade 3 of an average-level primary school. 3 out of the participants reported having used AR before. The most popular games was Minecraft, with 13 participants naming this as their most played game.

The Netherlands.

In the Netherlands, 18 participants (10 Males and 8 Females; M = 7.1 years, SD = 0.32 years) took part in the user study. They were students from one class in grade 3 in the international school who can speak English in the Netherlands. Among them, 3 out of 18 participants claimed that they have experience with AR technology before. Like the Chinese participants, 11 participants in the Netherlands said that Minecraft was their most played game.

4.2 Apparatus

The mobile devices used in the study were Galaxy S8 with the Android operating system. We used Unity 3D as the game engine to build the game, with the Vuforia plugin for AR features. The current 3D models of the animals and food items were purchased from the Unity Asset Store.

4.3 Procedure

China.

With the help of the teachers, we randomly assigned the 20 participants into two equal groups (Group A and B). We used a within-subject design for the study, where each group experienced the AR game and the paper exercise in different orders: group A played the game first and did the paper exercise, group B did the paper exercise first and then played the game. All participants individually performed 10 mathematics exercises each time with roughly the same difficulty level, on paper or AR game and vice versa. The exercises were chosen and modified from the math textbook of grade 3 by the teacher. The paper exercises contained the same animals and assignments as the AR game, so that purely the interactive AR aspects were tested instead of the fantasy narrative of anthropomorphic animals. Participants were told that there was no time limit and they could finish the exercises at their own speed. After the paper and the AR game, participants were asked to complete a questionnaire independently. At the end of the study, participants were interviewed regarding their preference between the paper exercise and the AR game.

The Netherlands.

Due to time constraints, the 18 participants were randomly divided into two groups in the study in the international school in the Netherlands (Group C and D). Participants did the experiment in groups of 9. Same as the user study in China, each group experienced the AR game and the paper exercise in different orders. The paper exercises featured the same animals and exercises with roughly the same difficulty level as the AR game. After both paper and the AR game, participants were asked to complete the questionnaire. An extra PENS questionnaire [23] was filled in by children after playing the AR game. In the end of the study, participants were interviewed in group with questions regarding their preference between the paper and the AR game, other possibilities in the game, other types of animals in the game, and the difficulties of the exercise in the game.

4.4 Measurements

The experiment followed a within-subjects design with counter-balancing to avoid carry-over effects. Enjoyment was measured adapting the Intrinsic Motivation Inventory [22], assessing the participants’ enjoyment while experiencing the AR game and the paper exercise. The questions for assessing the desire to do the exercise in free time were adapted from [18, 22], including “Given the chance I would do this activity in my free time”. The recommendation to others was assessed by “I would recommend this experience to my friends” [18]. Self-made questions were developed to measure the likability of the AR game and the paper exercise, and to what extent did the game or paper exercise make math more fun, using the statement, “I like playing this game” or “I like doing this exercise”, “This game makes math more fun” or “This exercise makes math more fun”. The Smileyometer designed for children was used to elicit children’s opinion on the AR game and the paper exercise, which is a 5-point Likert scale and uses 5 smileys [21]. The answers of Smileyometer were re-coded to 1 (strongly disagree) to 5 (strongly agree). The PENS questionnaire [23] was used for reflecting the perceived autonomy, competence, and relatedness when playing the AR game. A 7-point Likert scale was used (1 = strongly disagree, 7 = strongly agree). An open-ended interview was conducted after finishing all the exercises and questionnaires, aiming to collect more in-depth feedback and suggestions from children for the future development of the AR game.

5 Results

5.1 Mathematics Performance Test

A paired sample t-test was conducted to examine the final scores for the AR game and the paper exercise in China. There was no significant difference in the scores of the paper exercises (M = 8.40, SD = 1.603) and the game exercises (M = 8.25, SD = 1.585); t (19) = −0.429, p = 0.673. The result shows that the AR game does not have negative influence on children’s performance in doing mathematics exercise.

In the Netherlands, participants performed the study in groups of 9. Therefore, we were unable to record the scores of each participant in the AR game. Thus, we didn’t compare the performance between the AR game and the paper exercise for the international students.

5.2 Motivation Test

China.

When we compare the experience of the AR game with that of the paper exercises, significant differences were found in their likability of the experience (AR game: M = 4.6, SD = 0.598; paper: M = 4.1, SD = 0.788; t (19) = 3.249, p = 0.004), desire to do it in free time (AR game: M = 4.45, SD = 0.686; paper: M = 3.6, SD = 1.142; t (19) = 3.101, p = 0.006), perceived fun of doing math (AR game: M = 4.55, SD = 0.135; paper: M = 4.00 SD = 0.192; t (19) = 2.979, p = 0.008), recommendation of the experience to others (AR game: M = 4.55, SD = 0.686; paper: M = 4.1, SD = 0.912; t (19) = 2.651, p = 0.016), and enjoyment (AR game: M = 4.51, SD = 0.798; paper: M = 4.03, SD = 0.593; t (19) = 4.174, p = 0.001).

The Netherlands.

Whereas the children in China already evaluated the AR game very positively, the international school children in the Netherlands rated it even higher, leading to a strong negative skew and ceiling effect for many of the motivational correlates of the AR game (likability: M = 5.0, SD = 0; desire to do the exercise in free time: M = 4.67, SD = 0.97, skewness = −3.58; perceived fun of doing math: M = 4.78, SD = 0.94, skweness = −4.24; recommendation to others: M = 4.72, SD = 0.96, skewness = −3.89; enjoyment: M = 4.71, SD = 0.51, skewness = −1.82). Therefore, we decided to perform Wilcoxon signed rank tests.

The professed desire to continue playing the AR math exercises in the free time was significantly higher than the desire to continue doing the paper exercises (resp. M = 4.67, SD = 0.97 vs. M = 4.18, SD = 1.33; Z = -2.03, p = 0.042). After playing the AR game, the students were also more inclined to recommend it to others than the paper exercises (resp. M = 4.72, SD = 0.96 vs. M = 4.18, SD = 1.24; Z = −2.41, p = 0.016). All other tests n.s.

The Interview Results.

The interview results provided more positive and in-depth feedback for playing See Me Roar. We first asked children about their preference between See Me Roar and the paper exercise. In China, 18 out of 20 participants reported that they preferred See Me Roar more than the paper exercise, as the typical positive comments obtained by the participants reporting See Me Roar as more realistic, fun, and vivid. In the international school in the Netherlands, 17 participants out of 18 said that they preferred See Me Roar more than the paper exercise, especially the animated 3D animals were appreciated, as indicated by children that See Me Roar was “more fun and cool to play”, “giving opportunities to learn while playing”, “offering different options and multiple interesting stuffs to do”, “making it possible to see 3D animals which look real or are hardly to see in real life”, and “making learning more fun”. Conversely, one Chinese participant expressed negative feeling on the scanning of AR animals and another stated that she found no difference between the AR game and the paper exercise. One participant from the international school in the Netherlands complained about the difficulty to find out the AR animals in the game.

5.3 Cultural Differences

Regarding the likability of the experience, desire to do the exercise in free time, perceived fun of doing math, recommendation to others, and enjoyment, there was only one significant difference between the two cultures: the international children significantly like the See Me Roar more than Chinese children (F (1, 35) = 9.108, p = 0.005). See Fig. 2. There were also no interaction effects between culture and the likability of the game compared with the paper.

Fig. 2.
figure 2

Comparison on the likability of the experience of the AR game and the paper exercise between the Chinese and the international children.

Full size image

5.4 PENS Questionnaire

The result of the PENS questionnaire was considered as unreliable in this study since most participants (15 out of 17) chose strongly agree for each statement using the 7-point Likert scale, including negatively coded statements.

Although participants rated the game highly in the PENS questionnaire, the interview results revealed deeper insights. When asked about other possibilities in the game, participants expressed their different needs for the game story and game control, such as the movement of the animals, reactions of the animals (such as sound), and let animals have babies. They were also looking for more types of the animals in the game, including the sea creatures, ancient animals they have never seen before, wild animals, and fantasy animals such as unicorns. Participants reported different preferences regarding the difficulties of the math exercises, some expressed that they would like harder exercises in order to practice their skills to learn better and faster, and to feel more challenging. Conversely, some participants would like to start with easy exercises because they were not willing to deal with tricky exercises. It was also observed that during the gameplay, participants shared their screens and communicated with others a lot.

6 Discussion

6.1 Discussion of Results

From the result we can see, H1 is confirmed: that compared to the paper exercise, See Me Roar increased the likability of the experience, enjoyment, desire to do the exercises in their free time, recommendation to others, and perceived fun of doing math. While children from the international school in the Netherlands only significantly increased the desire to do the exercises in their free time and the recommendation of the experience to others by performing See Me Roar than paper exercise. Regarding to H2, no significant difference exists between Chinese children and the international children in the Netherlands, except for the likability of the game and the paper, where international children significantly liked See Me Roar more than the Chinese children. No significant difference was found in the number of items that were answered correctly between the game and the paper exercises.

The interview results provided interesting feedback for the study. Children were attracted by the AR animals and the rich interactions within the game. Feeding and helping animals while doing mathematics helped to immerse themselves into the game world and improved their learning process. Children also provided various ideas related to other possible options for the AR game related to the psychological needs in SDT, namely autonomy, competence, and relatedness. More types of animals, richer reactions from the animals, and different controls of the game were all expected by the children. In addition, the difficulty levels of the exercises were different based on children’s own abilities and skills. During the gameplay, it was observed that children tended to share their experience and help each other to play the game, while they also compared with each other in getting rewards and with the finishing speed.

6.2 Limitation and Future Research

Limitations remain in this study. First of all, the study procedure was introduced by the teacher in both China and the Netherlands, which might influence the choice of the children. Secondly, due to the condition limitations in the Netherlands, the exercise score was not recorded, and children were doing the study in groups, which could lead to different results compared to the individual study in China. Thirdly, the Smileyometer used in the study was designed for children aged from 10. Younger children around 7 to 8 years old tended to choose the most positive score.

What was especially noteworthy and came as a surprise to us, is just how much the students liked the AR game. So much in fact that it makes us a bit incredulous as to the veracity of the results. To our estimation, the game is barebones and lacks a lot of en- gaging game mechanics and design features. It is not yet designed to really stimulate competence, autonomy and relatedness, and the learning content is not well integrated with the game mechanics. For all intents and purposes, it should score worse than many other serious games which fail to be motivating [31]. It’s tempting to think the design of AR animals walking over one’s textbook is indeed by itself incredibly motivating for children of primary school age. However, it is also likely that this statistic is at least partially influenced by both novelty and Hawthorne effects. Regarding the PENS questionnaire, even though a literature search indicated that Likert-scales and the Smileyometer were suitable for children, we noticed a large number of children rating both positive and negatively worded statements with “fully agree”. This means that in their enthusiasm or desire to please the experimenter they did not read all the questions correctly. For the future, both quantitative and qualitative measures that tease out more useful or constructive critical reflections should be devised.

This study is the first step in our research, proving the positive motivating effect of the working prototype of the AR game for children from different cultures. Our future work will focus on the design of more specific features in the AR game based on SDT and game mechanics, developing different game features and measuring how these game design decisions influence motivation with the game.

7 Conclusion

To conclude, the presented study suggests that See Me Roar significantly improves the learning experience of children. The results of the study indicate that in general, See Me Roar received very good evaluations for enjoyment, desire to do in free time, perceived fun of doing math, likability, as well as recommendation to others. The game version achieved significantly higher ratings on these subjects by the participants over the paper version. It could be used to help children to do mathematics schoolwork in a more playful and fun way. The study indicates that an AR game with animals walking over ones’ textbook is globally accepted by both children from the Eastern and Western cultures. In the future, we will develop the game based on SDT, modifying the based prototype to include game mechanics to stimulate feelings of autonomy, competence, and relatedness. Through the design and implementation process, we will seek to chart the design space of AR games for learning, investigate the magic circle in AR settings and tease out the effects of game mechanics related to SDT on stimulating motivation and learning performance.