Keywords

1 Introduction

Technological advancements in software and hardware and specifically the major improvements of the performance of smart devices have diversified our everyday life. One of the areas in which these technological changes are gradually becoming apparent is that of education. Augmented Reality (AR) is expected to diversify teaching in the coming years as AR applications blend physical and virtual worlds through a touch screen, a camera, or a head-mounted device [1]. AR technology is compatible and can be used in conjunction with the Game Based Learning (GBL) methods to boost educational processes. It can also help students both in the cognitive part and in gaining motivation for learning, but also in developing basic 21st century skills [2, 3].

Traditional teaching methods do not make much use of new technologies with the result that students often lose interest in education due to the way they learn. While in their daily lives, students are accustomed to function and interact within environments with multiple stimuli, the traditional school remains a place with limited options [4]. The change to a new model is imperative to make learning more creative and interactive by introducing new visual forms of learning [5].

Through play students can develop their thinking and concentration [6] around a topic via a fun and entertaining way of achieving specific learning goals [7]. Students’ engagement helps to achieve multiple goals outside of learning, such as in-depth understanding of the topic and greater awareness of the topic [8].

Augmented reality allows information and 3D virtual objects to be displayed on top of real-world objects. Furthermore, AR offers the ability on such objects and information to be processed in real-time [9, 10]. As smart mobile devices become more powerful (computationally) they can support augmented reality applications and thus make AR reachable to more people.

In addition, software libraries and frameworks such as ARCore and ARToolkit have helped the software development community to have effortless access to AR technology and include it in their applications. Research about the interaction in such environments with bare hands are also evolving along with the improvements on the hand tracking field of study making possible, for mobile devices, to run such algorithms [11].

1.1 Game Based Learning

The educational community is shifting the goals, that had been set in previous decades, related to the acquisition of knowledge and the traditional model of teaching to a new approach. One of the main goals of this new approach is the utilization of knowledge but also the development of a pleasant environment for enhanced creativity in the classroom [4]. Discussion, research, and experimentation can help students activate their interest and make learning more interactive.

GBL creates a framework where students have additional motivation compared to the traditional teaching method. Their participation because of games is greater, while the willingness they show towards the lesson in general increases [2]. According to Fotaris et al. [12] learning using digital games in education reported significant improvements in subject understanding, diligence, and motivation. In this way the game becomes an educational tool to achieve the goals set by the teacher. In addition, as Sitzmann states [13], through play, students’ performance can be improved and relationships between them can be strengthened.

According to Prensky [14] an educational game is designed not only to develop players’ skills but also to maintain and apply this object in the real world. Thus, it is important to connect education with everyday life as in this way it will be possible to develop students’ metacognitive skills and their awareness of important issues.

1.2 Augmented Reality in Education

AR is a relatively new technology that combines the physical with the virtual world through the addition and integration of virtual information [15]. In terms of education, it is important to connect the two worlds namely the physical and the virtual as opposed to VR where the user is completely immersed in a virtual environment [16]. The use of AR enables students to diversify and enrich the physical world through new technology and the use of smart devices rather than completely replace it [9]. The additional information, images and objects provided help students to effectively understand abstract concepts, thus helping to improve knowledge and learning [17, 18].

Mobile AR is considered one of the most up-to-date technologies in education for the coming years as it is easily accessible due to the high penetration of smart devices [19]. The reasons are that it is an inexpensive technology, easy to apply in the classroom and safe. With AR, students can simultaneously interact with both the real and virtual worlds in real time [20]. This turns AR into an exciting entertainment and learning tool for kids. In addition, the new features it offers help to develop their imagination and creativity [21] and at the same time helps to connect the game with learning.

AR can be used as a learning tool in the hands of the teacher to activate students’ interest and gain their attention, so that they can focus more on the lesson. According to Oranç and Küntay [22], through AR applications, students delve deeper with this technology. In addition, they believe that to be effective there must be utilization of the knowledge and skills that students acquire in their daily lives. According to Hirsh [23], the principles that should guide the use of AR in education are essential: (a) to encourage students to be actively involved in the process, (b) to include additional material, (c) to provide important new experiences with which they can relate, and (d) to offer an environment of socialization and cooperation.

Moreover, the combination of AR with game-based learning (ARGBL) has gained momentum in the field of education as it creates a playful and enriching learning experience [24]. Utilization of this technology leads to the creation of new interactive visual learning environments [5] that allow new approaches to teaching and learning experience in various disciplines.

2 Background Work

Game Based Learning is an approach that is constantly gaining ground in the field of education and teaching due to the opportunity it gives students to learn through enjoyable and creative activities. The field of literature & language is no exception since this is one of the most frequent courses in the typical school curricula. Through the electronic or non-electronic games students can develop their vocabulary [50] both in their mother tongue and in the second foreign language [51-53].

2.1 Alternative Educational Tools for Literature and Language Education

The field of literature & language as mentioned is one of the most important in education. The pedagogical teaching methods used are many and range from traditional to modern and innovative. From creative writing and the use of everyday objects to the use of technology with computers, smart devices, applications, games, VR and AR, educational robotics.

Creative writing is a method that utilizes various techniques to spark students’ imagination and help them create their own texts using language in a different way [49]. However, in addition to the classical methods, the use of technology is a new way to encourage students to improve their language and vocabulary skills [28]. Mobile games [54, 55] are increasingly being used in classrooms, presenting positive results in the areas of knowledge and skills [26, 29]. In fact, studies have shown that mobile games are an effective tool for both high school and younger ages as students developed their knowledge in repeating/reviewing, using multiple modalities and means, and having control over their own learning [30].

According to Hwang, W. et al. [31] game-based learning activities can significantly improve students’ speaking skills if they work in combination with a mobile game-based learning system. The WhatsApp Social Networking Tool (WSNT) seems to work for older ages [25] as it helps students learn English interactively and collaboratively. Furthermore, Ghazisaedy [27] argues that educational robotics can help in learning English as a foreign language as the new way of learning helped them to develop additional motivation and to learn more effectively in the long run.

In their research, Cai, S. et al. [17], on 38 students of the eighth grade of an AR and motion-sensing learning technology that teaches magnetic fields in a junior high school physics course observed the effects of using natural interaction on students can improve students’ learning attitude and learning outcome.

2.2 AR in Literature and Language Education

A significant number of AR applications, related to education, have use cases from language & literature education as shown by Parmaxi’s et.al. [41] systematic review of 54 studies regarding language learning between the years 2014–2019. According the review, most AR studies dealt with vocabulary (23.9%), reading (12.7%), speaking (9.9%) writing (8.5%) or generic language skills (9.9%). However, the authors point out that no special connection was found with learning theories during the application of AR. This gap can be filled through the creation of applications in collaboration with stakeholders in the educational community.

AR-related language applications and research have shown positive results in terms of vocabulary and spelling development [4], improving student motivation and increasing memory. Similar results appeared in a study on the improvement of vocabulary and grammatical structures [42]. Also, positive results were found regarding the learning of English as a second foreign language through the application of AR [43] in relation to the traditional teaching methods.

The use of AR seems to help students significantly in the complex and demanding field of language as they seem to achieve higher performance, to retrieve knowledge from memory more easily [44], and also to improve their narrative ability, the size of their stories as well as their creativity and imagination [45]. Another important fact is that in most researches it seems that, at the same time with the cognitive objects increases the motivation, social collaboration, and interest of the students for the lesson [38, 46].

Collectively, we have researched more than 30 studies and the conclusions can be summarized to: (a) naturalistic approaches applied to AR technology help to foster positive attitudes towards AR, (b) AR facilitate collaboration and (c) AR-related language applications enhance the users’ social collaboration, personal development and skills.

2.3 Hand Interaction with AR

Research on hand interaction with the use of AR suggests that one should use the hands like when interacting with the musical instrument called Theremin. In this way the errors are minimized when hands block or confuse. A gesture-based interaction has been developed which includes functionalities as menu selection, object manipulation and more [33]. A method for realistic grasp is proposed that uses predefined rules for the movements of the hands as a physics simulation would be computationally heavy. The system checks if an object is considered grabbed and if two or more points of the hand are interacting with the object from opposite sides [34]. A similar method to the above is proposed but with the use of a glove that will provide haptic feedback as well [35]. A system to control the tv functionality is developed using Convolutional neural networks and Convolutional pose machines for hand recognition [36]. Another system for interaction with 3D objects in museum is proposed using the Leap Motion device for the hand recognition [37]. Using the Leap Motion device, researchers developed an application for learning geometry, that students can draw 2D and 3D shapes of geometrical shapes on top of a marker [32].

Objectives

The purpose of our study is to raise awareness and develop students’ vocabulary and creativity on the topics of recycling and Covid 19. The research utilized the approach of game-based learning and AR hand interaction technology. Our goals were for students to:

  • Develop their vocabulary around the topics of recycling and Covid 19.

  • Utilize AR hand interacting technology to become more active in class (motivation).

  • Develop concentration, engagement, and enjoyment using an AR game.

  • Develop their creativity by making a story of their own based on the objects they found.

  • Be aware of recycling and Covid issues 19.

3 Field Study Methodology

Our field study was carried out at the Elementary School of Plakia, Crete, Greece, in October 2020. While further expansion of the research was originally planned, the field study was stopped due to the mandatory quarantine in respect with covid-19. The research sample was homogeneous in language and consisted of 36 students (16 boys, 20 girls). The age of the students was 10 years old. The requirements regarding information, consent, confidentiality and use of data were met carefully, both orally and in writing, by informing school staff, students, and parents about the purposes of the study and their right to deny their participation. The students were divided into two groups based on their classroom.

We also obtained the approval of the Primary School Directorate to conduct this study in schools in its area. Then, in respect to the protocol of the institutional review committee, we contacted the school principals to describe the study and ask for permission to meet with primary school teachers, to explain the study and determine their interest in participating.

The study was conducted in two levels, the first concerned the teachers and the viewpoint they had about the lesson and the second had to do with the students. In this way, the aim is for the study to capture a more comprehensive opinion of the process and how to use an AR game in the learning practice. For the teachers, the semi-structured interview [47] was used as a tool while for the students, the observation method [48] was used. The interviews were conducted shortly after the end of the process, so that the conclusions and thoughts of the teachers are fresh.

4 Our AR Hangman Game

In this section of our paper we present a game that we have implemented in Unity3D Game Engine using (AR foundation and ARCore) a custom tool built to facilitate the process of game development that requires interaction with physical hands in augmented reality environments.

4.1 Constructing the Game

The design of our application is separated in two main modules (see Fig. 1), the first module, namely the hand tracking module is a service that is implemented for hand recognition and tracking and the second module, namely the Unity toolset, constitutes the functionality and representation of the hand in the game. The separation in two different modules allows for higher frame rates of the game as the device running the game is relieved from the hand recognition process.

The main components of our application, in a high architectural level, are presented in Fig. 1 while Fig. 2 shows the usage of these components described in a workflow diagram of a common usage scenario created with our application.

The Unity toolset provide basic components such as a socket client for the communication with the hand tracking service and an image capture method that is required to send a stream of images on the service. The rest components concern the game creation and interaction process. These components are a virtual hand, hand actions, gestures, selectors, and raycasters (see Fig. 1 right part & see Fig. 2(a)).

To better use the hand points, we have separated them on their corresponding fingers, so when there is a need to retrieve specific hand points, they can be accessed by calling the involved fingers. Predefined gestures also exist that can be used as is without the need of accessing the finger classes or the hand points. To add on this and make it even easier and simpler to use a gesture type enumerated value is assigned to each gesture that is implemented and a gesture manager class has been created. Through the gesture manager class all gestures can be accessed by using the interrelated gesture type name.

The HandAction class (see Fig. 2(a)), is responsible for defining the behavior of the virtual hand when interaction occurs with another virtual object in the application. Such actions involve the movement of a virtual object. This class exists mostly to cover additional behaviors or different implementations of already existing behaviors.

Selection Interfaces (see Fig. 2(a)) include Ray cast provider that is a component that describes the type of a ray caster to be used. This ray caster is used on the selector classes that their responsibility is to check if the hand is about to interact with a virtual object or not. Finally, in this category of components the selection action classes are also included, which are used to visually inform that a virtual object is about to or interacting with the hand.

The hand is represented virtually within the visualHand class which is wrapped in the actionHand class. In the actionHand class further information about the hand is given such as the method to detect possible interactions.

Fig. 1.
figure 1

Architecture diagram

Full size image

The service which connects the Image process module and Socket Server (see Fig. 1), is responsible for receiving a stream of images from the device and processing the image to a form that is compatible with the hand recognition algorithm. The hand recognition algorithm can be changed easily as the rest of the service is not bound to a specific algorithm. This allows each developer to use their desired algorithms. For this project we use a modified version of for the handtracking [39].

The game was implemented following the Unity toolset flow Diagram (see while Fig. 2(b)). When the game starts it initializes a connection to the service and the virtual hand. After the connection is established it starts to update the position of the virtual hand to the corresponding position of the real hand on the screen. Each frame we check if a “grab” gesture is detected that is defined by the ray cast hits of the thumb and one more finger at least to a virtual object. If the gesture is detected, then a “performAction” method is fired that starts to move the grabbed object with the hand. To place the grabbed object to the user desired position, the application calculates the distance between the target position and the grabbed objects’ relative position. To help the player to put the object easier we have set a “wider” area for the desired position.

Fig. 2.
figure 2

Component and flow diagrams

Full size image

The game has taken advantage of ARCore augmented images to link the 2D images that get acquired from the first part to their respective 3D objects.

4.2 Playing the Game: The Case of the Word Hangman

The game aims to (a) raise students’ awareness about recycling and Covid 19, (b) enable students to develop their vocabulary in relation to the thematic areas and (c) develop creativity skills by making their own virtual picture stories and telling their own stories as stated previously at Sect. 2 where the objectives of our work were defined.

The game involves two main tasks, the first task is to support language learning and the second is about enhancing creativity. In specific, the first task is based on the classic word game of hangman and its goal is for the student to complete five words correctly to progress to the second task of the game which is to present a short story involving the previously completed five words.

The first task of the game has 5 levels, each of increasing difficulty. Each level contains a word that the student needs to fill and a set of letters that represents possible matches to the word’s letters that needs to be completed. Additionally, to make it more challenging the student has five “lives” for each word. The given word is presented with the first letter on spot and the rest letters are presented by empty dashes. The student must grab a letter from the given set and place it on an empty dash. On a correct move the letter will be fit on top of the dash and stay there until the word is complete. On a wrong move the letter will change color as an indication of a wrong placement and after that it will disappear, and one player life will be lost.

For each level a pool of words has been created (see Table 1) for an indicative selection of words per game level. The selection of the word on each level is done randomly by the game. As levels advance the number of letters per word increases making it harder for the student to complete the word.

Each time a word is completed the student receives a card representing this word, then the game continues to the next level where the student must find a new word. The game completes when the student wins all levels or when the student ends the game earlier.

For every word the student has 5 wrong guesses. To help the student guess, if two wrong guesses are made, a picture representing the word is shown as help. If the student cannot find the word and loses all the guesses, then a “retry level” option is presented offering a restart of the level again with a new random word. When this option is selected the letters to choose from will be more straight forward as extra help to the student.

This help is given to the student as motivation and encouragement to not give up or have an unpleasant experience. The goal of the game is for the student to complete all the levels without skipping any. Also, it is important to increase students’ confidence and thus not lose interest on the game and complete all levels. The completion of the levels is also important for the next part of the game as it allows students to build on the imagination and storytelling when more words are involved.

Table 1. Sample words per game level.
Full size table

As stated, before a word pool has been created for the needs of the game and an indicative set of words is presented on the above table. The difficulty of each level is based on the number of letters in each word. As the levels progress the student has not only more compound words to find but also has to think and understand better the thematic areas of the game.

In respect with the game-flow, players first choose the thematic area and then the game begins by presenting as many dashes as necessary for the word of the first level. Players use their hand to grasp and move letters that appear on the screen (Fig. 3(a and b)). For each incorrect letter placement, the player loses one live. Player lives are denoted with red hart images.

Fig. 3.
figure 3

Virtual Hand Moving Letters

Full size image

When all letters are placed in the correct position a 2D object representing the word is displayed (Fig. 4). Consequently, in the classroom the teacher gives the student a paper-card corresponding to the word completed in the AR game (Fig. 5). In this way the game materializes Blended learning as students play in both virtual and real worlds.

Fig. 4.
figure 4

Completed word and 2D word object

Full size image

Through storytelling students have the opportunity to develop both their vocabulary and their imagination. They are able to consider new ideas and express them in words.

Once the last game level has been reached and the students have collected all their paper cards, representing the artefacts that describe the words found, they can place them on their desk in any order they want and create their picture story (Fig. 6). Once the paper-cards are in the desired sequence, students can place the mobile device that they use to play the game, e.g. a smart phone or a tablet, on top of the paper-cards to generate the 3D virtual objects (Fig. 6). The story can be presented to and shared with their classmates.

Fig. 5.
figure 5

Paper-card icons

Full size image

As explained at the Field study Methodology section above our students have been seperated into two groups according to their classroom. The students of the first classroom had at their disposal the teaching tools they use in their daily student life such as the blackboard and the projector. These students played the traditional hangman game and were then asked to write a story related to these words using as many as they could.

Fig. 6.
figure 6

AR based story

Full size image

The students of the second classroom were given smartphones with the application. During the use of the application they were given additional image paper cards after the successful completion of a word to place them anywhere in the room they wanted. Respectively, these students were asked to create a story. At the end of the process there was a general awareness of students about the issues of recycling and Covid 19.

5 Discussion - Research Outcomes - Results

Research has shown that using GBL in conjunction with AR can help elementary school students develop their vocabulary, become more active in class, and become more aware of important areas such as the areas recycling and Covid 19 that were investigated in our research. Both teachers and students converged in this direction and support the above statement. The table below (Table 2) shows the questions by topic and some typical answers of teachers.

Table 2. Questions & typical answers by topic.
Full size table

5.1 Field Study Results’ Analysis

The following results emerged through participatory observation and interviews with teachers. The students had a great interest and participation in the exploitation of the AR game. It piqued their interest as a new and different teaching tool through which they can learn by playing. At the same time, the teachers were positive with the use of this technology as a teaching tool. They believe that with game involvement teaching goals can be achieved as a result of a pleasant experience. More specifically, the results per thematic unit were the following:

In the field of vocabulary, no significant differences were observed between the two classes. Both in the class that the hangman game was played in the traditional way and in the second class, students developed their vocabulary and their ability to spell. The difference observed between the two classes has to do with the way the students behave. In the first case the students had a similar behavior in relation to the daily lessons. However, in the second case, they considered that they were participating in a game and not that they were taking part in a school lesson.

Significant differences were observed in the part where they used the cards with the QR codes and created their own stories. Students who attended the lesson in the traditional way did not have many ideas as opposed to those who used AR game technology. The students became more active from the moment the teacher announced that they would receive a card when they found a word and seemed to be more enthusiastic. They liked to place the cards with the QR codes in different parts of the classroom, “decorating” it with their own style. In this way they started to have more ideas and to think differently about the stories that the teacher asked them to make. The collaboration between the students increased and they started discussing and proposing thoughts and ideas to each other.

In terms of awareness, in both cases an increase was observed with a slight difference in the part using the AR game. Students in both classes learned about the possibilities offered by recycling and how they can protect themselves and those around them from Covid 19. After the end of the lesson, most of them started coming up with ideas for ways that could improve the spirit of recycling both inside and outside the school. This showed that through various activities within the school, students can become more aware and acquire new habits in their daily lives. In addition, their activities helped them gain a better knowledge and argument to propose changes in the local community. An example is that the students suggested a meeting with the local mayor to raise their concerns and suggest ideas about the recycling framework. This showed that there was a connection between the school and the local community. About Covid 19 the activities helped the students to better understand the dangers and to change some of their daily habits. In addition, they learned to use disinfectants properly and to be vigilant so as not to be exposed to the virus.

5.2 Conclusions

The use of new technologies can help significantly in the learning process if the tools used to fulfill the goals are set by teachers. Using Augmented Reality games as an educational tool can help students in a variety of ways. Students become more active throughout the lesson and are more focused for longer. This helps them to gain more benefits from the course than traditional teaching. In addition, the new way of learning and the different stimuli help them to develop their thinking and their creativity. This was evident in the present study as the stories they created had great diversity, strong signs of imagination and new ideas that these students had not used in the past. This had showed that it is not only the application that matters but the general educational approach that can highlight the potential of technology and is in line with Juhee’s findings in paper “Problem-based gaming via an augmented reality mobile game and a printed game in foreign language education” [40].

Using our AR game and the activities that followed, the students got to know better and in depth the areas of recycling and Covid 19. The AR game did not make much difference in terms of improving spelling, but students were more motivated for the lesson and were more positive in continuing until the game was over. Finally, addressing these two important issues of daily life helped students become more aware of recycling and Covid 19, take initiatives and develop active social action.

6 Future Work

The research was carried out during the first semester in only two classes. According to our original schedule, the goal was to expand to more students so that they could get safer and clearer results regarding the relationship between an AR game and the classic learning process. However, the field study was stopped due to the mandatory quarantine that last three months and up to this day. The goal for the future is to continue and add new modules to our AR game that can provide additional information such as the xAPI library which can create data for learning analytics. Through XAPI, the AR game will be able to note players’ actions in more detail, as well as the points that made it difficult or aroused their interest to a greater extent. Another point that can be explored in the future is the connection of AR games with the STEM teaching method. In other words, students should combine their ideas with virtual models and then apply them in practice.