Keywords

1 Introduction

The augmented reality (AR) domain has shown widespread growth in recent decades. This is not surprising given the scale of use of advanced mobile devices containing cameras that record reality and add different types of metadata (2D images, 3D images, text, audio or video) in real time. The application of AR can be useful in several areas: for example, in the field of medicine this technology could allow the surgeon to have a more detailed, real-time and 3D view of the organs when operating, which could also be an effective way to improve diagnosis.

As we show, the use of such technology is paramount in many areas. Indeed, we increasingly need to align knowledge and create a new reality in which connectivity, interactions and visualization of all activities are not centralized but distributed, collaborative and interactive.

The field of education can take advantage of this scientific progress to improve educational process. Nowadays, the use of computer tools to support education, known as e-learning, optimizes teaching and learning processes. According to [1], e-learning must go beyond platforms such as Learning Management Systems (LMS) to allow the students to work collaboratively using computer resources. In addition, AR could benefit e-learning processes through an intuitive way of interacting and collaborating between real and augmented objects.

According to [2], including AR technologies in educational settings improves learner outcomes, allows pedagogical contributions and allows more interaction. It can enhance learning achievement and motivation; improve satisfaction, confidence, comprehension and attitude; and decease cognitive load. It can provide the following interaction opportunities: student-student, student-material and student-teacher. Finally, AR can enable the visualization of invisible and abstract concepts and events. We propose a promising AR framework for e-learning called AugmentedBook.

One of the main ideas in our AugmentedBook framework is to provide collaborative and illustrative media (i.e. text, video, image) related to a target in the educational support system. This framework allow the students and teachers to work together for rich educational support. To the best of our knowledge, there is no previous work in AR applied in the e-learning domain that allows collaborative work on the educational support itself. Our system proposes to add various explanatory multimedia on any target without requiring a plug-in.

The rest of this paper is organized as follows: the literature review is described more in depth in Sect. 2, the AugmentedBook architecture is detailed in Sect. 3, experimental setup and results are presented in Sect. 4, and in Sect. 5 we conclude with a summary and perspective of our work.

2 Literature Review

Thanks to the development of mobile devices, AR technology has responded quickly. In fact, nowadays, mobile devices integrate all the functions necessary to launch AR applications, such as a broadband internet connection, excellent computing power, cameras and sensors such as GPS, accelerometer and fingerprint reader.

As a result, the use of AR has received a lot of attention in several domains and especially in education [3]. There is no doubt about the potential of AR content to improve motivation and learning as shown in studies such as [3] and [4], but a lack of standardization delays integration into educational environments. Currently, AR-based education systems still do not offer a standard solution that groups all the features mentioned earlier, according to [2].

In the last few decades, several AR e-learning frameworks have been proposed. One of the first approaches is described in [5]. They took advantage of the first versions of Web3D technologies to introduce a web-based editor for creating 3D content and a viewer for educational purposes. Although the idea was innovative, specific plug-ins were required and it was intended for a tabletop environment. Some developments use AR for a particular educational application. For example, one study [6] proposes a simple interface for examination applications that allows user interaction with one finger. This approach limits the user to response either yes or no, and there are still no standards for integration with an LMS. A system described in [7] uses an authoring tool to develop AR applications for industrial sequential procedures. Although sequential application is a considerable improvement, it requires specific hardware and software. An AR-based mobile software that provides 3D and multimedia e-learning material for complementing books or other educational support is presented in [8].

According to [9], mobile devices offer an ideal platform for AR applications as they are very cost-effective and easy to use, especially for younger students [10]. Desktop computers can support AR applications, but they are not portable due to hardware limitations [11]. In contrast, mobile devices provide many advantages, such as portability and encouragement of high social interactivity [12]. They are also beneficial to students who engage in outdoor observations and learning activities [13]. Furthermore, location-based AR on mobile devices enables students to immerse themselves in the learning process [11] and increases their collaboration skills [14]. The 2010 Horizon Report [15] indicated that users would no longer need to be tethered to their desktop computers thanks to wireless mobile technology.

To the best of our knowledge, there is no work in the field of AR e-learning that allows the teacher and students to add class notes related to the textbook. Videos and explanatory audio can be added to the course material itself, which allows students and teachers to interact and collaborate (in the same way as social media comments) by adding notes in real time and overlaying it directly on the course material. There is no similar mobile device application available that does not require installation of a plug-in.

3 AugmentedBook Architecture

As illustrated in Fig. 1, the AugmentedBook e-learning platform allow students and teachers to add collaborative comments on any target from the educational support. The system also recommends illustrative videos to a given target. The recommended videos and audio can help students to better understand the content of the educational support.

Fig. 1.
figure 1

Overview of the AugmentedBook framework

Full size image

AugmentedBook has a distributed architecture, using Cloud and mobile devices to create AR applications. The architecture of the education platform is n-tier. In the following sections, the different tiers and layers of the AugmentedBook are explained.

3.1 The Client Tier

The mobile device constructs our client tier and the Android engine communicates with sensors like the camera and accelerometer. The user event via GUI (Graphical User Interface) are listened and the target is tracked. The tracked target can be an image or text from the educational support, and is transmitted to the AR engine via the Android engine. We adopted VuforiaFootnote 1 as the AR engine because there is no plug-in to install. The last part of client tier constructs local storage. An SQlite database is used to store the last used target, to avoid sending a recognition query to the target management system each time.

3.2 The Application Tier

The second tier in our architecture is the application tier. It includes a Data Access Object layer which is responsible for getting/setting data from the data source. The second layer is a business layer responsible for processes like recognizing the target and video retrieval. The recognition of the target (text or image) is executed by the target management system. The last layer of the application tier is the web service layer, which can contain several technologies or types of communication such as Rest, Corba and SOAP. Communication is via Restful web service, which was chosen over other kinds of web services due to its popularity, simplicity, its perfect adaptation to the client-server architecture and because communication is independent of the state of the client. Communication with the wider web services is via http/JSON. The application communicates with three different services:

  • The first service is dedicated to recognition of the image targets. For this purpose, the Target Image Recognition service of Vuforia is used which is based on a machine-learning algorithm.

  • The second service is dedicated to recognition of the text target. For this purpose, the Target Text Recognition service of Vuforia is used. For the moment, the service supports only the English language.

  • The third service takes as input the result of the second service, with the objective of returning a video that is relevant to the recognized text. The YouTube API is used for this purpose and proposes the five most relevant videos. If the user validates the relevance of the video to the selected context, it is stored in a tree structure in the Firebase database (explained in Sect. 3.4). In this way, potential viewers of the same text will see the same videos judged relevant by the user. They can likewise appreciate the video, comment or add other videos.

3.3 The Data Tier

The last tier in our AR framework is the data tier, which establishes communication with the authentication service, Firebase storage and Firebase real-time database. These services are used as follows:

  • Storage: Firebase storage is dedicated to storing and retrieving user-specific large files such as photos or videos.

  • Authentication: allows easy creation and management of various means of authentication (such as Google, Facebook or email) for easy and secure access to AugmentedBook.

  • Real-time database: data is stored as JSON and synchronized in real-time to every connected client. This allow us to rapidly receive updates with the latest data and manage concurrent users more easily.

3.4 Firebase Real-Time Tree Data Structure

The two types of collaboration (student-student and teacher-student) help to create rich educational support. To allow collaboration between different participants (teachers and students), a real-time Firebase tree structure is used to store targets as well as all the associated metadata. Indeed, as illustrated in the JSON structure, each target is associated with an identifier, a type, a value and an idOwner (which is the identifier of the person who has marked this target). In addition, the target can be associated with different multimedia documents such as video, text, audio or image. In the same way, these multimedia documents are identified by identifiers, idOwners and popularity. For each multimedia document, a response can be associated, which constitutes social interactions between participants around the multimedia document that enriches the target. The documents themselves are stored in the Firebase Storage Service.

4 Experimentation

The most common and successful method for evaluating an HCI is to carry out a user study. This technique consists of proposing a survey to the participants, who can answer each question with a score. A substantial number of questionnaires have been used and reported in the literature to assess the perceived usability of interactive systems, namely the Questionnaire on User Interface Satisfaction (QUIS) [16], the Computer System Usability Questionnaire (CSUQ) [17] and the System Usability Scale (SUS) [18, 19] Questionnaire. The QUIS focuses on specific aspects of the human-computer interaction, while the CSUQ and the SUS questionnaires revolve around interactive system usability in general. Our proposed survey to evaluate our AugmentedBook system is inspired by [17].

A user study with 25 participants was performed to evaluate our system. The participants taking part were a different profile of students (computer science, IT, literature, science, medicine) from Albaha University who were not necessary familiar with Android OS. After testing the AugmentedBook system, participants completed a form containing the following questions. Scores range from 1 to 7 where 1 represents ‘strongly disagree’ and 7 ‘strongly agree’. The first eight questions assess system usefulness. Questions 9–15 assess participants’ satisfaction with the quality of the information associated with the system. Questions 16–18 allow the participant to rate the interface quality.

Subsequently, we provide a graphical representation (in the form of a chart) of these answers. The histogram (Fig. 2) shows that the majority of scores are between 5 and 7. Based on the result of this experiment, we conclude that using our AR system does not entail any major drawbacks. Nevertheless, there is a need to improve certain points such as detecting text written in Arabic in the system.

Fig. 2.
figure 2

Average score of AugmentedBook from 25 participants from different profiles

Full size image

5 Conclusion

In this paper, we propose a new AR framework called AugmentedBook. We also collected and classified AR applications based on several criteria, and through these comparisons found that AugmentedBook is the first educational application that combines the most important features and characteristics needed by both student and teacher in the educational environment and learning process.

Nevertheless, it remains to improve several aspects in our AugmentedBook system. We plan to extend the evaluation from 25 students to at least 25 teachers and 100 students from different profiles (computer science, literature, science, medicine). We plan to integrate a multimodal search starting from the text target. For the moment, our platform supports only explanatory video suggestions to the text recognized in the educational medium. We plan also to recognize Arabic text since the educational support system in our university is in both the English and Arabic languages.