Keywords

1 Introduction

Universities are applying technological tools for teaching, such as video lectures, interactive projects, and other supportive tools. However, universities mainly apply traditional teaching methods, and new alternative methods are rarely introduced. Furthermore, the difficult times of the global pandemic in 2020 have shown that having good technological tools in education is more important than ever.

The need for competence in artificial intelligence (AI) has increased at a high rate around the world in recent years. That is due to the discoveries of new successful applications of AI and advancements in hardware and cloud solutions. In January 2020, Norway released a national strategy to be at the forefront of AI education, research, and innovation [17]. With the large need for AI competence, universities worldwide need to focus on educating students on the topic. Companies also need to put their employees through lifelong learning programs. AI is currently taught through traditional methods like lectures, assignments, and hands-on projects. Students can also learn about AI through interactive projects like Google’s Machine Learning crash course [7].

Since the release of the consumer virtual reality (VR) headsets in 2016, VR has shown potential in multiple sectors, including education. According to Gartner, VR was considered an emerging technology until 2017. In 2018, Gartner stated that the technology had reached a mature stage, due to the variety of successful use-cases [5]. Even though successful ways of using VR have been discovered, VR has not yet had a mainstream breakthrough. Newer technological advancements make the technology more promising than ever. In May 2019, the Oculus Quest was released. The device proved that a room-scale and fully standalone VR headset could provide a highly immersive experience, without the need of an expensive PC and a cumbersome setup with cables and tracking sensors. The device shows potential in education since it can be brought anywhere and is easily set up for multiple students.

Studies have shown that the feeling of presence in VR can increase the users’ ability to recall information, compared to using a monitor [11]. With the emergence of educational escape rooms, and its successful use in programming education [14], this concept was thought to have potential in VR as well. The goal of this project has been to investigate how VR can be used as a tool for learning in AI education. As a topic within AI, we have tried to discover if VR can be used to give an engaging introduction to deep learning and to see what students think about using such an application.

2 Background and Related Work

Through the literature study, we did not find any VR applications used in AI education and little research about using VR in computer science education. To understand how an application could best be applied in an AI course, we studied VR applications used in STEM-courses to understand what makes these applications beneficial for learning. A study from 2016, defined some main aspects of what makes VR beneficial in education [15]. Their key points were that VR enhances the learning experience through studying 3D models, which further increases their motivation and engagement. In VR, they can interact, manipulate, and get immediate feedback, which can improve the learning outcome and experience. The new level of presence caused by being fully immersed in a virtual world that tracks the user’s movement and gives physical feedback through haptics shows the potential of creating experiences that would otherwise be impossible.

Game-based learning and gamification of education have also shown positive results for increasing engagement, motivation, performance, and learning outcome [8, 10, 21]. Effective educational games take advantage of the user’s ability to interact with the game to solve problems and reach goals. They also let the student control the learning experience, make the learning experience challenging and rewarding, and stimulates the user through the audiovisual works [20].

Few projects have explored the use of VR in computer science topics. Some of the previous projects at the Norwegian University of Science and Technology explored the use of VR in algorithms and data structures [13]. The project results showed positive indications that students were interested in solving tasks in a “learning-by-doing” setting to get a hands-on experience, where they were immediately scored based on performance. The thesis concluded that one of the main advantages of the application was to provide experiences that are otherwise hard to recreate. The tool showed the potential of being used as a supplementary tool. Since few projects have explored the use of VR in computer science topics, it is unknown how applying VR in this context affects learning outcome and performance. However, other projects where VR has been applied in STEM-courses has had a positive impact on learning. Examples of applications are the LabsterVR application [12], where students can learn topics from various subjects within biology, ecology, and physiology. In the application, they can do experiments that could otherwise be expensive or dangerous to conduct. VR is also currently being applied in various courses as a supplementary experience of the current subject being taught. By using ClassVR [2], teachers hand out teaching plans, and students use immersive experiences within a simple VR headset as part of the lesson.

Educational escape rooms are currently an emerging field. Students solve puzzles, riddles, and other activities to progress through one or more rooms. The activities often facilitate collaboration through class-room activities, but single-player escape room games have become widely popular among VR enthusiasts [3]. An educational escape room was applied in a web programming course successfully [14]. The learning activity has also been applied successfully in other fields, like pharmacy [4]. No studies were found where the educational escape room concept was exclusively applied within VR. However, the concept was thought to have much potential in VR by combining it with the elements that make other educational VR games useful. In the programming course study [14], they stated that the physical puzzles had a very positive impact on the experience, but that they could not be recreated for the digital tools. However, VR shows the potential for recreating almost anything from the real world, which we thought could prove beneficial in this context.

3 Methods and Implementation

3.1 Research Methodology

The project followed the Design and Creation strategy [18, chap. 8] to implement the VR application for giving an introduction to deep learning. The research strategy was considered suitable since there was no existing literature about using VR in AI education. The application was developed, following the steps of defining requirements, design, implementation, and testing from the waterfall development methodology [6]. Git was used for version control, with a simple workflow, to make sure that the project could be rolled back to a stable version in case something went wrong. The project is available in a public repository on Github [22].

3.2 Technology

Figure 1 shows a simple representation of how the chosen technologies described below were related.

Fig. 1.
figure 1

This figure is a simple representation of the chosen architecture.

Full size image

It was decided to develop the VR application for the Oculus Quest for two reasons; to be able to conduct user tests on multiple students simultaneously and to create a more accessible tool for students, compared to other VR headsets. The Oculus Quest is the first fully standalone room-scale device to reach the consumer market. It requires little effort to set up since there is no need for a powerful PC, cables, and external sensors. However, the device’s hardware is similar to 2017–2018 smartphones. Because of this, applications developed for the device requires higher optimization efforts, compared to other VR headsets to maintain good performance in not too complex environments.

Unity [24] was used as a game engine since it is easier to learn and use for independent developers and smaller teams, compared to alternatives such as Unreal Engine. The Oculus Integration SDK [25] was used for hardware integration, and the Virtual Reality Toolkit (VRTK) [28] was used for setting up interactions and locomotion. The fundamentals of the application were set up with inspiration from the Unity Learn course by Oculus [26]. It was decided only to use free assets found on the internet, to focus on developing a functional prototype. Prefabs were implemented for various objects, to effortlessly reuse them for activities, learning material, and other events triggered through the application. One thing that had a significant impact on the development efficiency was the use of Unity Snaps [27]. Unity Snaps consists of a variety of resources for efficiently creating real-size room structures.

3.3 Concept

Before developing the application, another VR application had been developed for the project. It was user-tested, and we revealed that the project had potential. However, many of the activities and the presentation of learning material did not justify the use of VR. Also, the learning material lacked structure. A new concept was defined based on the evaluation results and the literature study.

The new concept was to introduce educational escape room elements, like puzzles, calculations, and quizzes based on the course material. The different topics within deep learning were split into separate rooms to give the user intuitive progress through the curriculum. The sequence and topics of the different rooms were designed with inspiration from Nielsen’s book on deep learning [16] and the deep learning curriculum from a course called Visual Computing Fundamentals at our university. The idea was to split the curriculum into the topics; neurons, cost functions, gradient descent, and backpropagation, and further split the more complex topics into multiple rooms. The rooms were meant to have a close link to the sequence of the student’s learning goals. The idea of splitting topics into rooms was inspired by the learning technique, virtual memory palace, which has been used since the ancient Greek/Roman times. Throughout the application, users progress through rooms by solving puzzles or other activities. The puzzles should apply 3D-objects and interactive 3D-visualizations to immerse the student in the curriculum. The intended target audience was students in introductory AI and deep learning courses, but a final application could also be made accessible to anyone interested in the topic. A thorough tutorial for teaching every type of interaction was needed since most people in the target audience were assumed to be new to VR.

3.4 Evaluation

Two alternative ways of evaluating the application were planned and conducted due to the Covid-19 situation. The application was evaluated quantitatively through questionnaires that measured opinions related to various statements on a 1–5 scale. The questionnaires were designed to measure engagement, opinions about the concept and to see what students thought about using the application for learning. Also, the questionnaire used for the participants that tested the application attempted to measure usability and discomfort. The results of the questionnaires were analyzed by studying the average score on each statement and describing possible reasons for negative and positive results. The results of the two questionnaires were compared where it was possible. The application was also evaluated qualitatively through written feedback. All data collected was anonymous, and could not be used to identify a respondent. The goal was to get as many participants as possible in the period from the beginning of April until the middle of June 2020, despite the challenging situation with Covid-19.

Online User Tests. The application was distributed online in the IMTEL network [9] and the Reddit VR communities r/OculusQuest and r/oculus. People with VR headsets participated from home. They were given a guide for installing the application on any Oculus device and enough context to participate. Upon completion of the application, they submitted feedback through a questionnaire. Some of the first author’s classmates also participated. The online user test evaluation started in the beginning of April 2020.

Video Evaluation. A 5-min YouTube video explaining the application’s core concepts [23] was created and distributed online, along with a questionnaire. Most of the participants from the online user tests were highly experienced with VR. Therefore, it was attempted to reach out to AI research groups, AI students, and the Reddit AI communities r/artificial and r/ArtificialIntelligence and the groups involved in the online user tests. People were requested to respond to the questionnaire as well as they could, after watching the video. They were also given a chance to watch a full play-through for more insight. The questionnaire was designed knowing that the participants would not have the same insight as the people in the other group. Also, since the video evaluation started in the middle of May 2020, the questionnaire was improved and extended compared to the other questionnaire.

Fig. 2.
figure 2

Image to the left shows a task where users learn neural network notation by correctly placing neurons. Image to the right shows a visualization of gradient descent. Unity’s terrain builder tool was used to mimic a cost function surface with two-dimensional inputs. The ball simulates gradient descent’s process of taking small steps in the direction of the negative gradient, to reach a minimum.

Full size image

4 Results

4.1 Prototype Application

The resulting VR application is described in a 5-min YouTube video [23]. In the associated video description, there is also a link to a full play-through of the application and the tutorial. More work is needed before applying the application in a course. Therefore, it is considered a prototype.

Learning Material. Since the application was developed by a single person only, the main focus was on creating engaging activities and some 3D-visualizations. 3D-visualizations were developed for neural networks and gradient descent, as shown in Fig. 2. The application ended up having many text panels, which we knew were not optimal for VR. The plan was to apply more immersive types of learning material, such as audio, 3D-visualizations, and videos. However, an extensive amount of work was put into creating the contents of the text panels, so they show potential for being conveyed in more immersive ways. The curriculum’s topics were split into separate rooms, and within each room, the learning material was presented in a logical order. The learning material content was made with inspiration from Michael Nielsen’s book about deep learning and neural networks [16] and the 3Blue1Brown series on YouTube [1].

Fig. 3.
figure 3

This figure illustrates the application’s quiz system area, where users can bring cartridges loaded with quizzes. The unfinished red cartridge has been inserted in the cartridge player on the right side of the image. The quizzes appear on the big screen, and the user can respond to it using the physical buttons in front of it. When every cartridge in the application is won, the user can collect them to complete the application.

Full size image

Activities. The application has three main types of activities. The first one is the puzzles, where the user needs to place 3D-cubes, and spheres for learning notation and building neural networks. An example is shown in Fig. 2. The second type of activity is calculations, where the user needs to calculate a neuron’s output. This is partly done using mental arithmetic, but since one part of the calculation is too complex for mental arithmetic, the student can use a calculator. The final type of activities is quizzes. The user finds cartridges loaded with quizzes after learning each topic. They are then brought to the quiz system, shown in Fig. 3. The user completes the application by winning each of the quizzes and placing the cartridges in a bookshelf.

4.2 Evaluation

The application was distributed in the places mentioned in Sect. 3.4. 15 people responded to the online user test questionnaire. The respondents were either current or previous computer science students. Their experience in terms of AI was varying. Most of them were highly experienced with VR. 9 of the users were from the groups reached out to online, and 6 were classmates with the first author.

26 people responded to the video evaluation questionnaire. 88% were previous or current computer science students. 62% of them were current students, and we can assume that most of them were students in their third to fifth year of computer science, taking the Computer Vision and Deep Learning course at the Norwegian University of Science and Technology. We can assume this considering the time of the response, compared to when the questionnaire was shared. The rest of the previous or current computer science students were researchers, lecturers, or employed, and we can assume that most of them were members of AI research groups considering the time of the response. The participants’ experience with VR was ranging evenly from low to high. Their experience with AI and deep learning was also somewhat varying, but most of them had a higher experience.

Fig. 4.
figure 4

Responses to the online user test questionnaire.

Full size image

Questionnaires. The most interesting results from the questionnaires have been visualized as Diverging Stacked Bar charts [19]. Figure 4 shows the responses to the online user test questionnaire. Figure 5 shows the responses to the video evaluation questionnaire. The vertical axis shows the various statements, while the horizontal axis represents the total amount of responses on both the agree and disagree side of the scale. The numbers on a bar represent the number of people with that response. A positive or negative response can be on either side of the scale, depending on the statement.

Written Feedback. 12 respondents to the online user test questionnaire submitted written feedback. 8 submitted written feedback to the video evaluation questionnaire. Through their feedback, there were some key elements addressed. Most of the feedback covered below was submitted by the people who tested the application.

  • Concept. Multiple respondents had positive feedback considering the application’s concept. Some stated that dedicated rooms for each topic helped progress, and the room layouts looked good but should be more easily distinguishable. Also, the users enjoyed escape room elements like doing puzzles to progress.

  • 3D-visualizations. The respondents stated that they thought visualizations of gradient descent and neural networks were helpful. Some also stated that this was the best utilization of the technology. The users were interested in seeing more interactive 3D-visualizations.

  • Too much text. Most respondents stated that the amount of text used for learning material was too high. They suggested replacing the text panels with more immersive types of learning material, such as audio, 3D-visualizations, and possibly videos.

  • Activities. Multiple users stated that they thought the activities would be useful for learning. However, for backpropagation, some users were confused and ended up solving tasks by trial-and-failure since the learning material was somewhat lacking or confusing. Multiple users also enjoyed the quiz system solution and stated that collecting cartridges made the experience more engaging and rewarding. One user stated that collecting cartridges felt unnecessary. One user compared the activity for learning neural network notation to a lecture and felt that the hands-on experience made learning easier.

  • Applying the application in a course. One of the respondents of the video evaluation questionnaire compared the method of learning with traditional methods. His key point was that the application’s strengths were through the use of practical tasks and calculations for learning the curriculum. However, compared to traditional methods, he considered the application inefficient for entirely understanding a topic since this would require mass-training.

Fig. 5.
figure 5

Responses to the video evaluation questionnaire.

Full size image

5 Discussion and Limitations

5.1 Research Outcome

The present study explored what students, researchers, employees, and others related to AI and VR thought about the application. The questionnaires mainly measured engagement, opinions about the concept, usability, how students would like to use the application, and attempted to get an indication of whether the participants felt that they learned something or not. The results show that the respondents of both questionnaires were generally highly interested in the application’s concept. The Background and Related Work section mentioned the successful usage of classroom escape rooms in a variety of contexts, but no previous studies had explored the exclusive use of educational escape rooms in VR. This project shows that physical puzzles can indeed be ported to VR experiences, and be received with interest from users. The current escape room games for VR have reached high popularity, and with the participants’ interest in the concept, it can potentially prove beneficial in education in general. One of the main advantages of applying VR in this context is through the use of interactive 3D-visualizations, that are otherwise hard to replicate. The participants showed great interest in the visualization of gradient descent and requested more 3D-visualizations of the curriculum. Therefore, separating complex topics into different rooms can potentially prove to be beneficial in other courses that require a visual understanding, such as STEM-courses.

The activities used throughout the application was also met with interest. The application developed for the project showed an improvement of the activities compared to the first application we evaluated since the activities were mapped to 3D-objects in the virtual world. The users seemed to enjoy the task of getting the right output on a neuron, doing calculations, and quizzes. The notation activities seemed to be useful for some users. However, since others ended up solving these tasks through trial-and-failure, either the activity type or the learning material needs to be improved or redesigned.

Considering how the application can be applied in a course, we see from the results that the users were positive about using it as a supplementary tool, but not using it as a replacement of traditional learning methods. The interactive 3D-visualizations and hands-on tasks were met with engagement and interest from the users. However, the large use of text panels for learning material did not justify the use of VR. Multiple users stated that they felt the amount of text was too high and suggested replacing it with more immersive types of learning material, such as audio, 3D-visualizations, and videos. This feedback was expected since there was not enough time to create such learning material, considering the project’s scope. The results do not imply that text panels should be fully replaced, but that they should be designed for comfort, and the number of panels and amount of text should be kept to a minimum. Some parts of the curriculum, such as formulas, are better conveyed through text and smaller pieces of information. However, their contents can be conveyed through a combination of text and other types of learning material. The reading experience was somewhat worsened throughout the application since the performance was not optimal in parts where large amounts of content were being drawn. More efforts need to be put into optimization to fulfill the performance requirements laid out by Oculus. Furthermore, the results indicate that using such an application in this context could work well if the students can use it to get an introduction to new topics, and then proceed with traditional learning methods. Therefore, the application might be a suitable replacement for some lectures but mainly a supplementary tool for the rest of the teaching methods. However, it is important to keep in mind that VR can not replace the mass-training required to get the deeper understanding of the curriculum.

5.2 Limitations

This study did neither compare the learning method to other learning methods nor attempt to measure learning outcomes. For future work, this would be required to determine if using VR in this context is useful compared to other learning methods. The study shows some positive indications of what can make the tool useful, but developing for VR is more resource-intensive than developing for-instance an interactive website. Even though VR opens for new possibilities for conveying the curriculum, the method should be compared with traditional methods and other digital tools. However, the study shows that VR applications for learning can be developed efficiently with the right set of tools, assets, reusable contents, and good design and architecture.

The VR application was designed and developed by a single person with a software-engineering background. It would have been beneficial to develop the application using a dedicated team of designers, software developers, and more pedagogical content-creators.

The questionnaires were somewhat biased towards positive feedback and could have focused more on getting feedback from a more pedagogical point of view since this is an important aspect of the project. The questionnaire used for the online user tests was designed at an earlier stage than the one for video evaluation. During the period of online user testing, we discovered limitations with the questionnaire. Therefore, the questionnaire was revised and drastically improved for the video evaluation as an attempt to bias participants more towards negative feedback and attempt to discover more specific elements of engagement. One disadvantage of doing this was that it made the results more difficult to compare. For future work, there should be a more consistent and well-designed questionnaire for all evaluation. Future questionnaires need to be better designed to better cover the application from a more pedagogical and technical point of view. Also, there needs to be a better balance between gathering positive and negative feedback.

Due to the difficult times of the global pandemic of 2020, the application could not be tested on the intended target audience. The initial plan was to evaluate the application using students from an introductory deep learning course in a laboratory. This would be done in groups using six Oculus Quests. When we evaluated the other application developed for the project, we used this method and conducted interviews and observations. Conducting interviews were highly valuable to get more insightful feedback. Since this was not possible when Covid-19 broke out, we chose to alternatively evaluate the application using what we thought was the best alternative. However, the people that tested the application during the online user tests were generally highly experienced with VR, with some participants that had a lower experience. Therefore, the participants could have been more positive about using VR than the average person in the target audience. The respondents of the video evaluation reflected the target audience better. However, they did not have the same insight since there was no guarantee that they watched more than the 5-min concept video. Trying the VR application would be a highly different experience. The participants of the video evaluation could not interact and feel a presence in the virtual world like the groups from the other evaluation did. However, the questionnaire used for the video evaluation was designed with this in mind, so that the participants would not have to respond to statements that would require testing the application. Therefore, even though the participants responded somewhat based on assumptions, the evaluation method led to some valuable feedback, which would not otherwise be possible to gather during the global pandemic. For future work, the application should be user-tested by people that better reflect the target audience using the initially planned methods.

6 Conclusions and Future Work

Based on the feedback gained, we believe that the application’s concept of splitting the curriculum’s topics into separate rooms and encouraging students to solve hands-on puzzles to progress is an engaging and rewarding way of learning. The concept of an “educational escape room in VR” could potentially be beneficial in other types of STEM-education as well. There are two main advantages of using VR in the context of AI. The first one is to apply interactive 3D-visualizations based on the curriculum’s concepts that are otherwise difficult or impossible to convey. The second advantage is through “learning-by-doing” activities that the students need to solve to progress. Compared to other teaching methods, the VR solution gives students a new level of immersion in the curriculum and a more hands-on experience. From the results, we saw that the participants felt that they learned something. However, the VR solution needs to be compared with other methods to measure the effectiveness of learning in this setting. Still, we believe that we have uncovered a teaching methodology that can be beneficial in courses that require a visual understanding of the curriculum.

We believe that a VR application like this could work well in the context of AI if the tool is used for learning something new. The focus should be on teaching what is challenging to convey through other means. One possible use case for the final application is for students to get an introduction in VR and then let them proceed with traditional learning methods to understand a topic entirely. The application shows what is possible to develop by a single person with little experience with VR development prior to the project. However, the application suffered from some unresolved performance issues. Therefore, we believe that standalone VR headsets such as the Oculus Quest could be a great and more accessible tool for students when more resources are put into optimization and development. A tethered VR headset would require dedicated rooms for using a VR application, while multiple standalone VR headsets can be brought anywhere by the course staff. Companies who are putting their employees through lifelong learning programs can also apply VR applications with ease. Since some courses require prior knowledge in deep learning, the final application can also get students who lack this knowledge up to pace with the other students.

For future work, more efforts need to be put into improving the current application and designing new activities, learning material, and interactions. Before applying the final application in a course, greater efforts would be needed for replacing text panels, adding more interactive 3D-visualizations, improving and adding more activities, optimizing the application, and improving learning material contents. Furthermore, the application should be user-tested on a group that better reflects the intended target audience. Also, the VR application should be compared to alternative technological and traditional learning methods to measure the learning outcome and effectiveness of using VR. An interesting approach to this could be to apply the final version of the VR application in a course, comparing the effectiveness with a control group that exclusively uses the current methods for learning the curriculum.

Seeing how positive users were to the concept of an educational escape room in VR, some interesting future work would be to see how the VR application’s concept can be applied in other STEM-courses, such as physics or chemistry.