Keywords

1 Introduction

The new era of digital age and recent advancement in cutting-age immersive technology have opened new opportunities for accessing and presenting cultural heritage in interactive and interesting style. Virtual reality provide a new medium to provide experience and training for different application in education, medical, and industry [1,2,3]. VR can provides an immersive experience for different cultural heritage sites [4] in contrast to traditional methods of digitizing the heritage using basic storage medial such as video and audio. This could be sufficient documentation and not for preservation. For heritage preservation it is important to convey context and experience of the heritage community [5]. Virtual environment (VE) enable participants to explore, interact, examine, and have the sense of presence in the same atmosphere of old times. This work aims at capturing the heritage sites and convert it to 3D models and create interactive virtual environment of the site with animations, interaction, and avatars that can bring back the site to live.

Creating VR experience for many cultural heritage have been explored actively in the recent years, especially with advancement of VR headsets in the recent years where it becomes affordable and available to consumers and gamers. Fen et al. presented a virtual experience of Han Chang’an City [6]. Häkkilä et al. developed a virtual reality experience to visit the inaccessible Salla World War II graveyard where they created an accurate simulation and atmosphere [4]. There are many applications In the context of introducing the concept of serious games to the culture sites such as a serious game for learning Egyptian hieroglyphs where it uses immersive and interactive virtual learning worlds to support transfer of knowledge [7]. A serious game for the ancient Agora of Athens was presented in where it provided virtual tours of the area and a game quiz [8]. Eslam and his colleagues presented a cultural learning environment of heritage artifacts for Young Museum [9].

Creating a realistic 3D models of the site is another challenge, especially if the scale of the site is large. Many research papers had different approaches in order to utilize photogrammetry to create highly immersive virtual reality experience. Vajak and Livada [10] case study focuses on converting images of real time world captured using photogrammetry to a 3D model. Photogrammetry was done by capturing eight hundred images using Samsung Galaxy s6 phone. The photos were processed in different software to get the generated textured 3D model. The generated model was imported to Unity3d to create scripts, write code, add objects and mechanics to accomplish a high-quality VR interface. Grun et al. [11] mentioned a workflow that combines photogrammetry and virtual reality to restore cultural heritage. Photogrammetry and laser scanning were used for the interior structure of the temple. A set of steps and different software were used to construct a virtual reality interface with the help of photogrammetry. Yastikli [12] has done experiments that involved using digital photogrammetry and terrestrial laser scanning and concluded that using any of these methods can restore and document cultural heritage. The experiments include the workflow for each method from acquiring data, building of texture mesh to get the final product needed. Leonov et al. [13] had a different approach as laser scanning methodology was used on Shukhoov Hyperboloid tower in Moscow. It was mentioned that there are different techniques required to restore cultural heritage sites. Their approach was to create a 3D model of the tower and develop a software application for public users. Nevertheless, many approaches in how a three-dimensional model can be created and how it can be turned into an immersive interface using different workflows and techniques.

The presented work combines the previous researches by mixing between historical retrieval and creating an immersive and interactive scene by adding Game components, avatars and scenarios with the usage of a modern interactive head mounted display (HMD). The goal is to bring the historical sites to life by adding these elements together using game engine as well as photogrammetry.

2 System Architecture

Photogrammetry technique was selected mainly because it is a low-cost and affordable solution. The laser scanning has a smaller footprint and higher accuracy but very expensive. Our main target is to provide the workflow for an affordable and cheap solution with high accuracy. The first step is to capture the site using camera or drone, then will create a 3D model from the image using photogrammetry software. The 3D model needs cleaning to make it ready to export to the game engine. AgiSoft MetaShape was used to convert images to 3D model and MeshLab was used to clean the model. Also, Blender was used to remodel and re-sculpt some errors and finalize the final model and make it ready for rendering in the game engine. Some textures were damaged and needed editing, therefore, Paint3D from Microsoft was used to fix textures. Once the model and the textures are ready, it will be imported as a game components. In the game engine navigation as well as scenarios and characters were added that allows the users to interact with in the environment. Support for virtual reality is achieved by converting the system to VR components and implemented interaction with controllers. HTC Vive Cosmos were used as an HMD where it allows freedom of interaction and walking in a room-scale mode. This will make the interaction more intuitive by not restricting the user movement and apply walkaround and natural interaction with ability to move around and grasp objects. Figure 1 shows the overall system architecture.

Fig. 1.
figure 1

The overall system architecture.

Full size image

3 User Interface Design

Our aim is to build a user-friendly interface where user interaction is not done by a mouse and a keyboard, instead the user will use a wand or controller and a headset instead of a monitor to view the scene. The user can navigate and move freely in the scene. The interactivity in the system is what enhances the usability. The user has the ability to roam freely on valid locations and has the option to teleport as shown in Fig. 2, the system will show the teleportation in green as an indicator that it is a valid point. The user can point at the button on the board where it will change color upon hovering to blue and red when pressing. Once pressed the tour guide will walk towards the building and begin to speak as shown in Fig. 3. The user has the ability to interact with the many objects in the scene as they usually do in the real site. The user can interact with the well in the site, once he walks up to the handle, it will be highlighted in yellow as shown in Fig. 4 and he can rotate the handle which will produce a bucket of water. The user can also interact with various artifacts in the scene as shown in Fig. 5 and have a closer inspection of what the object looks like. This is provides a unique experience as it is usually not allowed in the historical and museums to touch such artifacts.

Fig. 2.
figure 2

The user move freely using teleportation technique.

Full size image
Fig. 3.
figure 3

Screenshots of the smart avatars to provide tour guidance.

Full size image
Fig. 4.
figure 4

The user can interact with virtual objects as he do in the real site to provide the same experience in real life.

Full size image
Fig. 5.
figure 5

The user can interact with many artifacts in the scene where usually are not allowed to touch in the real site.

Full size image

4 Evaluation

The system was evaluated based on different well established gaming questionnaires. Ten subjects were recruited from different background and wide spread ages.

4.1 Game Experience Evaluation

According to IJsselstejin and his colleagues [14], there are three types of categories to ask the users to get the best answers regarding the game experience of the system, so the first section of the survey was divided to three categories, in-game, social presence module, and post-game module.

In-Game Module

The objective of this test is to understand to what extent the users were immersed and their sense of presence in the virtual world. Also, how interesting or boring was their experience. The participants were asked while they are playing inside the virtual environment to get instant answers about what they are feeling at this moment. This is essential for such questions not to be postponed to after finish as they will try to recall how was their feeling which might not be easy to recall. The results shows that over 70% found the simulation impressive, contended, and successful. Between 57% to 62.6% found themselves absorbed and completely immersed in the simulation. Less than 35% felt bored and only 20.4 felt frustrated as shown in Fig. 6.

Fig. 6.
figure 6

In-game questions chart.

Full size image

Social Presence Module

Since the VE include human avatars walking around the scenes and some of them approaching the participants to provide help and guidance, it was necessary to investigate how users reacted to avatars and what was their impression and experience. Therefore, the users were asked while they still testing the system a set of questions to investigates their psychological and behavioral involvement with other social entities. The results shows that 62.5% felt very delighted, 52.5% found the experience enjoyable, 50.1% connected to the avatars, 47.6% paid close attention to others, and less than 25% did not feel that they affected the other’s actions (see Fig. 7).

Fig. 7.
figure 7

Social presence module chart

Full size image

Post-game Module

The objective of this test is to assess how the users felt after finished the whole experiment. Therefore, the users were asked after they had stopped completely and done with the experiment. The results shows that 67.5% felt that they have had a sense that they had returned from a journey which is very important and conclude the purpose of the whole system. 45.4% felt revived and 40.2% felt energized. Meanwhile, less than 20% felt exhausted or felt bad (see Fig. 8).

Fig. 8.
figure 8

Post-game module chart. x

Full size image

4.2 Simulator Sickness Questionnaire

VR headset can cause dizziness and sickness (cybersickness or motion sickness) where some users feel uncomfortable. This was a major issue with the old HMDs since the field of view (FOV) and resolution were very limited, with the advancement of technology the FOV and resolution is drastically improved and this side effect is reduced, however, there are still low percent of people suffer easily experience motion sickness.

In recent study [15], it has been found that more than half the sample (57.8%) have experienced motion sickness whilst 42.2% have never experienced it. For those who experienced motion sickness, 13.7% of people experience VR sickness ‘frequently’, 19.1% experience it ‘sometimes’, and 24.9% experience it ‘rarely’. These figures explains why we had some low figures in some of our evaluation criteria.

After the subject finished their experience and the stopped completely, the users were asked about their physical state such as general discomfort and vision (motion induced sickness) based on simulator sickness questionnaire (SSQ) presented in [16]. The result showed that 27.1% of the users felt some kind of discomfort which can be seen as dizziness, blurred vision, and fatigue. About 10% extreme discomfort like sweating and increased salivation as shown in Fig. 9. This is might be due to the fact that most of the subject never experienced VR before and this is was there first trail. The other reason is that the subjects who felt extreme discomfort were above 50 and had dizziness issues. However, these figures fails withing the range of the above study where 13.7% of people had a VR sickness ‘frequently’.

Fig. 9.
figure 9

Simulator sickness evaluation.

Full size image

4.3 System Usability Scale

Measuring the overall system usability is essential to inspect the users perception of ease of use and consistency. System usability scale (SUS) was used to have a low-cost, reliable and ability to get a global assessment of the system [17]. The overall usability of the simulation is above 82%, however, 27.5% felt that the system was not easy to use and this is reflected on the fact that 37.% thinks that they need the support of a technical person and also 32.8% needed to learn many things before being able to use it. Nevertheless, 55% think that they would like to use the system frequently. This evaluation clarified some issues in the usability of the system that need to be addressed to further increase the overall satisfaction (see Fig. 10).

Fig. 10.
figure 10

The overall usability evaluation using SUS.

Full size image

5 Conclusion

In this work, a workflow on how to create a virtual reality simulation for real world heritage site were explored and successfully were able to create a 3D model using photogrammetry on our targeted site. Many techniques were applied to capture the photos and create the model. Cleaning and mesh editing techniques with different software to make the 3D model ready for game engine rendering were applied. A virtual reality environment based on the site we studied while taking into consideration the realism and implementing immersion. Smart algorithm were implemented to control the avatar motion and make it respondent to the user interaction. The overall result based on many evaluation carried out demonstrated the high potential of such approach to create immersive and enjoyable experience. Most importantly, over 62% felt that they forgot everything around them which indicates that the system successfully achieved immersion. Also, more than 67% felt that they have returned from a journey which demonstrate that the system were able to induce the feeling of visiting and enjoying the virtual tour. With the recent Covid-19 pandemic and restriction to visit many historical sites, this project can provide a solution to have a virtual visit to many locations all around the world and create a unique experience from the comfort of our homes.