1 Introduction

In the past 30 years, the field of computer graphics has developed rapidly and introduced ever more complex and realistic graphical tasks (Mutlu et al. 2014). Augmented Reality (AR) is one of the most sophisticated computer graphics areas and has become popular with the development of smart phones and other mobile devices.

AR offers extended vision to the user by overlaying real-world images with computer-generated objects or information. In the earliest days of this technology, a Head Mounted Display (HMD) and a portable computer were required to acquire the real-world imagery and combine it with virtual content. Now AR can be achieved with a smart phone or AR glasses.

AR has a wide range of application areas from education to personal health (Li et al. 2020). It also promises great potential for cultural heritage. The reconstruction of cultural heritage sites typically is a time consuming and costly process. In addition, the possibility of damage to the remains during construction needs to be considered. With AR technology, it is possible to place 3D models that are designed in accordance with the originals on top of the ruins in cultural heritage sites without physically reconstructing areas. Using this technology for cultural heritage sites will be a very advantageous process in order to protect the remains and to save the time and cost of reconstruction (Unal 2017).

This chapter mainly focuses on Augmented Reality and its applications for cultural heritage. Our aim is to show the studies in the literature and display the potential of AR technology in cultural heritage. This study also includes our previous works in a detailed manner.

The rest of the chapter is built as follows: First, AR technology is explained in detail with its application areas included, then the cultural heritage domain of AR is demonstrated and the motivation behind using AR in the heritage sites and museum is explained. Next, some of the literature on AR in cultural heritage has been presented in two sections, namely Visualization and Gamification, then future work and the Total Augmentation Paradigm are described. Finally, the last part of this chapter concludes the study.

2 Augmented Reality

Augmented Reality (AR) is a concept of binding real-world images and artificial assets and information in real time to enrich the human perception of environment. AR can sometimes be confused with Virtual Reality (VR). The difference between AR and VR is the use of background real-world images. In other words, VR can be achieved using only an artificial environment, while AR incorporates a level of reality.

According to Azuma (1997), an AR application should have the following features:

  • combine the real and virtual worlds,

  • be interactive in real-time,

  • operate in 3 dimensions.

Within the boundaries of this definition, the concept of AR is not limited to Head Mounted Display (HMD) but can be suitable for new mobile systems. Presently, AR technology is usable and applicable for different devices, including mobile phones, tablets, and smart glasses.

AR has been implemented for very different devices. The earliest applications of AR used an HMD with a low range of motion which was developed by Sutherland (Sutherland 1968). These HMDs became more sophisticated with advancing technology and now, two types of HMD are available on the market which are video-see-through and optical-see-through (Carmigniani et al. 2011). As the respective names suggest, the video-see-though HMD stream the video to the user while the optical-see-through models use a half-silver technology to allow the user to see the vicinity and augment virtual information on the screen. Smart glasses, the successors of the HMDs, are providing a much more suitable environment for the use of AR technology.

State-of-the-art smart glasses like Google Glass or Epson Moverio promise great potential for AR technology, yet it seems likely that it will take some time to start using this technology in daily life (Ro et al. 2018). Also, notebooks (laptops) were used with a camera mounted on a user for AR (Stricker and Kettenbach 2001; Piekarski et al. 2004) in the 2000s. With the new millennium, Personal Digital Assistants (PDAs) were introduced as a new medium for AR technology which has a more practical usage than notebooks or HMDs (Wagner and Schmalstieg 2003).

After the PDA era, smart phones were introduced and become very popular in a relatively short period of time. These devices have much greater processing capability compared to PDAs and are more portable than notebooks. Consequently, smart phones have emerged as a much more suitable environment for AR (Azuma et al. 2011). Since then, a tremendous number of AR applications have been presented in the literature. One of the interesting implementations of AR applications is using drones as a capturing device (Unal et al. 2020a). Although, it requires sophisticated methods for precise tracking (Unal et al. 2020b), the results are promising for the future. Lastly, Microsoft Kinect, which is a motion detection sensor, was used for AR studies in the literature (Vera et al. 2011; Casas et al. 2012).

The type of virtual data to be superimposed on top of the real image also increases the variety in applications. If the data to be superimposed on the image is part of an interface, a marker or a text, a 2D virtuality is provided and depth is not required (Liou et al. 2016; Nuernberger et al. 2016). There are also 3D AR applications that are more interesting, popular, and perhaps more challenging than 2D applications (Mourtzis et al. 2017; Panou et al. 2018). Nowadays, photo and video sharing applications like Instagram and Snapchat for smart devices come with small 3D AR add-ons.

Different surveys in the literature present a very broad range of application areas for AR (Wang et al. 2016; Chatzopoulos et al. 2017; Kim et al. 2017). Accordingly, AR can be employed for entertainment, advertisement, education, assembly, and medical areas. Furthermore, the literature indicates that cultural heritage is one of the most promising areas for AR.

3 Augmented Reality in Cultural Heritage

Cultural heritage AR applications have been shown as an important area of AR by Azuma (1997) and Papagiannakis et al. (2008). With the developing technology, the cultural heritage applications of AR became a very important instrument to transfer historical knowledge and experience. Studies have shown that the interest is increasing when an AR application has been employed in a museum or historical site (Haugstvedt and Krogstie 2012). In this way, historical awareness and knowledge of societies can be increased (Kounavis et al. 2012).

Reconstruction models or virtual tours of archaeological sites provide an enjoyable learning tool. Applications of AR in cultural heritage can improve the visiting experience of a heritage site by in situ reconstruction that elevates it with 3D models of ancient buildings. While AR systems take considerable time to develop, another beneficial feature of such systems is that they can be implemented in situ with minimum physical damage to the remains or artifacts. AR reconstructions can be in different configurations, while some applications provide a roam-able environment around the heritage site, others can only present the reconstruction of old buildings from a fixed perspective as they are (Bostanci 2014).

3.1 Visualization

In this subsection, visualization-based applications of AR which include superimposed virtual objects on top of real-time cultural site images will be discussed.

Cultural heritage visits can be enhanced by showing 3D models of ruined ancient buildings on site. Historical structures built by ancient civilizations have often been destroyed or ruined by wars, earthquakes, and other disasters. Physical reconstruction is required to return these structures to their state when they were built. However, this is a very laborious and presumably costly process. Instead, it is much easier and presumably cheaper to superimpose virtual models designed in a computer environment on historical buildings or sites.

The literature presents different visualization applications for virtual reconstruction of historical sites since the beginning of the 2000s. Sticker and Kettenbach (2001), implemented an AR application where its location was determined with the help of reference photographs for open spaces. In this application, the computer-generated 3D model of the historical building was placed in the correct position with the help of reference photographs.

Vlahakis et al. (2001) developed ARCHEOGUIDE project which is a tour assistant and guide for the archaeological site of Olympia, Greece. Projects include real-time AR reconstruction of ancient buildings and a personalized guide for visitors.

Bruns et al. (2007) demonstrated one of the earliest mobile AR applications that can be used with a camera-equipped mobile phone. The application used object recognition to identify and track the historical artifacts at a museum. The detailed information and multimedia content about the tracked artifact were shown to the visitor. The application also provided location-based contents to the visitors of the museum.

Choudary et al. (2009) presented a mobile application (MARCH) to enhance the visiting experience of restricted prehistorical sites. The application is designed to augment the prehistoric drawings on caves on the French Pyrenees through the interpretations of experts. The system is designed to work without a marker.

Zoellner et al. (2009) proposed an approach to using AR to virtually reconnect an artifact on a museum with its excavation site. The paper emphasizes that the artifact that was excavated from its original site and placed into the museum loses its context. Thus, the study presented an AR system for museums to provide contextual awareness to the visitors. The application binds a large-scale image and virtual information (like text and videos) on top of the historical artifacts. Also, 3D reconstructions models of the artifact are superimposed to the historical structure. The system is designed to work on video see-through tablets.

Damala et al. (2012) developed a prototype of personalized AR guidance system for museum visitors. The project, titled as “ARtSENSE,” uses data from three different sensors (visual, audio, and psychophysiological) to adapt the multimedia content according to the interests of the visitor.

A different approach has been developed using Microsoft Kinect by Bostanci et al. (2015). In this study, Kinect was used to calculate the camera pose and augment the 3D model of the columns by locating the rectangular features. Human tracking has been implemented to the project and a 3D model of clothes of roman soldiers is augmented to the human on the screen. Results of tracking and augmentation can be seen in Fig. 12.1.

Fig. 12.1
figure 1

The augmentation results of a Kinect-based application (Bostanci et al. 2015)

Full size image

A mobile AR application was presented by Galatis et al. (2016) to overcome the usability and acceptance problems of AR. The application, KnossosAR, is a mobile AR guide to support guided visits of students, designed for a historical site Knossos in Greece. The application superimposes the computer-generated content on top of actual historical site to enrich the visiting experience of the students. The study claims that using the mobile AR technology successfully increases the interest of the students and that AR is a usable technology in cultural heritage sites.

The literature presents many different visualization applications of AR for cultural heritage. In light of this, we designed and implemented an AR application which works in Android mobile devices, for Roman Baths of Ankara (Turkey) (Unal et al. 2020b). This project will be explained in the remainder of this subsection.

As the first step, we designed a 3D model of the Roman Baths using 3DS Max (Autodesk Inc 2020) according to the drawings of experts because the original baths have fallen into ruins. After the raw design, an optimization phase was employed to reduce the size of the 3D model which is important for mobile devices. The last step was assigning the texture to render a more realistic model. The final state of the model can be seen in Fig. 12.2.

Fig. 12.2
figure 2

Model of Roman Bath (Unal et al. 2018)

Full size image

The project was designed as a mobile AR application which can be run on smartphones and tablets. Nowadays, these gadgets are widespread and easily accessible, which provides access to AR technology for anyone who installs the application. The Unity 3D (Unity Tecnologies 2020) which is a widely used game engine and provides a suitable medium for AR has been chosen as the development environment.

The main challenge in an AR application is the tracking of the user (Bostanci et al. 2013). In this project, we used two different tracking techniques, namely: geo-location based and vision based.

The geo-location-based method depends on data from two sensors which are GPS (Global Positioning System) and gyroscope. These sensors are able to integrate with almost all mobile devices. GPS and gyroscope data are used to find the position and orientation of the user, respectively.

The vision-based technique has also been implemented. This method uses camera image and distinguishable objects around the site. Algorithms detect objects in the vicinity and calculate the position and orientation of the mobile device held by the visitor.

After the implementation of the tracking phase, the augmentation phase which includes placing the 3D model of the bath on top of the camera image has been employed. The augmentation system uses both geo-location-based method and vision-based method separately. The result of the augmentation can be seen in Fig. 12.3. Please note that the case study has not been implemented on the actual site due to the risk of damage to the historical remains.

Fig. 12.3
figure 3

In-situ augmentation result (Unal 2017)

Full size image

In the final phase of the project, a distant augmented reality system was developed. This system uses the same tracking methods mentioned above, to track a drone. The camera stream and location data of the drone are transmitted to the mobile device and a 3D model is then augmented on top of the stream. The result of the distant augmentation can be seen in Fig. 12.4.

Fig. 12.4
figure 4

Distant augmentation result (Unal et al. 2020b)

Full size image

3.2 Gamification

The AR-based games have been developed for many purposes, especially entertainment and education (Ozdamli 2017). With the widespread use of mobile devices, many AR games have been developed (Jang and Liu 2019). In 2016, Pokémon GO, a location-based AR game, was introduced and became the most popular game on Google Play (Rauschnabel et al. 2017). Interest in AR games has increased tremendously with this development (Flavián et al. 2019). The game also demonstrated the high potential of AR games (Paavilainen et al. 2017). This subsection presents game applications of AR in the cultural heritage in the literature.

Historical places and buildings can become more interesting with AR games and it can especially appeal to the new generation. Also, visitors can learn more about the site history with location-based AR puzzles (Dieck and Jung 2018).

One of the earliest examples of a tourism-based AR game is the TimeWrap which is an interactive mobile location-based game used to explore the history of a city (Herbst et al. 2008). The aim of the game is to allow the visitor to discover the history of Cologne, a city largely destroyed in World War II. The game requires visitors to equip both an AR system which includes a laptop and head-worn optical see-through display, and a handheld device which provides an information page and interaction map. Visitors need to carry a laptop with an onboard graphic card. The final version of the game had problems with GPS jitter and registering.

Angelopoulou et al. (2011) introduced a multi-user AR application to use in the museum and historical site of Sutton Hoo which has both indoor and outdoor environments, to help inform the visitor about the site with an entertaining puzzle game. The application was designed with the ARToolkit to work with markers which are placed around the site.

Thon et al. (2013) designed an interactive game using drones to increase the number of visitors to the Arlaten Museum in France. This study aims to reduce the effects of stereotypes about museums, using a camera-equipped drone. The drone is used by the visitors to fly around the perimeter and then virtually shooting the stereotypes about the museums which are located in the vicinity. In this way, the museum became more popular and appealing for visitors of all ages.

Bostanci et al. (2013a) designed an AR game titled Treasure Hunt. The game uses Simultaneous Localization and Mapping (SLAM) to locate the user in the vicinity. The SLAM algorithm, which was originally used with robots, strengthened with robust data association to reduce tracking errors. The game was designed to work on a simple system which includes camera mounted helmet and a laptop. The aim of the game is to collect the coins and treasures around the environment by walking and reaching the virtual objects. An in-game image can be seen in Fig. 12.5.

Fig. 12.5
figure 5

The Treasure Hunt game (Bostanci et al. 2013b)

Full size image

4 Future Work

Augmented Reality promises great potential and yet needs powerful mobile devices to render 3D models (Ar et al., 2018). Cultural heritage sites and museums can become more entertaining and informative with AR technology. Although current studies in the field of AR are generally still theory based, it has the potential more widespread in daily life in the near future.

The literature presents many different ways of enhancing user experiences in historical sites. The view of the visitor can be enriched with text and sound as basic augmentation. More sophisticated systems can provide an environment of augmented 3D models.

As future work, we aim to reach a Total Augmentation Paradigm which includes using not only mobile devices but also smart glasses and drones in a heritage site (Bostanci and Unal 2016)—see Fig. 12.6. In the system, the drones fly and capture the historical site and transmit the image to mobile devices or smart glasses. The augmentation is achieved on these smart devices in real time. The visitor can see other visitors with the era-appropriate clothes of the historical civilization. Also, a virtual storyteller gives information as the visitors roam around the vicinity. This system can be implemented with powerful mobile devices and smart glasses in the future.

Fig. 12.6
figure 6

The Total Augmentation System (Bostanci and Unal 2016)

Full size image

5 Conclusion

With advancing technology, the models designed in more detail will give more realistic results in more powerful devices, and the “reality” we know in the near future could begin to be insufficient for people.

In the early 2010s, with the increase of smart devices that are suitable for application development, the number of mobile applications developed has increased incredibly. Augmented Reality technology has also received its share from this increase. Many applications have been developed with this technology and the success of some of them has made a global impression. Many Augmented Reality applications have also been developed for cultural heritage sites with different tracking techniques including not only marker based but also vision based and location based.

The implementation of AR to cultural heritage has been an interesting matter of research, as it assists the conservation of original artifacts that have been exposed to aging for hundreds of years and offers a fun way to learn their history by viewing original artifacts in situ. Accurate user tracking is the most important part of developing and implementing an AR application for cultural heritage. In this study, we presented AR technology and its usage in cultural heritage in the literature. The AR studies in cultural heritage context have been demonstrated in the Visualization and Gamification subsections. As a result, the authors believe that AR technology has a promising and bright future and has the power to radically change the visiting experience of the cultural heritage sites.