Keywords

1 Introduction

Extended reality (XR) technologies such as augmented reality (AR), augmented virtuality (AV), virtual reality (VR), and mixed reality (MR) have influenced many industries. The Architecture, Engineering, and Construction (AEC) industry has no exception [18]. A clear definition of MR, AR, and AV is provided by [16], where AR refers to augmenting the real background with virtual contents (e.g., text, images, videos, and virtual objects). In this definition, AV, as opposed to AR, uses computer graphic images or videos as the background behind the real-world elements. However, MR is considered as any environment that consists of a blending of real and virtual objects. As shown in Fig. 1, AR and AV are both subcategories of MR and are mutually exclusive. For VR lying at the right end of the RV continuum, the environment is considered one in which the operator is totally immersed in a completely synthetic world [3]. In recent years, due to equipment updates and mature technology, the use of AR has vastly increased [24]. However, the number of fields that have adopted AV is much less compared with AR, indicating that AR is more popular than AV [4]. Aside from that, it usually is ambiguous to distinguish AR from MR, as the primary function for both is to superimpose virtual information onto the real world. In this paper, articles focusing primarily on AR and MR are investigated. However, articles with a significant focus on a mere virtual environment, i.e., VR, are excluded.

Fig. 1
figure 1

Adapted from [16]

Milgram’s reality–virtuality continuum.

Full size image

AR and MR development history refers to the first see-through head-mounted AR display developed by Ivan Sutherland at Harvard [25]. More recently, Google introduced the Glass [9] to the market with AR function. Microsoft released a head-mounted device named HoloLens [14] that connects users with remote colleagues in real-time and provides hand tracking tools in addition to its head-mounted display. In recent years, researchers have also investigated the use of AR on various mobile computing interfaces, including smartphones [10], laptops [22], and tablets [21]. This paper focuses on the AEC industry since its practical implementation in this industry is still in its infancy despite the rapid growth of the technologies. It aims to provide a comprehensive investigation of the state-of-the-art applications and categories of AR and MR technologies in the construction industry that is required to guide researchers and industry experts to choose the most suitable AR/MR solution for research and implementation.

Several comprehensive literature review publications have been done throughout the recent years. They classified the articles based on various categories and applications in the AEC, AECO (Architecture, Engineering, Construction, and Operation), and AEC/FM (Facility Management) industries. Chi et al. [5] focused on four technologies—localization, natural user interface (NUI), cloud computing, and mobile devices in a literature review of 101 research efforts. The author outlined the research trends and opportunities for applying AR in AEC/FM. Rankohi and Waugh [20] reviewed and categorized a group of 133 research articles within eight well-known journals in the (AEC/FM) industry until the end of 2012. More recently, Cheng et al. [4] reviewed academic journals within the domain of the AECO industry and classified a group of 87 journal papers by the end of 2018 into four application categories: architecture and engineering, construction, operation, and multiple stages. All in all, there is still a need to review the articles based on their application areas and a group of comprehensive dimensions such as the technology type, construction phase, and the improvement focus in the AEC industry. Compared to the review articles that are mentioned above, Rankohi and Waugh [20] included a more comprehensive system for classifying the articles (i.e., improvement focus, industry sector, target audience, project phase, stage of technology maturity, application area, comparison role, technology, and location). Therefore, this paper majorly follows the proposed system in that article, reviews the articles from 2013 to the end of 2021, and extends the proposed approach by adding some categories to the existing categories within the “Technology Type” and “Location” dimensions. The following sections describe the dimensions and the categories in the reference system and the extended one.

The general structure of this paper is described below. First, the criteria for selecting the journals and articles are expressed. The following section demonstrates the distribution of the articles by year and by journal. Then, we provide an overview of the selected categories to classify the AR and MR articles and introduce the extended categories used for the classification practice. Finally, we discuss the results and elaborate on the current and future trends of AR and MR technologies in the AEC industry. Figure 2 illustrates the methodology of this review paper.

Fig. 2
figure 2

Research methodology

Full size image

2 Selection of Journals and Articles

This paper reviews the articles from well-known academic journals from 2013 to 2021 by extending the literature reviews carried out to the end of 2012 by [20] to reflect the current development and the future trend of AR and MR technologies in construction. Therefore, the methodology is similar to the review article by [20]. The selected journals are included in the Science Citation Index Expanded (SCIE) database and are determined using SCOPUS and Google Scholar search engines. The articles were selected in two phases. In phase I, a total of 118 articles (from 2013 to 2021) were found in these journals using a combination of key phrases, including “augmented reality”, “mixed reality”, “construction”, and “AEC industry”, all of which separated with the “OR” Boolean operator. Then in phase II, the authors read the abstract of each article to ensure that the primary focus is within the AEC industry. The articles were limited to only research articles, literature reviews, and case studies within each journal, which finalized this phase with 103 articles. Table 1 shows the selected journals for this purpose.

Table 1 Selected journals for this review paper
Full size table

3 Review and Identification of the Article Characteristics

The distribution of articles by journal and by year is presented in Fig. 3. AIC with 46 articles (45% of the articles) focused more on AR and MR articles compared to other journals such as CR and CEE, covering three articles (or 3%) and two articles (or 2% of the articles), respectively. The maximum number of articles published in a single year is 25 in 2021. Moreover, 17 articles published in 2013 in AIC include a group of 13 articles published in a special issue (August 2013) of the AIC journal with a focus on AR technology. It is entitled “Augmented Reality in Architecture, Engineering, and Construction” [11].

Fig. 3
figure 3

Distribution of articles by journal and year of publication

Full size image

4 Identifying the Existing Categories

In order to better comprehend and further segregate the literature, this paper first adopted the dimensions and categories defined by [20] for classifying the articles. Then, by doing the review and classification, a few other categories in addition to the existing ones are identified to provide this review paper with a comprehensive system of classification and to pinpoint the current and future trends among the AR and MR articles.

5 Proposing Additional Categories

Firstly, from the technology-type perspective, the delivery type of AR and MR hardware systems [20] was divided into two categories web-based and standalone. Web-based AR was defined as technologies that can deliver project information to remote locations instead of standalone individual delivery of AR. However, Craig [6, 7] indicates that web-based AR means that the tool the participant uses to interact with the application is their web browser. On this basis, not all delivery types of AR devices connected to a server to deliver the project to remote locations can be considered web-based. In other words, there are some AR applications that run on devices such as smartphones or tablets, and depending on the project requirement, they might be connected to a server [12] or not connected to a server [23]. In some online platforms, users only need internet access and run the application by logging into an Android or iOS application [19] but not necessarily run on the web browsers. Therefore, from the “technology type” perspective, the categories are redefined within the delivery type, as listed in Table 3 and furtherly discussed in the following sections.

Moreover, from the “Technology Type” perspective, spatial registration, also known as AR/MR tracking system, is a critical technology type. It is defined as the ability to combine the virtual world and the real world through a proper relationship of the relative positions [5]. This is because accuracy was identified as one of the current challenges of AR/MR applications [4]. However, there are minimal literature review papers that investigate and compare the articles based on the type of spatial registration technology they used (2 from 103 articles) [4, 5]. Therefore, the “spatial registration” category is added within the “technology type” dimension, including two markerless and marker-based registration subcategories. In summary, the reference system of classification and the new suggested categories are presented in Table 2.

Table 2 Defined dimensions and their relevant categories, including both referenced and proposed additional categories
Full size table
Table 3 Categories with the current focus of AR and MR technologies in construction
Full size table

From the “Location” perspective, augmented reality technologies can be implemented in different locations during a construction project defined as (a) field and (b) home-office [20]. However, the category of “field” includes a variety of locations in AR/MR implementations. For instance, Koch et al. [13] assessed the possible solutions for indoor navigations during the facility maintenance projects and presented a natural marker-based AR framework that can digitally support facility maintenance (FM) operators. Moreover, some articles considered outdoor light conditions to be a crucial parameter in the success of their developed AR/MR technology [17]. Some other articles showed a focus on both indoor and outdoor built environments throughout their research in the field of AR/MR technology [8]. Therefore, as proposed in Table 2, the category of “field” is divided into two subcategories of “indoor” and “outdoor” environments to comprehensively categorize the articles from the location perspective.

6 Classification of Articles

In this section, 103 articles found within nine years from 2013 to 2021 are classified based on the number and percentage of articles in each category.

6.1 Research Methodology

As expressed in Table 2, five categories are identified in this dimension. Figure 4 illustrates the percentage of articles based on their research method. As shown, experimental/empirical methodology with 41% of total articles is the dominant category among AR and MR articles. Next is the proof-of-concept methodology, with 24% of the total articles. Literature review, questionnaires, and case study methodologies encompass around 12% of the articles and are among the least dominant categories in this dimension.

Fig. 4
figure 4

Percentage of articles based on their research methodology

Full size image

6.2 Improvement Focus

There are four categories to identify where the improvement of AR/MR technologies may occur, including the AEC industry, organization, projects, and individuals. Figure 5 depicts the number of articles within each improvement focus category. It is shown that 45 articles (44%) focus on individuals, while 32 articles (31%) have a principal focus on projects. Additionally, 17 articles (17%) and 9 articles (9%) focus on the AEC industry and organization levels, respectively.

Fig. 5
figure 5

Number of articles by improvement level

Full size image

6.3 Industry Sector

AR and MR technologies can facilitate various project types in the construction industry. As indicated in Table 2, the categories comprise Municipal/infrastructure, Residential, Building/commercial, Heavy/highway, and Industrial. Figure 6 illustrates the number of articles within each industry-type category. As shown, 37 articles have a principal focus on the building/commercial sector of the construction industry using AR/MR technologies. Residential, industrial, heavy/highway, and municipal/infrastructure have 18 articles (17%), 12 articles (12%), 3 articles (3%), and 2 articles (2%), respectively. Twenty articles (19%) cover multiple categories, and 11 (11%) articles were not applicable in any category.

Fig. 6
figure 6

Number of articles by industry sector

Full size image

6.4 Target Audience

An extensive range of target audiences benefits from AR and MR technologies. These target audiences are divided into eight categories, including (1) design team (e.g., architects, interior and exterior designers), (2) project manager (e.g., schedule and budget professionals, site manager), (3) worker/technician (e.g., machine operators and technicians, assembly operators), (4) inspector (e.g., project safety officers, facility manager), (5) project end-user (e.g., building occupants, office employees), (6) building systems engineer (e.g., structural, mechanical, and electrical engineers), (7) students/researchers (e.g., engineering students, researchers), and (8) other stakeholders (e.g., clients, building owners). Since each article may refer to more than one category, the percentage of articles in each category is reported. For instance, for the articles in which AR and MR technologies proposed a change in the work of three of these audiences, each audience category counted as one-third of a sole category. Then, the total sum of numbers is reported as a percentage for the contribution of each category. Figure 7 depicts the percentage of articles in each category type of target audience. As shown, workers/technicians with 29% of the articles are the dominant target audiences in AR and MR articles.

Fig. 7
figure 7

Percentage of articles by target audience

Full size image

6.5 Project Phase

Each project consists of various steps and phases throughout its whole lifecycle. As mentioned in Table 2, these phases start from (1) initiation and outline design, (2) design development, (3) procurement, contract, and pre-construction, (4) construction, and finally ends in (5) maintenance and operation. Figure 8 shows the number of articles by project phase. As shown, 39 articles (38%) mainly focus on the construction phase, and 18 articles (17%) primarily focus on the operation and maintenance phase of the projects. Figure 9 illustrates the number of articles for each project phase by year of publication. This diagram excludes articles that focus on multiple phases (which reduced the number of articles to 69).

Fig. 8
figure 8

Number of articles by project phase

Full size image
Fig. 9
figure 9

Distribution of articles by year and project phase

Full size image

Both Figs. 8 and 9 show that construction and operation/maintenance are the dominant categories among the articles published from 2013 to 2021. Moreover, as shown in Fig. 9, the highest number of articles occurred in 2021, with 15 articles. Additionally, a growing trend in articles focusing on the construction, operation, and maintenance phases can be interpreted.

6.6 Stage of Technology Maturity

AR and MR technologies are leveraged in different maturity levels, based on what is described in Table 2, and they include five categories. Figure 10 presents the number of articles within each stage-of-technology-maturity category. In this perspective, only two articles focus on the theory of AR/MR technologies. In comparison, the most significant number of articles focus on the system application (55 articles or 53%) and system development (30 articles or 29%) in this dimension.

Fig. 10
figure 10

Number of articles by stage of technology maturity

Full size image

6.7 Application Area

There are various application areas for augmented and mixed reality technologies in the AEC industry. As indicated in Table 2, this dimension is classified as simulation/visualization, communication/collaboration, information modeling, information access/evaluation, progress monitoring, education/training, and safety/inspection. Figure 11 illustrates the percentage of articles considered in each category from the application area perspective. Due to the overlapping nature of application areas, some articles were applicable in several categories. Therefore, the classification results are not presented by their numbers but based on the percentage of articles identified in each category. As shown in Fig. 11, about 22% of the articles focus on simulation/visualization as the primary application area of AR and MR technologies, while 16% of them mainly focus on the progress monitoring area. Information modeling shows the least focus (8%) of AR and MR articles from the application area perspective.

Fig. 11
figure 11

Percentage of articles by application area

Full size image

6.8 Comparison Role

Construction practitioners use different modes of comparison in implementing the AR/MR technologies, and as mentioned in Table 2, they are divided into three categories of Model vs. Model, Model vs. Reality, and Reality vs. Reality. Each comparison mode pursues a comparison purpose using the AR and MR technologies, including: (a) progress monitoring, (b) defect detection, (c) validating the model, (d) updating the model, and evaluating the model. Of 103 articles, 58 articles carry out a comparison practice in their work, about 56% of all. The rest do not show any interest in comparison modes and purposes of implementing AR and MR. Of these 58 articles, 29 articles (50%) focus on comparing model versus model, primarily to evaluate the model (28 articles or 48%). Comparing model versus reality (27 articles or 47%) with the purpose of progress monitoring (16 articles or 28%) is another primary focus of AR and MR articles in this category. Figures 12 and 13 depict the percentage of these 58 articles in both comparison modes and purposes.

Fig. 12
figure 12

Percentage of articles based on comparison modes

Full size image
Fig. 13
figure 13

Percentage of articles based on comparison purpose

Full size image

6.9 Technology Type

From a technology perspective, the articles can be classified into the following categories: (1) User experience, (2) Device, and (3) Delivery.

  1. 1.

    User: (a) Immersive and (b) Non-immersive. Immersive environments allow participants to feel as though they are inside the environment. Examples include HMDs, data gloves, and AR glasses. In contrast, non-immersive environments only allow participants to see the contents based on how the device in use—PC, smartphone, or tablet—is held and moved [27]. From a user perspective, 47 articles (46%) show a principal focus on non-immersive technologies, while 39 articles (38%) have a primary focus on immersive technologies. Additionally, 17 articles (17%) were not applicable in this category. However, as depicted in Fig. 14, the growing trend (from zero articles in 2014 to 12 articles in 2021) among the articles is using immersive forms of AR and MR technologies.

    Fig. 14
    figure 14

    Immersive and non-immersive AR technology by year

    Full size image
  2. 2.

    Device: (a) mobile and (b) non-mobile. With mobile augmented reality, the hardware required to perform an AR application is something that you can take with you wherever you go [7], while non-mobile augmented reality uses an ordinary desktop PC equipped with a Webcam (desktop AR) where the fusion between real-world and its digital augmentation is displayed on the computer screen [2]. Figure 15 illustrates the number of articles focusing primarily on mobile and non-mobile augmented reality. A total of 74 articles (72%) have a primary focus on mobile AR technologies, while only 16 articles (16%) focus on non-mobile AR. Moreover, as shown in Fig. 15, a growing trend among the articles focusing on mobile augmented reality is recognized (from 2 articles in 2015 to 19 articles in 2021). Thirteen articles (13%) were not applicable in these categories.

    Fig. 15
    figure 15

    Number of articles by mobile/non-mobile AR technology and by year

    Full size image
  3. 3.

    Delivery: In terms of delivery perspective, there are several configurations/architectures used for augmented reality applications: (a) application run on handheld systems such as smartphones, (b) application run on handheld systems connected to remote server(s), (c) application run on desktop/laptop computers, (d) application run on desktop/laptop computers connected to remote server(s), (e) application run as a web application, and (f) application run on a cloud with a thin client [6]. Additionally, there are some examples of applications run on lightweight wearable devices such as Microsoft HoloLens 2 (a self-contained computer with Wi-Fi connectivity) [15] or Google Glass which is a small, lightweight wearable computer with a transparent display for hands-free work [9]. Accordingly, in this section, the subcategory of (g) self-contained headsets is also defined for classifying articles. From the delivery point of view, 26 articles (27%) have a primary focus on self-contained headsets, while only three articles (3%) focus on cloud-based augmented reality. Figure 16 illustrates a distribution of articles by year and delivery perspective. As shown, the largest number of articles in a single year belong to the self-contained headsets’ category in 2021. Moreover, since there are always combinations of local and remote systems of AR and MR applications, 18 articles (18%) are considered in this category, while six articles were not applicable from the delivery perspective.

    Fig. 16
    figure 16

    Distribution of articles by the year and by delivery perspective

    Full size image
  4. 4.

    Spatial Registration: One of the most important requirements for AR systems is tracking. Visual tracking attempts to calculate the trajectory of an object in the image plane as it moves around a scene through features detected in a video stream. There are two primary tracking systems in AR implementations: (1) marker-based and (2) marker-less. Marker-based AR relies on placing fiducial markers (such as barcodes, QR codes, to name a few) in the real world, which is captured by a camera, thus creating an AR experience. In contrast, markerless AR does not depend on fiducial markers; however, the systems rely on natural features to execute tracking [1]. The articles are also classified based on their tracking system into two categories: marker-based and markerless systems. Figure 17 shows the number of articles by these two categories. Forty-three articles (42%) focus on marker-based tracking methods in AR and MR, while 38 articles (37%) focus on markerless methods. Two articles used both methods (mainly for comparison purposes), and 20 articles were not applicable in these categories. As shown in Fig. 17, there is a rising trend in using markerless AR and MR technologies among the articles.

    Fig. 17
    figure 17

    Distribution of articles by spatial registration system in AR and MR technologies and by year

    Full size image

6.10 Location

Augmented reality technologies can be implemented in different locations during a construction project. The categories include (a) Field: (a-1) Field-Indoor: e.g., building facility maintenance [13], manufacturing plants for off-site construction [26], (a-2) Field-Outdoor: e.g., urban area infrastructure, buildings construction sites, and (b) Home-office.

Figure 18 shows the number of articles by location and by year. It can be inferred that 39 articles focus on implementing AR and MR in on-site outdoor environments. In comparison, 27 and 18 articles focus on implementing AR and MR for on-site indoor and home-office purposes, respectively. Moreover, a rising trend among the articles (from one article in 2016 to 7 articles in 2020) is recognized in using AR and MR for on-site indoor implementations. The increasing trend also appears in the field-outdoor locations (i.e., from one article in 2016 to 16 articles in 2021).

Fig. 18
figure 18

Number of articles by location and by year

Full size image

7 Discussion: Current and Future Trends

In recent years, the rapid development of AR and MR technologies revealed the significance of implementing those technologies in construction. Consequently, a higher number of construction practitioners are inclined to leverage the capabilities of AR and MR and will encourage their passion for research, development, and investment in this area. Based on the classification of articles in previous sections, some of the current and future trends of AR and MR in construction are discussed. Tables 3 and 4 summarize the current and future trends, respectively. Some categories are indicated as dominant categories among the articles based on the total number of articles in each category. In contrast, in other categories such as target audience and application area, since most articles target multiple audiences and application areas, they are reported based on the percentage. From the results shown in Table 3, it can be inferred that Automation in Construction published most AR and MR articles. The articles used experimental/empirical research methodology focusing on the individual level. Most articles focused on the building/commercial sector of the industry with an emphasis on improving the work of workers/technicians during the construction phase. The principal stage of technology maturity was leveraging the application of AR and MR systems rather than the theory, the framework, or the sub-system technical issues. The comparison approach appeared to be mainly to compare model versus model to evaluate the model. The major application area that the articles focused on is simulation/visualization, which can aid workers/technicians, project managers, and inspectors to visualize the 3D models for progress monitoring and inspection purposes during construction and maintenance phases.

Table 4 Categories with expected future trend of AR and MR articles in the construction industry
Full size table

Although the total number of articles used AR and MR in non-immersive environments, there is a growing trend in using AR and MR in immersive environments such as MS HoloLens and Google Glasses. The articles’ current and growing trends are both on mobile devices such as tablets, smartphones, and head-mounted devices (HMD), as indicated in Tables 3 and 4. Moreover, the articles’ primary focus and growing trend show that most researchers and industry experts tend to use self-contained headsets to efficiently run the AR and MR on the device itself without requiring any other devices. Additionally, in terms of the location, most of the articles used AR and MR technologies in outdoor spaces such as construction sites, with a clear rising trend for outdoor areas as well.

8 Conclusion and Future Works

In this paper, 103 articles are selected from credited journals in the AEC industry and classified with a comprehensive set of dimensions and categories. The aim is to indicate the current focus of articles in using AR and MR technologies and identify the categories and areas with growing trends to pinpoint the potential areas that need more research and investment. As described in Sect. 7, a few areas need more focus in research and development, which are explained below.

Although worker/technicians are the dominant target audience in AR articles (29% of the articles), little research has been done with a focus on education/training of the workers (of 24 articles with emphasis on improving the work of workers/technicians 7 articles focus on training workers). Therefore, more research needs to be done on training workers and technicians and to develop improved AR-based work instruction to help novice workers quickly become familiar with the correct construction and assembly steps. Little focus is on using AR and MR technology in the procurement phase of the projects (10 articles, 3 of which are in the year 2013). Therefore, more research needs to be done on demonstrating the capabilities of AR technology in procurement management plans, such as quality management plans, which are a crucial part of construction projects and prefabrication plants for construction elements. Little work has been done on information access/evaluation area among the articles (13% of the articles), from which only one article focuses on inspection application and one on progress monitoring. Therefore, more research needs to be done on leveraging mobile AR capabilities and integrating them with BIM to help inspectors retrieve inspection data, checklists, and inspection lots during inspection tasks. The growing trend is toward using self-contained headsets (11 articles only in 2021). However, little work is done comparing the impacts of each delivery type (User Interface) of AR on the workers/technicians’ cognitive behavior and task performance during assembly and construction tasks. From the Stage of Technology Maturity Perspective, little work is done with a primary focus on Subsystem Technical Issues (8% of the articles). However, spatial registration accuracy and occlusions are still among the common technical issues in overlaying the AR content on real objects. Therefore, more emphasis on this category needs to be done in future research.