Keywords

1 Construction Hazard

Construction sites have been considered as some of the most hazardous venues. Many previous researches have studied the causes of construction accidents and various safety measures. One company in Hong Kong aspires to improve safety via the utilisation of virtual reality [10, 11]. While most of these studies have highlighted the human aspect, we propose to utilise the fast R-CNN object detection method to detect the construction hazard on sites and employ mixed reality to enable the artificial intelligence to detect the hazard.

2 Object Detection—Related Works

Object detection has been prominently adopted within image recognition processes and computer vision tasks such as pose estimation [6, 8], vehicle detection [14, 25], surveillance [7, 13], face detection, face recognition [12] and pedestrian detection [23, 26]. In object detection, an algorithm is required to solve two basic problems which are image classification and localisation.

This paper attempts to construct an object detection model to identify hazardous objects which may exist in construction sites. The photograph database we have built recognises and locates similar objects such as holes and wires which can then be displayed in an MR environment projected on HoloLens. Once the database sets up, it can establish the prediction API for training iteration (Figs. 1, 2, 3, 4 and 5).

Fig. 1
figure 1

Sample object input of the ‘hole’ figures

Full size image
Fig. 2
figure 2

Sample object input of electrical wire figures

Full size image
Fig. 3
figure 3

Test for hole 1

Full size image
Fig. 4
figure 4

Test for hole 2

Full size image
Fig. 5
figure 5

Test for wire

Full size image

Mixed Reality

In recent years, technological breakthroughs in mixed reality (MR) have allowed us to apply such holograms in various ways. Using the novel techniques in MR, real-life situations and computer-generated visual information are combined to create a hybrid expression which enhances the technological developments and practices in the architecture, engineering and construction (AEC) industries [1, 19].

Basically, MR consists of a variety of realms from real reality to virtual reality as precisely shown in Fig. 6. According to Milgram and Colquhoun [15], the ‘real’ and ‘virtual’ environments do not imply merely conceptual alternatives; rather, they are poles lying at opposite ends of a reality–virtuality (RV) continuum. Essentially, within the RV continuum, this is where the MR technologies take place [20]. We can use the mixed reality to see the real hazard but with a virtual label, say, for example, ‘hole’.

Fig. 6
figure 6

Order of reality concepts ranging from reality to virtuality [15, 20]

Full size image

Although researches in MR have been carried out continually for several decades, the application of MR technologies is becoming mature in practical use [19]. This includes various degrees of applications such as the collaborative web space [2], scientific visualisation [18], the augmented reality (AR) system [2, 3, 5, 21, 22], displays on unmanned air vehicles [16] and 3D video-conferencing systems [17]. More recently, incredibly specific topics in MR-based technological developments have been initiated by global IT leaders such as Microsoft.

In the construction sector, the AR system has widely been adopted in practices [4, 9, 24]. AR-based devices such as head-mounted displays (HMDs) and portable tracking systems are used in different situations. Meanwhile, this paper aligns its concern specifically on the object detection embedded in MR environments, which can provide extraordinary enhancements to construct safety training and management. After successfully detecting a hazardous object, we propose to utilise the mixed reality hologram for construction workers to detect the existence of hazard or safety training, which can then allow them to have an objective criteria on whether or not that is a risk, and enhance safety on sites.

3 Conclusion

It can be advantageous to apply MR technologies into construction practices as such technologies entail a wide range of benefits ranging from efficiency enhancement to safety improvement. This paper highlights the critical architecture in object detection in the fast R-CNN method; hence, with the assistance of Azure, it can adapt the pre-structured scripts to save time in code development. At the same time, it performs effectively with a sufficient database installed and accurate training tests. The custom vision tests 1–3 demonstrate that it is capable of training and recognising the hazardous objects and their locations which may exist in construction sites. If we go one step forward, we can deploy it to HoloLens and consequently identify the hazard on sites.