Keywords

23.1 Introduction

A large portion of infrastructure in America has exceeded or will exceed its intended design life in the following years [1]. For example, New Mexico is the home of many 100-year-old fracture-critical steel railroad bridges [2]. These bridges present the possibility of failure without regular and detailed inspections [2].

Although human-based inspections are largely adopted, a number of factors are negatively affecting the results obtained with this approach. Variations in training and experience, as well the information inspectors choose to report after an inspection may cause inaccuracies and lack of consistency when the same infrastructure is inspected by different teams [3, 4].

In the field of engineering, there is a push to embrace new technology to help engineers produce the best work possible [5]. Augmented reality, a technology which allows the user to see computer generated images superimposed on the real world, has the potential to minimize inspection inconsistencies. The augmented reality device used in this paper is the Microsoft HoloLens [6]. Augmented reality differs from virtual reality by allowing the user to be able to see the real world with overlaid holographic images (holograms).

This paper will explore augmented reality capabilities in order to create a tool for more consistent inspections. Using the capabilities of the HoloLens, a model of a structure can be created. Inspection data and measurements made by the HoloLens can then be combined with the model. This model can be used by future inspectors, improving consistency of inspections over time. The experimental procedure and results obtained with the current work are presented in the next sections.

23.2 Experimental Procedures

In the field of infrastructure inspection there is a need for improved measurement tools and a standardized system for reporting results. The following apps were developed to address the current needs of the inspection community. The Microsoft HoloLens was used as the augmented reality headset. All the apps presented in the current section were created with the help of the game development platform Unity and Microsoft Visual Studio. The apps created are categorized as either measurement apps or data storage apps. The measurement apps were created for the purpose of improving accuracy and efficiency of inspection measurement, as well as improving measurement reporting. Data storage apps were developed to store information about a structure which can then be reviewed, shared between inspectors, and used in additional data analysis.

23.2.1 Measurement Apps

23.2.1.1 Surface Area App

The surface area measuring app was designed to calculate the surface area of specific regions of sidewalks. The types of surface areas the app would need to measure are horizontal and vertical surface areas, to measure flat sidewalks and curbs. To accomplish these tasks, the app needs to be capable of measuring irregular shapes, such as multi-sided polygons. In order to address these needs, two algorithms were developed to measure horizontal and vertical surface areas. The user can define these surface areas through the placement of holograms to define the edges of the surface areas. The remaining of the current subsection provides details about the implementation of the horizontal and vertical surface area measurement algorithms, as well as a complete description of the user feedback for hologram placement and the output files generated by the app. Limitations of the current surface area measurements algorithms are highlighted.

23.2.1.1.1 Horizontal Surface Area Measurement Algorithm

The polygon surface area algorithm by Finley was adapted into C# code to measure the surface area of horizontal polygons [7]. Using the x- and z-coordinates of user-placed holographic markers, the horizontal surface area of the defined polygon can be calculated [7]. Figure 23.1 shows an example of the type of polygon area that can be calculated. X- and z-coordinates were chosen because the HoloLens defines the x-z plane as a horizontal plane. This approach would meet the need to measure horizontal surface areas with irregular shapes. Limitations of this algorithm are that the polygon cannot cross itself, as seen in Fig. 23.1. Also it should be noted that the user-defined surface area is projected onto the x-z plane, ignoring any vertical gradient present. This projection introduces some error into the measurement.

Fig. 23.1
figure 1

Figure (a) is an example of a polygon where the algorithm can calculate the area [7]. Figures (b–d) are examples of polygons the horizontal algorithm is not able to calculate [7]

Full size image
23.2.1.1.2 Vertical Surface Area Measurement Algorithm

The horizontal polygon algorithm could not be adapted to a vertical algorithm because it takes coordinate points from 2 planes, so a vertical surface area, such as a curb, would have to be projected on either the x-y or z-y plane. If an angle existed between the curb and the projected plane, significant surface area could be lost. This is illustrated in Fig. 23.2. The vertical algorithm was developed to limit vertical surface area projection error.

Fig. 23.2
figure 2

An example of how vertical surface area could be lost to projection error

Full size image

The user-placed vertical surface markers consist of “top” and “bottom” markers, which define the top edge and the bottom edge of the vertical surface to be measured. The horizontal distance between top markers is found using the distance formula and the x- and z-components of the vector between two markers, as in Eq. 23.1.

$$ d=\sqrt{{\left({x}_2-{x}_1\right)}^2+{\left({z}_2-{z}_1\right)}^2} $$
(23.1)

Similarly, the distance between the bottom markers is computed. The vertical distance between the top and bottom markers is found by taking the y-component of the vector connecting them. With the markers placed as shown in Fig. 23.3, the area of four rectangles defined by the top and bottom markers is averaged to get a final area.

Fig. 23.3
figure 3

An example of 2 top and 2 bottom markers placed to define a vertical surface area. Also shown are the four areas calculated to get a final average surface area

Full size image
$$ Area\ 1={X}_1\ast {Z}_1 $$
(23.2)
$$ Area\ 2={X}_2\ast {Z}_1 $$
(23.3)
$$ Area\ 3={X}_1\ast {Z}_2 $$
(23.4)
$$ Area\ 4={X}_2\ast {Z}_2 $$
(23.5)
$$ Final\ Area=\frac{Area\ 1+ Area\ 2+ Area\ 3+ Area\ 4}{4} $$
(23.6)

The surface area measurement algorithm still contains some inherent error because the height of the area is found using only the vector y-component; angles between the surface area and the y-axis are ignored. Also, the algorithm assumes that the user-defined area is a rectangle. This limits the complexity of vertical surface area shapes that can be measured.

23.2.1.1.3 User Feedback and Output Files

Three types of markers were created for the surface area app. A yellow cube with side lengths of 3 cm is the marker for defining horizontal surface area. A yellow sphere with a diameter of 3 cm is the marker for defining the top edge of a vertical surface, and a blue sphere with a 3 cm diameter is the marker used for defining the bottom edge of a vertical surface. Visual feedback from both the shape and the color of the marker helps the user keep track of the type of markers he/she has already placed. When calculating surface area, the markers are connected with a red line to provide the user with feedback on the total surface area the app is measuring. The surface area calculated is also instantly displayed in a textbox on screen for the user to view. An example of what the user would see is shown in Fig. 23.4.

Fig. 23.4
figure 4

An example of what the user would see when defining a horizontal surface area using the Surface Area App

Full size image

Text files with date and time information, marker coordinate positions, and the calculated surface areas are generated automatically for the user. These files allow the user to engage in extra data processing if desired.

23.2.1.2 Depth App

A depth measuring app was developed as a tool for evaluating structural health. The depth of a crack or pothole is one indicator of structural deterioration and damage progression. To properly monitor structural health, damage must be periodically quantified. The depth app aims at allowing the user to efficiently measure and record the magnitude of damage to a structure. Details about the depth measurement algorithm, user feedback and output files are provided hereafter.

23.2.1.2.1 Depth Measurement Algorithm

Depth is calculated using 4 user-defined points. The first three markers are placed in order to establish a reference plane for the depth measurement. The last point is placed at the depth that the user wants to measure. The shortest vector is then calculated between the depth marker and the defined plane. The magnitude of that vector is reported to the user as the depth. Figure 23.5 illustrates the three steps taken by the depth measurement algorithm and a screenshot of the scene viewed by the user.

Fig. 23.5
figure 5

An illustration of how depth is calculated along with what the user would see when the app is in use

Full size image
23.2.1.2.2 User Feedback and Output Files

Once four markers have been placed the depth will be calculated. The depth is instantly displayed in a textbox on screen for the user to view. Similar to the Surface Area App, text files with date and time information, marker coordinate positions, and the calculated depth are generated automatically for the user. These files allow the user to engage in extra data processing if desired.

23.2.1.3 Grade App

Grade is calculated as the amount of vertical height change over a horizontal distance, which can be calculated as the slope of a line. This is found by creating a ratio between the horizontal and vertical change between set points. The purpose of this app is to allow the user to calculate the grade of a sidewalk to determine if it meets building code. Details about the grade app are provided in the next two subsections.

23.2.1.3.1 Grade Algorithm

The grade algorithm works by finding the vector that connects two user-defined points. The slope of the vector is calculated by using the y-component of the vector as the “rise” of the slope. The “run” is calculated by using the x- and z-components of the vector and the distance formula from Eq. 23.1. Grade is then found by dividing the “rise” by the “run”. Since grade is reported as a percentage, the rise over run is then multiplied by 100%. Figure 23.6 illustrates how grade is calculated as well as what the user sees when using the app.

Fig. 23.6
figure 6

An illustration of how grade is calculated along with what the user would see when the app is in use

Full size image
23.2.1.3.2 User Feedback and Output Files

Once two markers have been placed, a grade can then be calculated. The grade is instantly displayed in a textbox on screen for the user to view. Text files with date and time information, marker coordinate positions, and the calculated grade are generated automatically for the user. These files allow the user to engage in extra data processing if desired.

23.2.2 Data Storage Apps

23.2.2.1 Accelerometer Location App

The accelerometer location app is meant to be used as a tool for documenting accelerometer location and orientation on a structure for the purpose of modal analysis. The app enables the user to place a holographic marker onto a structure to permanently record the location and orientation of an accelerometer at its installation. User feedback and output files details follow.

23.2.2.1.1 User Feedback and Output Files

The marker created consists of three cylinders which are aligned to form axes as seen in Fig. 23.7. These cylinders are also color coordinated to correspond to the axes defined by Unity. When a marker is placed, the y-axis is automatically aligned to be normal to the surface that it is placed on. Once the user has placed all markers at the desired locations and orientations of each accelerometer on the structure, a text file is created. This text file contains individual marker locations in local coordinates created by the HoloLens. The text file also outputs the orientation vectors for each marker based upon the respective orientation of their axes.

Fig. 23.7
figure 7

Accelerometer App markers placed on a barrel with accelerometers attached

Full size image

23.3 Future Work

After creating the apps previously mentioned, the team found several areas of structural inspection where augmented reality could also be of use. The two main areas of future potential were found to be modal analysis, particularly accelerometer placement, and file storage of inspection results. With regard to the first point, a model of the structure could be created on the computer, and accelerometers could be placed on the model. The model would then be projected on the real structure using augmented reality, showing the user where and in what orientation the accelerometers should be placed. With regard to the second point, the HoloLens can be programmed to generate world anchor holograms. A world anchor is a type of hologram specific to the HoloLens that can be placed by the user, and, once placed, this hologram remains permanently at this defined location. If the user reopens the app at a later time, the world anchor would appear in the same location where it was saved. As a result of its ability to have a permanent location, world anchors can be used as a tool for information storage.

23.4 Conclusions

Augmented reality has the potential to help structural inspections meet the needs of a constantly growing and developing modern world dependent on its infrastructure. The research presented shows how the game development platform Unity and Microsoft Visual Studio can be successfully used to develop a set of apps to enable infrastructure inspections with a Microsoft HoloLens augmented reality headset.

Current inspections are subject to variability, and are in need of a standardized system for reporting inspection results. An answer to this need can be provided with augmented reality apps created to perform inspection measurements, such as measuring surface area and grade, as well as to store information pertaining to a structure, such as accelerometer placement.

Aside from the apps already created, there is potential for the development of future apps to further explore the measurement and data storage capabilities of augmented reality.