Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Structure health monitoring (SHM) is defined as a process to detect and characterize the damages occurring on an artificial structure, and it becomes an important issue to sustainable structure managements. Traditionally, the damages occurring on a structure can be identified by employing destructive and nondestructive testing methods; usually, nondestructive testing methods are preferred because they can identify the damages without causing extra damages on a structure. There are many nondestructive testing (NDT) methods have widely been employed to evaluate the aged buildings because of their reliability and effectiveness. NDT methods can not only permanently alter the inspected objects but also provide the defect information. In general, NDT methods can be classified into Visual Inspection, Proof Load Test, Vibration Testing, Impact Testing, Ultrasonic NDT, Conductivity and Radar (McCann and Forde 2001). The visual inspection mentioned above is the most economical way to detect the damages. However, for a large structure (like wind turbines), visual inspection is hard to apply to locate the damages. Recently, Unmanned Aerial Vehicle (UAV) is employed to examine the structures instead of applying the visual inspection because the examined structures are usually huge. Phung et al. employed the camera installed on UAV to develop an optimal flying path to examine the surface conditions of buildings and bridges (Phung et al. 2017); Moss et al. demonstrated that structural health monitoring systems could be based on a UAV system (Moss et al. 2017). Terrestrial laser scanning (TLS) is an innovate technology to collect the spatial information of a structure, and it is based on the transmission and receiving of pulsed light (Chen 2012). Yang et al. employed TLS to collect the deformation information of concrete composite structures (Yang et al. 2017); Liu applied the TLS to monitor the deformation behaviors of bridges (Liu 2010). Both systems offer huge spatial data such that the structure health conditions can be monitored.

Infrared thermography is the product of remote sensing techniques and is usually used to record the surface temperature of the observed objects. Infrared thermography is widely used to identify defects presented in objects by observing the radiant heat pattern (Huang and Wu 2010). For a thermal camera, radiation is converted to temperature depending on the emissivity value of the specimen. Emissivity values are in the range of 0 and 1, and they are defined as the ratio of a body’s emission spectrum at a given temperature to that of a blackbody at the same temperature (Plotnikov and Winfree 1998). If the emissivity value is 1, it means that the test body is a black body; if it is 0, the test body completely reflects all the received energy. Infrared thermal cameras are designed to record the temperatures transformed from emissivity values. Changes in the recorded temperatures on thermal images reveal possible surface flaws (Maldague 2001). Furthermore, those areas in which the surface temperature information is different with their surrounding neighborhoods can be used to indicate the locations of the potential defects presented on objects.

The paper integrates the spatial information collected by employing both UAV and TLS and surface temperature information recorded by a thermal infrared camera to identify the defect information of structures. The spatial information of a structure is too abundant to handle. Hence, a 3D model is built to simplify the spatial information collected from UAV and TLS. Then, the thermal images are analyzed by segmenting each thermal image into a series of sub-regions such that the surface temperature distribution in each segmented region is homogeneous. In doing so, the surface temperature information in each segmented region is illustrated such that the temperature differences can be illustrated and used to identify the defect locations. Eventually, the segmented regions are treated as texture information and attached on the 3D model such that the textured model provides defect information.

A wind turbine located at Taichung County Gaomei Wetlands is used as an example. The remainder of this paper is organized as follows. In the next section, the ways to build the 3D model of the wind turbine is introduced, and multilayer level set model used to analyze the thermal images are briefly introduced. In Sect. 3 illustrates the 3D model and the segmented results by employing the multilayer level set approach on real thermal infrared images. Eventually, some conclusions and related discussions are provided.

2 Hyper Approach by Integrating Different Point Clouds and Thermography

Structural health monitoring systems usually adopt a system to monitor a structure. Both UAV and TLS used in SHM can offer spatial information of the monitored structures. However, UAV provides better spatial information illustrated on the top of the monitored structures; oppositely, TLS offers better spatial information shown in the structure except for the top because the top information is blocked. Hence, the paper employed the point clouds from UAV and TLS to build the 3D model of the structure. The way to generate the 3D model is introduced. Then, thermal infrared cameras were employed to record the surface temperature information. The collected images were analyzed by applying the multilayer segmentation such the recorded surface temperature information can be grouped into few groups such that the surface temperature information in each group can be replaced by the average temperature of the grouped region. The multilayer segmented approach is briefly introduced.

2.1 3D Model Generated from Different Point Clouds

Point clouds are the products by using UAV or TLS to collect the spatial data of the monitored structure. Both of them will have their coordinate systems such that the collected data from two systems cannot be directly combined. In this paper, ground control points were established such that the collected data by UAV or TLS could be projected to the same coordinate system. The projection between the collected and controlled points is assumed that the projection can be explained by a polynomial function. The corresponding points of the ground control points are chosen from the point clouds; the optimal locations are determined by weighting the possible points and their neighborhood from the given point clouds. In doing so, two point clouds from UAV and TLS are combined. The combined point cloud is imported and transformed into the vector format to build the 3D model of the monitored structure.

2.2 Analyzing Thermography by Applying the Multilayer Segmentation

The fundamental idea of applying image segmentation on thermography is to partition a given thermal infrared image into a series of sub-regions such that each sub-region is homogeneous. In doing so, the surface temperature distributions can be identified and located in the thermal infrared image. Multilayer segmentation proposed by Chung and Vese segments the given thermal infrared image according to the pre-defined thresholds such that the pixel values shown in the segmented regions are grouped and centered by the pre-selected thresholds (Chung and Vese 2010). In the multilayer approach, let \( {\text{I}}_{0} \) be a thermal image and the two level set functions are used to segment the image into a series of sub-regions with distinct level values \( \left\{ {l_{1} < l_{2} < \cdots < l_{m} } \right\} \) and \( \left\{ {k_{1} < k_{2} < \cdots < k_{n} } \right\} \) for \( \phi_{1} \) and \( \phi_{2} \) the level set functions, respectively. Then the energy functions generated by level set functions for \( \phi_{1} \) and \( \phi_{2} \), pre-selected thresholds \( \left\{ {l_{1} < l_{2} < \cdots < l_{m} } \right\} \) and \( \left\{ {k_{1} < k_{2} < \cdots < k_{n} } \right\} \), and the regional constants c (the average values of the segmented regions) can be defined as follows (Chung and Vese 2010):

$$ \begin{aligned} (c,\varphi ) = & \sum\nolimits_{,i,j = 1}^{m - 1,n - 1} {\int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} - c_{i,j} } \right|^{2} } \chi_{1} } dx + \sum\nolimits_{i = 1}^{m - 1} {\int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{\text{i,0}} } \right|^{2} \chi_{2} dx + } } \sum\nolimits_{i = 1}^{m - 1} {\int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{\text{i,n}} } \right|^{2} \chi_{3} dx + } } \sum\nolimits_{j = 1}^{n - 1} {\int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{{ 0 , {\text{j}}}} } \right|^{2} \chi_{4} dx} } \\ & + \sum\nolimits_{j = 1}^{n - 1} {\int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{\text{m,j}} } \right|^{2} \chi_{5} dx + } } \int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{ 0 , 0} } \right|^{2} \chi_{6} dx + } \int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{{ 0 , {\text{n}}}} } \right|^{2} \chi_{7} dx + } \int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{\text{m,0}} } \right|^{2} \chi_{8} dx + \int\limits_{\varOmega } {\left| {{\text{I}}_{ 0} { - }c_{\text{m,n}} } \right|^{2} \chi_{9} dx} } \\ & + \mu \sum\nolimits_{i = 1}^{m} {\int\limits_{\varOmega } {\left| {\nabla H(\varphi_{1} - l_{i} )} \right|dx} } + \mu \sum\nolimits_{j = 1}^{n} {\int_{\varOmega } {\left| {\nabla H(\varphi_{2} (x) - k_{j} )} \right|dx} } \\ \end{aligned} $$
(1)

where H is the Heaviside function, \( \mu > 0 \) is a weight parameter, and \( c_{ij} \) is the sub-regional constant. The parameter \( \chi_{i} \) is the combinations of level set functions \( \phi_{1} \) and \( \phi_{2} \); for an example, \( \chi_{1} \) is defined as \( H\left( {\phi_{1} - l_{i} } \right)H\left( {l_{i + 1} - \phi_{1} } \right)H\left( {\phi_{2} - k_{j} } \right)H\left( {k_{j + 1} - \phi_{2} } \right) \). As for other parameters \( \chi_{i} \), those parameters can be referred the paper of Huang et al. (2014). The optimal approximation I derived from the segmented results can be shown as:

$$ \begin{aligned} {\text{I}} = & \sum\nolimits_{i = 1}^{m - 1} {\sum\nolimits_{j = 1}^{n - 1} {c_{ij} } } \chi_{1} + \sum\nolimits_{i = 1}^{m - 1} {c_{i0} \chi_{2} + } \sum\nolimits_{i = 1}^{m - 1} {c_{in} \chi_{3} + } \sum\nolimits_{j = 1}^{n - 1} {c_{0j} \chi_{4} } \\ & + \sum\nolimits_{j = 1}^{n - 1} {c_{mj} \chi_{5} + } c_{00} \chi_{6} + c_{0n} \chi_{7} + c_{m0} \chi_{8} + c_{mn} \chi_{9} \\ \end{aligned} $$
(2)

The level set functions \( \phi_{1} \) and \( \phi_{2} \) can be found by employing finite difference such that initial level set functions will change their shapes to meet the energy minimization steps by steps by using iteration scheme (Chung and Vese 2010; Huang and Wu 2010; Huang et al. 2014).

3 Processed Results by Employing the Proposed Approach

A wind turbine located at Taichung County Gaomei Wetlands was chosen because the structure is hard to approach. The spatial data collected by UAV is illustrated in Fig. 1. Similarly, TLS is used to collect the spatial data, and the results are illustrated in Fig. 2. The 3D model of the wind turbine is generated by employing the point clouds collected by applying UAV and TLS, and the 3D model is presented in Fig. 3. As for thermography, NEC Thermo Gear-G120 and FLIR VUE were used to record the surface temperature information of the wind turbine. Thermo Gear-G 120 was installed on the ground to measure the surface temperature information, and FLIR VUE was installed on UAV to collect a series of images. With the multilayer segmentation, the thermal images were segmented into few regions. The processed results are illustrated in Figs. 4. and 5, respectively.

Fig. 1.
figure 1

The collected spatial data of the wind turbine located at Taichung county Gaomei Wetlands

Full size image
Fig. 2.
figure 2

The point clouds of the wind turbine collected by employing TLS.

Full size image
Fig. 3.
figure 3

The generated 3D model of the wind turbine

Full size image
Fig. 4.
figure 4

The segmented results of applying the multilayer segmentation on the thermal image collected by FLIR.

Full size image
Fig. 5.
figure 5

The segmented results of applying the multilayer segmentation on the thermal image collected by Thermo Gear-G120.

Full size image

4 Conclusions

The proposed approach employs the 3D model generated by using point clouds from UAV and TLS to illustrate the spatial information of the wind turbine. Then, thermography is used to record the surface temperature information, and the multilayer segmentation is employed to analyze the given thermal images. The thermal camera FLIR VUE cannot directly collect the surface temperatures; hence, the surface temperature information is obtained by comparing the surface temperature information collected by Thermo G-120. The conclusions are summarized as follows:

  1. 1.

    3D model provides the spatial information of the monitored structure;

  2. 2.

    Thermography offers the surface temperature information of the monitored structure, and the differences of the analyzed surface temperatures are the important clues to identify the defects shown in the structure.