Keywords

1 Introduction

Bridges act as a significant role in linking one area to another, making it possible for the transportation network to expand and function. As a fundamental infrastructure of a nation, bridges are designed to carry pedestrians, road traffic, trains, or even canals for waterborne transport. Each day, thousands of cargos and people pass through the bridges; these activities create numerous economic values and keep people connected. Among the many types of bridges in the present world, beam bridges are the most general ones due to their cost-effectiveness. Cable-stayed bridges, suspension bridges, and arch bridges are also commonly built in many regions. Those bridges are mainly constructed with concrete, steel, and brick materials followed by modern standardizations, allowing them to be inspected systematically with the corresponding specifications.

Bridge inspection is pivotal in civil infrastructure maintenance, enabling engineers to discern trivial defects and potential problematic regions before they grow into irretrievable issues. Despite the difference in regulations and practices around the world, inspecting a bridge is mandated at least once every two years in most countries [1]. The inspection procedure is conventionally carried out by the bucket truck, which is also known as the “snooper truck” [2]. At its core, a man-carrying bucket is fitted at the end of the hydraulic pole on the truck, where the workers can complete their inspection jobs safely on the lifted bucket underneath the bridge. There are four mainstream bridge inspection methods: superficial, routine, principal, and special inspections [3]. Routine and principal inspections refer to periodic and detailed inspections, where special inspection is applied to investigate the specific bridge problems. During the inspection, inspectors often use four common techniques; they are visual, acoustic, thermal, and ground-penetrating radar [3].

In contrast to modern bridges, there were no consistent standards for the construction of ancient bridges. The earliest bridges were spans made of wooden planks or stones with simple supports and crossbeam arrangements. The origin of arch bridges can be traced back to the Roman Empire thousands of years ago, some of which are still standing today. In European countries, many stone arch bridges remained a crucial component of modern transportation networks. Brick and mortar bridges were later invented, and rope bridges were found to be used in South America in the 1500 s. During the eighteenth century, engineers designed timber bridges, followed by the development of steel bridges in the nineteenth century after the emergence of the Industrial Revolution. Up to now, the arch bridge is the one with the greatest number preserved by relatively intact structures amidst all ancient bridges. Two typical examples are the Pons Fabricius Bridge [4] in Italy and the Anji Bridge [5] in China, which were built in 62 B.C. and 605 A.D., respectively. Due to its distinctive design and prominent performance of resistance, the arch bridge is still considered one of the most popular solutions to span a river, valley, or gorge nowadays. However, unlike modern arch bridges built from reinforced steel and concrete, ancient arch bridges were prevailing constructed with natural materials, which limited their pliability. With repeated applied stresses and the effects of efflorescence, disintegration and cracking may occur over time between the mortar and the natural materials, leading to structural problems in ancient arch bridges. Because of its representativeness among ancient bridges, a masonry arch bridge was investigated as the research subject in this paper.

It is generally known that the longer the time, the greater the opportunity of the object being damaged, and this also applies to ancient bridges. Ancient bridges have accumulated more injuries than modern ones and undertook many circles of stresses to their surfaces and internal structures. Therefore, it is imminent to see that additional maintenance is added to prolong the lifespan of ancient bridges. Unfortunately, current specifications only focus on the inspection and maintenance of recently built bridges, while the maintenance of ancient bridges lacks explicit provisions. One reason is that variations in characteristics such as modality, material, age, and span make ancient bridges difficult to be classified. Another ground is that the use of bridges built earlier as main transport arteries is becoming less and less frequent, as the development of advanced transportation networks relied more on modern bridges in the present day. So far, many countries and regions have begun to pay more attention to the maintenance of ancient bridges, as many are still being used as the main traffic routes. However, the problem of lacking inspection remained unaddressed in the overwhelming number of ancient bridges around the world.

2 UAV-Based Ancient Bridge Inspection

2.1 Background of UAV-Based Inspection

Although traditional inspection tasks are performed by professional inspectors, it still faces the problems of low efficiency and high cost [6, 7]. One major challenge in the inspection task is the difficulty of accessing the corner and bottom of the bridge. Another challenge is the complex environment during the bridge inspection process. In some cases, the inspection work consumes considerable time to inspect all the details of the bridge. Safety is another concern, while falling from the bridge would be a fatal accident [7].

With the development of aerial robotics, UAVs (Unmanned Aerial Vehicles) offer bridge inspectors a solution to overcome the problems discussed above. UAVs, by another name, are drones and can be defined as a class of automated or remotely controlled aircrafts. Nowadays, most commercial drones are equipped with wireless transmission and camera systems. The operator can visualize and capture the images by the remote controller, which can protect the inspectors from danger. Due to the drone’s highly autonomous controller and agile flying ability, it can improve its working efficiency significantly [8]. The UAV can be divided into several types, including fixed wing, airship, helicopter, and multirotor [8, 9]. Researchers have tried different platforms for the inspection task. Fixed-wing drones are suitable for long-distance and large-scale surveys, but inspecting the side of constructions is strenuous. Meanwhile, airship and helicopter drones are inappropriate for conducting civil infrastructure inspections due to their massive size. Multirotor drones can take off and land vertically in a tiny space and hover stably in the air. With the assistance of gimbals, multirotor can take images from different angles. Benefiting from these advantages, the multirotor is more suitable for bridge inspection tasks.

Today, UAVs are used in civil engineering to solve various problems with high-definition cameras. Kumar et al. [10] employed the UAV to perform the damage detection task and achieved the automatic detection pipeline. Ellenberg et al. [11] developed a monitoring system for the bridge using UAV. Costa [12] and Álvares [6] applied the UAV in the working progress monitoring and safety checking. Predictably, the use of UAVs in the construction industry will increase in recent years with the innovation of technology; this trend can further promote the application of UAVs becoming more sophisticated in civil engineering fields, such as bridge inspection.

2.2 Significance of UAV-Based Inspection for Ancient Bridges

UAVs are flexible platforms that can inspect areas inaccessible to manual inspectors. In manual inspection, regions such as corners, bearing connections, and underneath bridge regions are difficult from being viewed by inspectors due to spatial constraints. The bridge inspection truck, also named the “snooper truck”, will be employed to inspect those hard-to-reach regions. For some ancient bridges, the primary issue is the difficulty of deploying the snooper truck to the work site. One reason is the concern of potential deficiencies caused by structural deterioration, where some ancient bridges may not be able to withstand the load of a truck; another is due to environmental and geometric limitations of the bridge. For example, it is challenging for a working platform on an inspection truck to pass under narrow or various ancient bridge structures. Accordingly, many ancient bridges are still not well inspected despite the massive development of today’s bridge inspection technologies.

Nowadays, it is a norm to apply non-destructive testing (NDT) [13] techniques for ancient bridge inspection and monitoring. NDT methods are prevalently adopted in civil engineering surveys because they do not introduce any damage to the structure of infrastructures. NDT methods for ancient bridge inspection applications include infrared thermography [14], sonic transmission [15], seismic reflection [16], ground-penetrating radar (GPR) [17], electrical impedance tomography [18], radiography [19], and laser scanning [20]. Different NDT approaches have their unique operational principles:

  • Infrared thermography discovers the subsurface defects based on the radiant energy captured by the specialized camera;

  • Sonic transmission and seismic reflection can investigate the internal flaws using sound waves;

  • Ground-penetrating radar detects the irregularities in the subsurface with radar pulses;

  • Electrical impedance tomography allows the formation of 3D tomographic images for particular body parts by measuring the surface electrical properties;

  • Radiography records the internal view of an object by penetrating it with electromagnetic radiation;

  • Laser scanning describes the object surface by generating its 3D point clouds with laser beams.

However, adopting the above NDT methods for ancient bridge inspection normally requires professional knowledge and skills. Their implementation is also accompanied by excessive time consumption for equipment manipulation and data analysis. Furthermore, their application in bridge inspection is significantly restricted by the surrounding environment and equipment size. Lastly, the limited interior bridge information obtained by most sensor-based technologies during each detection can hardly be utilized to assess the health of entire ancient bridge. Hence, employing a vision-based inspection of the ancient bridges is more straightforward and efficient than using the sensor-based NDT methods.

The UAV-based inspection provides an effective solution to the ancient bridge inspection as three major advantages brought by this robotic application: access to more informative data, faster inspection speed, and safer working environment. In many cases, inaccessible areas can be inspected by adjusting the angle of the mounted camera and the aerial position of the UAVs, which provides adequate information for a comprehensive assessment of bridge health. Concurrently, UAVs allow faster inspections than human inspectors due to their inherent high maneuverability and wider field of view. Most importantly, the entire inspection was carried out using the UAVs without imposing any load on the bridge. The deployment of UAVs to inspect ancient bridges does not cause any damage to the structure, which further ensures the safety of the inspection procedure. In general, due to the various benefits offered by UAV applications, it is possible to make UAV-based inspections a universal approach for most ancient bridge inspections.

2.3 Crack Segmentation

Today, implementing deep neural networks has become a trend in engineering applications because it can solve the problems of low efficiency and impracticality faced by traditional industries. In traditional bridge inspection, detecting defects is laborious, while using deep learning models can detect bridge defects automatically with higher efficiency and accuracy. In this research, the segmentation network U-Net [21] is employed, instead of using the object detection networks. Because the bounding box given by the object detection network will contain multiple objects in the same box, the operator may be confused when detecting defects using this method. For segmentation, a pixel-wise mask is generated for each object in the image, where irrelated issues will be excluded. Two popular architectures, VGG16 and ResNet_101, are used as transfer learning architectures to enhance the model performance in this research. U-Net, known as its “U”-shaped structure, is a typical segmentation model that can be simply viewed as two parts. The first part of U-Net is used for the feature extraction, while the second part is the feature convergence network. The output of the U-Net model has the same size as the input image with high resolution, and each pixel is classified into a certain class.

2.4 Evaluation Metrics

To evaluate the performance of the proposed models, the evaluation metric is adopting IoU (Intersection over Union) and Dice, where Dice is also called “F1 score”. For object detection methods, the typical evaluation metrics are precision and recall, which represent the correct prediction rate in positive samples and ground truth, respectively. It is inappropriate to use precision and recall in a segmentation problem because they cannot describe the classification of pixels. The IoU and Dice metrics are used in segmentation problems as they represent the overlapping between the predicted output and target mask. In general, the Dice coefficient often gives a higher score than the IoU. Below are the expressions of IoU and Dice:

$$IoU=\frac{TP}{TP+FN+FP}$$
(1)
$$Dice=\frac{2\times TP}{\left(TP+FN\right)+\left(TP+FP\right)}$$
(2)

In this paper, TP (True Positive) indicates the prediction of a defect on the ancient bridge is correct; TN (Ture Negative) indicates the prediction of a non-defect object is correct; FP (False Positive) represents the prediction of a defect on the ancient bridge is wrong; FN (False Negative) represents the prediction of a non-defect object is wrong.

2.5 3D Modelling of Ancient Bridges

A 3D model of the target bridge will be generated using the images captured by UAVs, and the digital bridge information can be stored in the database for bridge health monitoring in due course. After acquiring the bridge images from UAVs, the method of Structure from Motion (SfM) [22] will be employed to reconstruct the 3D model of ancient bridge. Using techniques such as Scale-Invariant Feature Transform (SIFT) or the Speeded-Up Robust Features (SURF), correspondences between images can be determined, and a 3D model of the bridge can be generated as a result. In SIFT, the points of interest of the objects are first detected and stored in the database. Then, features extracted from new images were used to compare with the key points in the database to find the matches based on Euclidean distance. A feature enhancement approach, Difference of Gaussians (DoG), is applied to obtain the features from the images. Knowing the match information and filtering out false matches, the object location, scale, and orientation can be computed. With sufficient matches and feature trajectories, the 3D positions of the object can be determined, along with the parameters for camera pose and calibration. Eventually, two-dimensional imagery information can be converted into three-dimensional form.

2.6 Ancient Bridge Maintenance

Based on the obtained detection results and the 3D reconstruction model, successive bridge maintenance strategies can be introduced as the defect information has been revealed. However, there is no standard specification for the maintenance of ancient bridges. In this paper, an evaluation criterion of health conditions for ancient bridges and the corresponding maintenance strategies are proposed for reference. First, the health condition of the ancient bridge can be classified into 1 to 5 levels. Level 1 indicates the bridge is in good condition, and so on; level 5 indicates that the bridge has critical problems and cannot be used. The criteria of the ancient bridge health condition assessment are listed in Table 1. The assessment is referenced to the modern bridge appraisal; the difference is that each ancient bridge has a different degree of deterioration. With the assessment of the bridge health level, the corresponding measurements can be determined. Table 2 shows the possible defects and the causes for these phenomena and provides solutions for each health level of the ancient bridges.

Table 1 Definition of health condition levels of ancient and modern bridges
Full size table
Table 2 Possible defects of ancient bridges with different health condition levels and corresponding maintenance measures
Full size table

Worth mentioning, all ancient bridges were not designed to carry the modern traffic volume, but using ancient bridges as the main transport networks is prevalent in some regions. For example, some ancient bridges were assigned traffic volumes including pedestrians, cars, and heavy trucks; some century-old bridges have been incorporated into the railway system and given the mission of rail transportation. Bridges in these conditions require particular cautiousness and extra attention, as structural deformation can easily occur when ancient bridges experience unexpected loads. In general, most ancient bridges are experiencing the slow effects of environmental problems such as sunlight and rain. It is inevitable to induce a gradual deterioration of bridge components and triggers many potential problems for the entire bridge structure.

3 Experiment

3.1 Site Condition

The experimental site was chosen the stone bridge located at Tai Tam Tuk Reservoir, Hong Kong. The Tai Tam Reservoir Stone Bridge was built in 1888 as an arch bridge with vehicular and pedestrian access. Since the bridge has constructed about 130 years, the surface has been covered with plants, mosses, and molds. Below the bridge is a reservoir, where the water level rises or falls with the seasons. Due to the water level and the limited arch space, performing traditional manual inspection for the Tai Tam Stone Bridge is not feasible. Moreover, applying other inspection technologies to the entire bridge, such as laser scanning, sonic tomography, or ground-penetrating radar is not applicable as there is no appropriate space for equipment installation and operation. Thus, conducting a UAV-based inspection is preferred in this experiment because UAVs can not only eliminate the effects caused by geometric constraints but also provide information about the bridge from different perspectives (Fig. 1).

Fig. 1
A photo of a stone bridge. It stands on the flowing water source underneath.

Tai Tam Stone Bridge

Full size image

3.2 Implementation of RTK

The DJI Phantom 4 RTK and D-RTK 2 station were used in the experiment to achieve better UAV positioning performance. RTK [23] is known as Real Time Kinematics, a technique for correcting GNSS (Global Navigation Satellite System) errors. The accuracy of RTK is measured in centimeters compared to the positioning accuracy of GPS (Global Positioning System), which is measured in meters. In general, the location of an object is computed by calculating the distance from satellites to the GNSS receiver. The distance is related to the speed of the travelling signal and is often accompanied by a delay caused by the Earth’s atmosphere. These delays will induce computational errors in signal propagation, so GNSS receivers cannot provide data with better than meter-level accuracy. In RTK, there are two receivers: one is the static station fixed at a specific location, and another is the rover responsible for data collection. In this paper, rover refers to the DJI Phantom 4 drone. In addition, the RTK station can be replaced by an NTRIP (Networked Transport of RTCM via Internet Protocol) service in certain scenarios. The GNSS errors can be eliminated when two receivers in close proximity because signals observed by the two receivers can be treated as identical. Furthermore, to achieve centimeter-level accuracy, RTK uses carrier phase measurement to resolve the integer ambiguity problem. Carrier phase measurement is an advanced technique for calculating the phase of the signal wavelength; the distance between the satellite and the GNSS receiver can be determined accurately. In the experiment, owing to the high real-time precision of RTK, the data acquisition is relatively safe and facilitates the post-processing of data. In addition, the 3D reconstruction accuracy using SfM is further promised (Fig. 2).

Fig. 2
A photo and an illustration. A, the photo of a ghat section-type location with a stone bridge. At the beginning of the bridge is a D-R T K 2 mobile station. B, the working principle illustrates R T K correction of G N S S data from the base station with interaction between R K T-u a v and satellite.

a D-RTK 2 mobile station b working principle of RTK

Full size image

3.3 Dataset and Operational System

In this paper, around 11,200 images from several public datasets [24,25,26,27,28,29] were used, and all images were resized to 448 \(\times \) 448. A total of 271 labeled images of the Tai Tam Stone Bridge taken by UAVs were utilized to test the proposed method. In practice, the model will have good performance if the image contains only cracks and a pure background. However, detecting cracks and other defects is intricate in reality as different objects will appear in the same image. To improve the robustness of the proposed method, the images involved in the training include the categories:

  • Pure cracks;

  • Cracks with moss;

  • Cracks with blocky surfaces;

  • Cracks in a large context.

The experiment is conducted in an environment of Intel Core i7-6700 CPU @ 3.4 GHz \(\times \) 8, and an NVIDIA GeForce GTX 1070 GPU. It is clear that the operating system used in this research is generic, which demonstrates the universality of the proposed approach, as its replication cost is affordable.

4 Result and Discussion

The results obtained from the two proposed models are shown in Table 3. The IoU and Dice coefficients of UNet_VGG16 are 0.4727 and 0.6012, while for UNet_ResNet_101, the values of IoU and Dice are 0.3792 and 0.5021. Apparently, the combination of U-Net with transfer learning of VGG16 is outperforming the U-Net with ResNet_101. Examples of detection results are shown in Fig. 3. The outcomes indicate that the proposed methodology can significantly localize the cracks in the images. A model based on SfM has been created to provide a comprehensive view of the Tai Tam Stone Bridge. Figure 4 shows the geometric bridge model, which is reconstructed using the software “ContextCapture” from Bentley. As shown in the figure, the reconstructed model with the aid of RTK is rich in detail. According to the detection results and the information observed from the 3D model, the health condition of the Tai Tam Stone Bridge is evaluated as level 2. The bridge has experienced mild deterioration and can still undertake loads from vehicles and pedestrians. Some cracked areas, such as the main road of the Tai Tam Stone Bridge, require to be examined in detail. The following inspection can be carried out by NDT methods, such as chain dragging and hammer sounding to identify the delamination areas under the cracks by listening to the sounds produced by vibrations.

Table 3 Performance of the proposed models
Full size table
Fig. 3
6 photos arranged in 3 columns, a, b, and c. A, present the photos of a bridge flank wall with grown plants and cracks. B and C, are the detection results of the same images from 2 proposed models.

Example of detection results for bridge flank wall and deck images a original images b detection results obtained by UNet_VGG16 c detection results obtained by UNet_ResNet_101

Full size image
Fig. 4
A 3-D model of a stone bridge with a water source underneath and trees around it.

3D model of Tai Tam Stone Bridge

Full size image

Essentially, datasets in this research contain cracks in different occasions and backgrounds to achieve the goal of generalization, but the concomitant is a decline in the algorithm performance. Another concern regarding the model performance is that the cracks with different backgrounds are not evenly distributed in the training dataset. Last but not least, a problem revealed in this research is the lack of ancient bridge images for training purposes. In future studies, the research direction could be focused on developing a model capable of detecting multiple defects with higher accuracy. Correspondingly, it requires more data to facilitate more robust predictive performance to better solve the ancient bridge inspection problem. Collecting more images of ancient bridges and building relevant datasets is of great significance for the future image-based maintenance of ancient bridges.

In this experiment, it is noteworthy that UNet_VGG16 outperforms UNet_ResNet_101, which is against common cognitions. Theoretically, the deeper or more complex the network, the better the model performance. As the transfer learning part of the model, ResNet_101 is expected to conquer VGG16 because ResNet_101 has 101 layers, while VGG16 contains only 16 layers. Additionally, ResNet (Residual Network) has been proven to surpass VGG in many image detection studies because it solved the problem of gradient vanishing and exploding by introducing the shortcut connection, whereas plain networks such as VGG always suffer from such problems. However, the application of more advanced techniques does not make a model perform better to some specific problems. Most deep learning models are developed to solve problems observed in previous models, which cannot be simply inferred that it will appear to work well on other occasions. Moreover, the detection process is performed by U-Net, and the impact that transfer learning brings to the performance of the final model is elusive. Thus, it is not impossible for UNet_VGG16 to outperform UNet_ResNet_101.

5 Conclusion

This paper proposes an integrated method using UAV images for ancient bridge maintenance. A criterion for the health condition of ancient bridges is established, and corresponding measures are suggested for each health status. An ancient stone bridge in Tai Tam Tuk Reservoir was chosen for the experiment. As a reservoir surrounds the bridge and the inspection truck is not available to enter, employing UAVs has become the primary option for the inspection. Defects on the bridge were segmented using U-Net with transfer learning of VGG16 and ResNet_101. The results indicate that the performance of UNet_VGG16 is better than the combination of U-Net with ResNet101. To improve the inspection performance, RTK was introduced in the experiment to enhance the localization accuracy. Additionally, based on the SfM, a 3D model is generated using UAV images to further monitor the target bridge. With the obtained detection results and 3D model, the health condition of the ancient bridge can be eventually assessed.

In summary, the proposed methodology can provide useful guides to ancient bridge maintenance. The UAV image-based inspection is considered beneficial in ancient bridge inspection regarding automation, flexibility, and inspection deliverability. In future research, by enhancing the detection model and acquiring more ancient bridge images, UAV image detection can become more sophisticated and thus can be expected to serve as a universal solution for the maintenance of ancient bridges.