Keywords

1 Introduction

In recent years, with the development of materials, micro-electromechanical systems (MEMS), micro inertial measurement unit (MIMU) and flight control technology, micro UAVs are growing rapidly and drawing more and more attentions. Particularly, multicopter stands out because of its compact size, simple mechanical structure, good agility, and capability of vertical landing and hovering [1]. In addition to aerial photography, multi-rotor UAVs have many potential applications in industry, agriculture, public security, scientific research and even military [2–4]. Especially in industrial field, compared to the traditional stationary industrial robots, micro UAVs, as one kind of new mobile robots, can work in three-dimensional space and have a wider range of applications [5].

Bridge inspection is one such scenario that could well demonstrate the superiorities of the multicopters. In order to prevent the occurrence of bridge structural failures, regular bridge inspections are required to find the defects of the bridges, especially in China, bridge-building continue to develop during the last decades. And current detection methods are mainly performed manually, which are cumbersome and unsafe. New bridge detection methods are urgently demanded.

Motivated by the requirements for autonomous inspection, some research institution also tried to use UAVs to do bridge inspections. Central Laboratory for Roads and Bridges in Paris tried to use a micro helicopter for bridge inspection [6], a vision system was installed for local position estimation and a front camera utilized for image capturing. The process of inspection was done manually. Such system was not so human-friendly for non-professional operators to control a helicopter without security protection. Some companies like AIBOTIX from Germany also offer solutions for bridge, wind turbine, and power line inspections with UAV [7]. One example from AIBOTIX is a hexarotor equipped with a protected frame and an upward camera, operated manually with the help of live video. However, without sensors to detect the environment, such system cannot be close enough to the bridge to take photos and also need to be operated manually under the bridge without GPS. Thus localization is a great challenge for autonomous flight under the bridge, and some related work in other fields can also provide reference for bridge inspection like the application for culverts inspection by UAV system from MIT [8], which used a Light Detection and Ranging sensor (LIDAR) for localization without GPS. Another research from the University of Pennsylvania can also inspire us [9–11], they fused multi-sensors including IMU, GPS, laser scanner, and vision system to estimate the position of the drone for indoor and outdoor autonomous flight.

To draw a conclusion from the current research and application of bridge inspection with the UAV, some challenges still need to be overcome. First is to find a precise localization method in bridge environment for position control of the UAV. Second is the integration of a high-performance autonomous flight system with automatic obstacle-avoid algorithm especially for bridge inspection. Last but not least is to make the flight controlling system more human-friendly.

In light of this, a hexarotor UAV system is designed and developed in this study, which is mainly used for bridge inspection. According to the characteristics of bridge environment and detection principle, a hexarotor frame with an upward camera gimbal is specially designed. Control system, sensor system, and image system are integrated into the system on the ROS. Moreover, position estimation and obstacle detection by fusion of multi-sensors is done for obstacle avoidance, human-friendly control flying under the bridge environment without GPS. Test results are shown in the result section and limitation are discussed in the conclusion section.

2 Design and Integration of the System

2.1 Modeling

Mathematic model and dynamic model of the hexarotor UAV are studied in this part.

Mathematic Model.

For the inertial coordinate, a north-east-down (NED) orthogonal coordinate system is established by the right hand rule. For the body fixed coordinate system, the head direction is the chosen as X axis, the right as Y axis, and down as Z axis.

The conversion between the inertial coordinate system and the body fixed frame is described via Euler-Angles. The position of the drone in the inertial frame and the attitude vector is defined as

$$ r = \left[ {\begin{array}{*{20}c} x \\ y \\ z \\ \end{array} } \right],\;q = \left[ {\begin{array}{*{20}c} \varphi \\ \theta \\ \psi \\ \end{array} } \right] , $$
(1)

where r is the position, q is the attitude, φ is the rotation angle around X axis anticlockwise, θ around Y axis, and ψ around Z axis.

With the attitude angle defined, the rotation matrix to transform a vector from the inertial frame to the body fixed frame is

$$ R = \left[ {\begin{array}{*{20}c} {cos\psi cos\theta } & {cos\theta sin\varphi } & { - sin\theta } \\ {sin\varphi sin\theta cos\psi - cos\varphi sin\psi } & {sin\varphi sin\theta sin\psi + cos\varphi cos\psi } & {sin\psi cos\theta } \\ {cos\varphi sin\theta cos\psi + sin\varphi sin\psi } & {cos\varphi sin\theta sin\psi - sin\varphi cos\psi } & {cos\theta cos\varphi } \\ \end{array} } \right] $$
(2)

Dynamic Model.

Referring to the modeling of quadrotor in [12], define the input control variable as

$$ \left\{ \begin{aligned} U_{1} & = F_{1} + F_{2} + F_{3} + F_{4} + F_{5} + F_{6} \\ U_{2} & = \frac{\sqrt 3 }{2}\left( {F_{2} + F_{3} } \right)L - \frac{\sqrt 3 }{2}\left( {F_{5} + F_{6} } \right)L \\ U_{3} & = \left( {F_{1} + \frac{1}{2}F_{2} + \frac{1}{2}F_{6} } \right)L - \left( {F_{4} + \frac{1}{2}F_{3} + \frac{1}{2}F_{5} } \right)L \\ U_{4} & = M_{1} + M_{3} + M_{5} - M_{2} - M_{4} - M_{6} \\ \end{aligned} \right. $$
(3)

where \( F_{1} , F_{2} , F_{3} , F_{4} , F_{5} \;{\text{and }}F_{6} \) are the thrusts, and \( M_{1} , M_{2} , M_{3} , M_{4} , M_{5} \;{\text{and }}M_{6} \) are the momentums, and L is the diagonal wheelbase of the frame.

Then the rigid body dynamics can be derived as

(4)

where m is the mass, I is the inertia, g is the gravity constant, R is the rotation matrix, q and r are defined as the attitude and the position to the initial coordinate system.

2.2 Structure Design and Analysis

The frame consists of several parts, including the body, arms, landing gear, protection parts, upward camera gimbal, and laser scanner gimbal. Based on the mathematic and dynamic model of hexarotor aerial vehicle, a framework is designed as shown in Fig. 1.

Fig. 1.
figure 1

The structure of designed hexarotor framework. (a) Overview of the framework. (b) Folding design of the structure. (c) Upward gimbal for camera. (d) Gimbal for laser scanner

Full size image

Compared with quadrotor, hexarotor has two more rotors to provide a redundant power for the vehicle, which can keep the vehicle flying safely even when one of the rotor is broken down. The diagonal wheelbase of the framework is designed to 950 mm which is capable for 18 inch’s propeller. Such size can guarantee a stable flight performance with or without wind. Considering the size and portability, the framework is designed to be foldable to make it convenient to be carried and transported.

The materials in consideration for the design include aluminum alloy, and carbon fiber reinforced polymer (CFRP) material. The carbon fiber tube and plate frame is used because it is strong and light and can be easily manufactured.

The upward two-axis camera gimbal is designed to reduce the vibration of the body and stabilize the camera to capture a clear image. The camera gimbal is mounted on the top of the vehicle in order to get the images of both bottom and side of the bridge.

The laser scanner gimbal is designed to keep the attitude of the laser scanner to get point cloud data of the environment in a same level. What is more, it can also rotate the laser scanner to get three dimensional point cloud data.

2.3 System Design

The system consists mainly four parts: mechanical system, control system, sensor system, and image system as shown in Fig. 2.

Fig. 2.
figure 2

Composition of the system

Full size image

Control System.

The control system is the core of the whole system. It plays a role in processing data of the sensors, mission planning, and flight control. It includes four parts: an embedded flight controller, an onboard computer, a ground station, and a remote controller.

The embedded flight controller is used for flight control. We choose an open source autopilot named Pixhawk to take up this position. The accelerometer, gyroscope and magnetometer are integrated in Pixhawk to estimate the attitude of the vehicle, while the barometer and GPS are for position estimation. It ensures a basic flight function of the vehicle.

The onboard computer is mainly for processing data and sending command to the flight controller. We use the Raspberry Pi 2 whose computing performance can meet the requirement. An Ubuntu server system is installed, with the Robot Operating System (ROS) running on it.

The ground station performs as a human computer interaction interface as shown in Fig. 3. Status information of the vehicle can be acquired from it such as attitude, altitude, and velocity and so on. Images from the camera can also be seen from the ground station to monitor the defects of the bridge in real time. Moreover, mission planning can also be done on it to make the vehicle fly autonomously.

Fig. 3.
figure 3

Ground station for bridge inspection

Full size image

Sensor System.

Besides of the onboard sensors including IMU, barometer and GPS for flight control, some additional sensors are also integrated into the system for environment detection, obstacle avoidance and position estimation in no GPS signal environment. Additional sensors include a laser scanner, ultrasonic sensors, a single-point LIDAR.

The laser scanner can detect the environments of the bridge, we can use the scan data to realize obstacle avoidance, localization and mapping. UTM-30LX from Hokuyo is chosen, which has 30 m and 270° scanning range and can be used in outdoor environments.

Ultrasonic sensors are also used to detect the obstacles where the laser scanner cannot reach in order to guarantee the safety of the vehicle.

An upward mounted single-point LIDAR is used to detect the distance between the drone and the bottom of the bridge. With this data, we can avoid the drone crashing upward to the bridge. The data can also be fused to estimate the altitude of the vehicle.

Image System.

The image system includes a camera and a wireless video link. The image system is for online monitoring and offline image processing.

For online monitoring, the wireless video link can transmit the compressed images of the camera to the ground station in real time so that the operator can determine whether there is a defect in time. There is an infrared LED on the drone to control the camera remotely. The uncompressed pictures can be saved on camera for offline image processing including image mosaic and defects recognition.

3 Fusion of Multi-sensors for Bridge Environment

Fusion of multi-sensors will be introduced in this section to estimate the status of the vehicle. With the status information, obstacle avoidance algorithm and human-friendly flying control algorithm are designed especially for bridge environment.

3.1 Position Estimation

The accelerometer, barometer and LIDAR are fused to estimate the altitude of the vehicle. The accelerometer can provide high dynamic acceleration data in short term, however its error will accumulate over time, while barometer and LIDAR will not. However, the barometer is very sensitive to the pressure change especially under the bridge where the wind is strong. The LIDAR can provide precise distance data, however, it will be disturbed by the step change of the surface. So we use complementary filter to estimate the altitude and vertical velocity of the vehicle as shown in Fig. 4.

Fig. 4.
figure 4

Fusion of sensors for altitude estimation

Full size image

The accelerometer, laser scanner and GPS are fused to estimate the horizontal position of the vehicle. When the GPS signal is good, the GPS and accelerometer is enough to estimate the position. When under the bridge where GPS is poor, laser scanner and accelerometer will be fused to estimate the position as shown in Fig. 5.

Fig. 5.
figure 5

Fusion of sensors for position estimation

Full size image

As for laser scanner, an incremental laser scan matcher [13] can be used for indoor environment to estimate position, however this algorithm fails to work under the bridge, because few features can be detected except for the bridge piers. So here we make use of such special feature to find one or two of the piers, and make an average of the point cloud to represent the center of the pier, and then calculate the relative position of the vehicle to the pier, as shown in Fig. 6. And then we use this data to fuse with the accelerometer.

Fig. 6.
figure 6

Schematic for position estimation of the laser scanner under the bridge

Full size image

One thing need to pay attention to is the coordinate transform. The inertial coordinate is a NED system, meanwhile the position estimated by the laser scanner is under the bridge coordinate where X axis is the connection direction of the two pier, so when the system is initialized, head the drone at the direction perpendicular to the connection direction of the two piers, so that the rotation matrix can be determined between the bridge coordinate and the inertial coordinate be measured.

3.2 Obstacle Avoidance

Obstacle avoidance algorithm will ensure the flight safety under the bridge. The laser scanner and the ultrasonic sensor can detect the obstacles in the environment. Once obstacle is detected, an artificial potential force field will be generated around the obstacle to push away the drone as shown in Fig. 7.

Fig. 7.
figure 7

Schematic of obstacle avoidance

Full size image

Set a safe distance \( {\text{d}}_{\text{safe}} \), when the distance of the obstacle d is less than d_safe, it generates an artificial potential field

$$ F = \frac{P}{{d^{2} + \alpha }} $$
(5)

where d is the distance of the obstacle, P is coefficient corresponding to the attitude adjustment, α is a constant.

In order to avoid a violent change of attitude, make a saturation to the attitude set point produced by the obstacle, and add this value to the original attitude set point by vector sum

$$ \overline{Att}_{ \cdot sp} = \overline{Att}_{ \cdot manual} + \overline{sat} \left( {\frac{P}{{d^{2} + \alpha }}} \right) $$
(6)

In the vertical direction, safe distance is \( d_{h\_safe} \), and the distance of the bridge is \( d_{h} \), when \( d_{h} \le d_{h\_safe} \), disable the manual input of upward motion, and set the vertical position setpoint as

$$ z_{sp} = z - (d_{h} - d_{h\_safe} ) $$
(7)

where \( z_{sp} \) the setpoint of the altitude, z is the current altitude.

3.3 Human-Friendly Control

In order to make the it easy to operate, some human-friendly control algorithms are designed to the bridge inspection.

For bridge inspection, the piers are required to be inspected. In order to get good picture of the piers, the drone need to keep a constant distance with the pier to take photos for image mosaic and defects recognition. So we make use of the characteristic of the pier to control the drone with the laser scanner.

As shown in Fig. 8, The laser scanner gets the point cloud of the environment, set a point cloud filter range so that we can get the point cloud of the pier, then calculate the center of the pier, so we can get a relative position between the drone and the pier.

Fig. 8.
figure 8

Flying of constant distance to the pier

Full size image

The position control system includes a position controller, a velocity controller, an attitude controller and an attitude rate controller as shown in Fig. 9. Such control model is a most common method to control the position of a drone [13].

Fig. 9.
figure 9

Control diagram

Full size image

To keep the vehicle flying with a constant distance to the pier, the input of the position controller is the reference position to the pier, and the relative position is the feedback, which is fused of relative position calculated by laser scanner and the integral of the acceleration from Sect. 3.1.

4 Result

The hexarotor aircraft is manufactured and assembled as shown in Fig. 10. Then we carried out some tests on the hexarotor aircraft, including weight measurement, flight endurance testing, and obstacle avoidance testing, to verify the performance of the vehicle. We also tested the system at the Nanpu Bridge and Yangpu Bridge in Shanghai to verify the capability of the inspection system.

Fig. 10.
figure 10

Prototype of the hexarotor

Full size image

4.1 Flight Performance

The weight of the developed hexarotor frame is about 4.7 kg. And its size is 940 mm. With a 10000 mAh battery, it can fly about 25 min without payload. With sensors, gimbals and the camera integrated to the system, its weight is 7.4 kg, which can fly about 14 min full load with 10000 mAh battery. And with 16000 mAh battery, its flight time is 20 min with 8.1 kg take-off weight.

The width of the bridge is about 36 m, and the length between two bridge pier is about 50 m, the average height of the pier is about 40 m. To guarantee the safety, we suppose the velocity of the drone is 3 m/s and it stops about 2 s to take a photo every 3 m. To finish inspection task of such region, it will take about 12.7 min. So the flight time of our system is enough to complete the inspection of the region enclosed by four piers. However, if another region need to inspected, we will replace with another battery.

4.2 Image Processing

We take some photos from Yangpu Bridge by our UAV system, and make some simple process on these photos.

The result of image mosaic of the bridge pier and bottom using the SURF feature matching algorithm are shown in Fig. 11.

Fig. 11.
figure 11

Flight test and image mosaic of the bridge. (a) The drone is conducting bridge inspection. (b) Several pictures of the bridge pier. (c) The result of image mosaic.

Full size image

4.3 Comparison

As shown in Table 1, compared with some similar products, most of the multicopter like DJI S1000 + are only for aerial photograph, so they cannot get the upward view of the bridge bottom and there is no additional sensor for no-GPS flight. Some products like Aibotics X6 are especially for industrial inspection, but without sensors to detect the environment, such system cannot be close enough to the bridge to take photos and also need to be operated manually under the bridge without GPS. So our developed UAV will be more capable to do bridge inspection work.

Table 1. Comparison with other products
Full size table

5 Conclusion

In this paper, a hexarotor UAV system is designed and developed for bridge inspection. With fusion of multi-sensors, the drone can detect the environment and estimate current status to guarantee the safety under the bridge. Human-friendly control algorithm makes it easier for operators to carry out the inspection process. Compared with the traditional bridge inspection method, such UAV bridge inspection system can partly replace human work, which will simplify the inspection process, improve the inspection efficiency, and guarantee personnel safety.

However, position estimation using laser scanner can only work in the specific bridge environment, it is still a problem without GPS when the environment is much more complex. To fly in a more complex environment, other localization method need to be integrate to the system to get more precise position to control the flight of the drone. Only with precise and reliable position estimation and control, further work including path planning and fully autonomous flight can be carried forward.