Keywords

1 Introduction

Modernized war deeply depends on the battlefield intelligence information. Ground reconnaissance equipment would be influenced by the obstacle and earth curvature. Satellite reconnaissance equipment couldn’t detect in real time and the details are not clear. Thus, these two reconnaissance methods have their own limits. Reconnaissance UAV is an important complementary to the other two detective means [1]. With different payloads carried, reconnaissance UAV could detect the details of battlefield situation in real time [2]. In the prewar period, reconnaissance of UAV could construct the battlefield situation, such as the map of battlefield, helping the commander make correct decision about the forces deployment. In the wartime, reconnaissance of UAV could detect the change of battlefield, detect and recognize target, helping the commander revise war policy in time [3]. In the postwar period, the damage assessment is an important evidence of summary [4]. The major form of reconnaissance is to capture images and videos from the battle field. The image acquired by the payload is an intuitive intelligence source. Thus, image processing technology directly influenced the intelligence result.

Figure 1 shows the process of UAV participation in the war. The UAV is reconnoitering the battlefield situation, sending the reconnaissance results to the receiving station and command and control vehicles by data chain in real time. The intelligence processing system is installed on the vehicle to generate the intelligence information and report the information to command post to help the commander make decisions about how to deploy forces by battlefield network.

Fig. 1
figure 1

UAV participated in the war

Full size image

This paper is organized as follows. Section one educes the importance of image processing in intelligence mining system with the way UAV is providing the intelligence support in war. Section two analyzes the requirement analysis of image processing technology. Section three describes the implementation of the used image processing technology. Section four concludes this paper.

2 Requirements Analysis

The UAV reconnaissance information processing system integrates the intelligence information obtained by various means during the reconnaissance task. The function of this system contains intelligence generation, sharing and distribution, and intelligence support throughout the process of mission execution. It is the information analysis and processing center of UAV reconnaissance mission system.

The requirements of UAV reconnaissance intelligence include two aspects. One is obtaining the high-precision reconnaissance panorama image. The other is detection, recognition, and location of enemy target. To obtain high-precision and large-scale images in the reconnaissance area to help construct the battlefield situation in real time, image mosaic method is applied. After image mosaic, combined with the map of the battlefield such as “Google Global Satellite Map,” we can get the location of target in the panoramic image. The location and property of targets are reported to the command post to decide whether to attack. Then artillery correction technology is used to amend the fire attack system if the fire did not hit the target. Also, if the target is hit by artillery, the damage assessment could help commander to summary the war to determine whether the target is damaged. Meanwhile, in most cases, the battlefield environment is complex. Single mode cameras may not be able to detect targets. SAR (synthetic aperture radar) or infrared camera may be also used to detect target.

3 Key Technology

In this section, we discuss the image processing technology in the intelligence mining system, containing (1) image mosaic and geometric correction, (2) image fusion, (3) target detection and recognition, (4) target location, and (5) artillery correction and damage assessment.

3.1 Image Mosaic and Electronic Map Fusion

After the mission is distributed to UAV, it flies to the mission area to execute reconnaissance mission. Because of the miniaturization of UAV and the limitations of the reconnaissance equipment, the scene structure information in a single frame image is very limited, which is not sufficient to obtain wide-area reconnaissance information. Thus we need to stitch the continuous multi-frame images to form a panoramic image. The panoramic image could also fusion with the electronic map to form the battlefield situation through coordinate transformation so that we could locate any point in the image in the world coordinate system [5]. The whole process of the algorithm is shown in Fig. 2.

Fig. 2
figure 2

Flowchart of building battlefield situation

Full size image

The algorithm divides into two parts, image mosaic and geometric rectification.

  1. 1.

    Image Mosaic: Firstly, we screened the image data. Due to the sensor equipment trouble or the influence of data chain interference, the image data acquired by UAV may contain some outlier data. The payload on UAV captures images about 25 frames per second, resulted in the large overlap region in the adjacent images, which means massive repeated information is contained in the image data. Considering the effectiveness of image data and the overlap region between images, several images are selected to stitch the panoramic image, such as six or seven images per second. Image mosaic is to transform all the images into a uniform coordinate. The first step is to register the image. Keypoint based method is fast and robust way to register images, containing keypoint extraction such as SIFT, SURF, ORB, and so on [6,7,8], construction of descriptor, and keypoint match. According to the matched keypoint, the transformation between images is calculated. Let the first image to be the benchmark and all the images could be transformed to the same coordinate.

  2. 2.

    When the panoramic image is acquired, according to the flight parameters of UAV, we could calculate the coordinate of the corner points on images in WGS84 (world geodetic system 1984), which is explained in the subsection target location. These corner points are used to be the pseudo GCP (ground control point) set to simulate the geometric distortion of images. The transformation model is based on the quadratic polynomial model, and the mathematical transformation relation is as follows.

    $$ \left\{\begin{array}{l}x={a}_0+\left({a}_1X+{a}_2Y\right)+\left({a}_3{X}^2+{a}_4 XY+{a}_5{Y}^2\right)\\ {}y={b}_0+\left({b}_1X+{b}_2Y\right)+\left({b}_3{X}^2+{b}_4 XY+{b}_5{Y}^2\right)\end{array}\right. $$
    (1)

    (x, y) and (X, Y) are the coordinates of arbitrary pixels in original and corrected images. (a i, b i), (i=0, 1, …, 5) are the coefficients of this model, which could be obtained by the least square method.

Figure 3 shows the fusion result of panoramic image and electronic map [9].

Fig. 3
figure 3

Fusion result of panoramic image and electronic map

Full size image

3.2 Visual and Infrared Image Fusion

The reconnaissance UAV carried a photoelectric payload including a visual camera and an infrared camera. The fusion between visual and infrared image could extract the most useful information of the two images, integrating them into a high-quality image. This effort could improve the utilization rate of image information.

Firstly, as the field of visual camera and infrared camera are not the same. We need to extract the overlap region of visual and infrared image to fuse. While the visual camera and infrared camera are in one pod, these two cameras would be focused on the same place, so that the center of visual image and infrared image are the same. According to the camera parameters, focal length and pixel size, we could easily extract the overlap region between visual camera and infrared camera.

Now, there are two of the most commonly used ways to integrate visual and infrared image: pixel level image fusion and feature level image fusion. Pixel level image fusion are fused on the original data level, contains more abundant details, while feature level image fusion extract features in the two images such as shapes, edge, contour, regions, etc. Thus feature level image fusion not only keeps effective information, but also compresses information.

In our system, the pixel level image fusion is utilizing the weighted mean of the luminance component of visual image and infrared image.

As is known to us, the background of infrared image is smooth and the difference between feature and background is in brightness. Thus, the most important thing is to extract the features of infrared image. We firstly utilize quad-tree decomposition to divide the infrared image and select the points in each image patch uniformly to be the control point of Bessel interpolation to reconstruct the background of infrared image. This method would result in the blocking artifact because of the different control points used in the stitched surface. Thus, the reconstructed background should be smoothed by a Gaussian filter to eliminate the blocking artifact. With the background of infrared image, we could easily obtain the features of the infrared image.

Figure 4 shows the fusion result of visual and infrared image with pixel level fusion and feature level fusion.

Fig. 4
figure 4

Pixel level image fusion result and feature level image fusion result. (a) is visual image; (b) is infrared image; (c) is pixel level image fusion result; and (d) is feature level image fusion result

Full size image

3.3 Target Detection and Recognition

Object detection system can detect and recognize targets on visible images, SAR images, and infrared images. The whole process is shown in Fig. 5.

Fig. 5
figure 5

The flowchart of target detection and recognition

Full size image

In the object detection on visual images, deep learning has become one of the most popular methods in recent years [8]. It is a brain-like architecture that can be trained with massive data to extract features and learn the key features in sight of these data automatically. Deep learning network has deeper network than normal machine learning model and so that it can handle with more complicated features and realize object detection with high accuracy (Fig. 6).

Fig. 6
figure 6

Detection results on visual images

Full size image

In the object detection on SAR and infrared images, which have only one channel, denoising, thresholding, and dilating are used to make objects much prominent from the background. Then use the convolution filter to sample the objects and filter the boxes with non-maximum suppression to get the accurate location of objects (Fig. 7).

Fig. 7
figure 7

Detection results on SAR images

Full size image

3.4 Target Location

After the target is detected and recognized, the location of target is an important property used to guide the fire attack system to attack the target. The coordinate system we utilized is WGS 84 (World Geodetic System 1984). The relevant method is target location. It is based on photogrammetry and image processing technology, and it aims to calculate the latitude and longitude of target with the flight parameters of UAV in our system.

There are mainly three methods to locate the target.

The first one is based on the telemetry data of UAV. It could locate the target in real time while the location accuracy is not enough. The real time location of UAV is measured in time by the GPS (global position system) mounted in the UAV. With the altitude of UAV and elevation data in DEM format, the distance between UAV and ground is calculated. Also, the position, posture, azimuth, and pitch angle are measured by the sensor mounted on the UAV in real time.

According to the principles of imaging, the point location formula is as follows:

$$ \left\{\begin{array}{c}{X}_{\mathrm{A}}-{X}_{\mathrm{s}}=\left({Z}_{\mathrm{A}}{-}{Z}_{\mathrm{S}}\right)\left({a}_1\times x{+}{a}_2\times y{-}{a}_3{\times} f\right)/\left({c}_1\times x{+}{c}_2\times y{-}{c}_3\times f\right)\\{}{Y}_{\mathrm{A}}{-}{Y}_{\mathrm{s}} =\left({Z}_{\mathrm{A}}{-}{Z}_{\mathrm{S}}\right)\left({b}_1{\times} x{+}{b}_2{\times} y{-}{b}_3\times f\right)/\left({c}_1\times x+{c}_2\times y-{c}_3\times f\right)\end{array}\right. $$
(2)

(X A, Y A, Z A) is the coordinate of object point A in ground photogrammetric coordinate system, and (X S, Y S, Z S) is the coordinate of photography center. (x, y, −f) is the coordinate of image point a in image space coordinate system S _ XYZ . f is the focal length.a 1, a 2, a 3, b 1, b 2, b 3, c 1, c 2, c 3 are the elements of rotation matrix R . The rotation matrix R could be calculated as follows

$$ {\displaystyle \begin{array}{c}R=\left[\begin{array}{c@{\quad}c@{\quad}c}{a}_1& {a}_2& {a}_3\\ {}{b}_1& {b}_2& {b}_3\\ {}{c}_1& {c}_2& {c}_3\end{array}\right]\\ {}=\left[\begin{array}{c@{\quad}c@{\quad}c}\cos k& \sin k& 0\\ {}-\sin k& \cos k& 0\\ {}0& 0& 1\end{array}\right]\left[\begin{array}{c@{\quad}c@{\quad}c}\cos \omega & 0& -\sin \omega \\ {}0& 1& 0\\ {}\sin \omega & 0& \cos \omega \end{array}\right]\left[\begin{array}{c@{\quad}c@{\quad}c}1& 0& 0\\ {}0& \cos \varphi & -\sin \varphi \\ {}0& \sin \varphi & \cos \varphi \end{array}\right]\end{array}} $$
(3)

φ, ω, k are the pitch, roll, and yaw of payload in φ − ω − k rotation system.

The second method is based on the space intersection. It is the extension of the first method. Laser range finder is used in this method. It is used to measure the distance between target and UAV relatively accurately. With the location result when the UAV is locating the same target at different place, we can construct the intersection model. Least-square method could be used to reduce the location errors. Since the laser range finder is a more stable and accurate device, the measured result could be used as a benchmark value.

The formula is as follows:

$$ \left(x,y,z\right)=\min \sum {\left(f\left({x}_n,{y}_n,{z}_n\right)-{L}_n\right)}^2 $$
(4)

(x, y, z) is the location result. f(x n, y n, z n) is the distance between UAV and target calculate by the first method in time n. L n is the measure esult of laser range finder in time n. Compared with the first method, the method with space intersection model is a more accurate way to locate the target.

The third method is based on the map match. It is also the extension of the first method and is similar to geometric rectification. From the first method, we can calculate the latitude and longitude of the visual image’s four vertex to superpose the visual image over “Google Global Satellite Map” with some errors. Then, register the visual image and map image automatically or manually. Figure 8 shows the result of map matching between visual image and the relative image in “Google Global Satellite Map.” The first image is the visual image captured by the payload. The figure shows the visual image superimposed over the “Google Global Satellite Map.”

Fig. 8
figure 8

location based on map match

Full size image

3.5 Artillery Correction and Damage Assessment

After the target property and location are sent to the command post, fire attacking system decides to attack the detected target. UAV would hover over the target to monitor it to know the strike situation by comparing the images before and after attack. The process is shown in Fig. 9. Artillery correction is to calculate the distance and angle between the location of target and the impact point. By sending the distance and angle of impact point relative to the target to the fire attacking system, they can amend system to attack again until the target is hit. After the target is hit, the damage image is captured. By comparing the images before and after attacking, according to the attributes of the target, type and killing radius of the warhead, and other features, the damage situation is assessed to decide whether to attack again. The result can be seen in Fig. 10.

Fig. 9
figure 9

The flowchart of artillery correction and damage assessment

Full size image
Fig. 10
figure 10

Comparison before and after attack. (a) Before attack. (b) After attack

Full size image

4 Conclusion

UAV intelligence information processing method is a branch of UAV combat application research. Many mature technologies in image processing will be applied in the information processing of UAV, such as moving target tracking, removing fog or rain effect from images, and so on. With the improvement of the technical level of the UAV and the diversification of the reconnaissance payload, the ability of intelligence mining system also needs to improve. Therefore, image processing method will also face many new challenges.