Keywords

1 Introduction

As an indispensable military equipment in modern warfare, the unmanned aerial vehicle (UAV) system has been widely applied. Among the multiple tasks, the information reconnaissance and the war field surveillance are major combat tasks for the UAV system. An UAV can be used for strategic campaign or tactic reconnaissance [1]. It is capable of flying above the enemy target and scout both on day time and at night. It can transmit the real-time image and other information of the target to the command center so that the commander is able of making up combat plans based on the information; on the other hand, by the UAV system’s mission evolving from only reconnaissance to reconnaissance and assaulting, tracking the enemy target steadily and accurately has become a primary precondition for UAV to achieve various tasks; therefore, a real-time tracking algorithm adaptable to complicated war field environment is of great importance for the UAV applications and development [1].

2 Tracking Algorithms

In the visual tracking system, the selection of the target tracking model would directly affect the accuracy of the tracking result. Generally, the color, the pattern, or the gradient of the target is often selected as a feature of the target for tracking. In different application scenarios, the feature selection varies. In this chapter, all the algorithms use the color of the target as the tracking feature. After the region of tracking is chosen, the histogram model of the color for this region is built as the target tracking model.

2.1 Mean Shift Algorithm

The mean shift algorithm is a nonparametric probability estimation algorithm [2, 7]. It applies to the gradient optimization method to the probability density function (p.d.f.) of the feature pixel. By recursively optimization, it converges to the local maximum of the p.d.f. rapidly by which the target can be tracked swiftly. The mean shift algorithm is able of tracking the target in real time and adapting to some levels of target morphing and rotating.

In order to apply the mean shift algorithm, the target model needs to be constructed first, that is, for the initial image, compute the feature value probability in the feature space of all the pixels in the target region. After that, the candidate models for the current image need to be built. Similar to the target model, it is to compute the feature value probability in the feature space of all the pixels in the candidate region for the current image. Finally, with the similarity measurement function, the similarity between the target model in the initial image and the candidate model in the current image is calculated. The average offset vector can be obtained by maximizing the similarity function. This average offset vector is the vector that the target is moving from the initial position to the current position. With the convergence attribute of the average offset vector, the computation of the average offset vector recursively will make the vector finally converge to the true position of the target in the current frame so that the target is successfully tracked. The mean shift algorithm is a data-driven tracking algorithm.

2.2 Auxiliary Particle Filter Algorithm

The particle filter algorithm [3, 5, 6, 9] is one of the most popular mathematical tools in solving the nonlinear problems in the target tracking area. It provides a probability model framework of target state prediction by the posterior p.d.f. Based on the importance sampling and the resampling technique, this algorithm is a model-driven tracking algorithm. The classic particle filter method needs a huge amount of dense samplings to achieve a high accuracy of target estimation; however, the algorithm often encounters the problem of lacking divergent particles. Based on the basic particle filter, the auxiliary particle filter (APF) algorithm includes the current measurements to the importance distribution to provide an importance distribution with relatively even weights of the particles. The divergence of the particles is improved, and the problem of lacking divergent particles is solved [3]. It is suitable for the target tracking with the nonlinear model and non-Gaussian noise.

The joint p.d.f. \(p({x_k},i|{z_{1:k}})\) can be described as the following by the Bayes’ rule:

$$ \begin{array}{c}p({x_k},i|{z_{1:k}}) \propto p({z_k}|{x_k})p({x_k},i|{z_{1:k - 1}})\\ = p({z_k}|{x_k})p({x_k}|i,{z_{1:k - 1}})p(i|{z_{1:k - 1}}).\\ = p({z_k}|{x_k})p({x_k}|x_{k - 1}^i)\omega_{k - 1}^i\end{array} $$
(66.1)

\(p({x_k},i|{z_{1:k}})\) is the nonlinear model in UAV video target tracking; the particle is not able to be sampled directly, so the importance density function satisfies the following proportional relationship:

$$ q({x_k},i|{z_{1:k}}) \propto p({z_k}|\lambda_k^i)p({x_k}|x_{k - 1}^i)\omega_{k - 1}^i, $$
(66.2)

where \(\lambda _k^i\) \({x_k}\) given \(x_{k - 1}^i\). It can be the expectation \(\lambda_k^i = E[{x_k}|x_{k - 1}^i]\) or a sample \(\lambda_k^i \sim p({x_k}|x_{k - 1}^i)\). In the experiment, we set the state transferring noise \({\omega_k}\) as a zero-mean Gaussian noise. Thus, the auxiliary particle collection can be chosen based on \(\lambda _k^i \sim p({x_k}|x_{k - 1}^i)\). Let

$$\lambda _k^i\, = \,x_{k - 1}^i + B \cdot no_k^i,$$
(66.3)

where \(x_{k - 1}^i\) is the sampled particle based on the (k − 1)th tracking result, \(no_k^i\) follows a Gaussian distribution with mean as 0 and variance as 1.

2.3 Mean Shift of a Regional Color Distribution Combined with the Particle Filter Algorithms

Mean shift of a regional color distribution combined with the particle filter algorithms [4] combines features of the mean shift algorithm and the particle filter algorithm. It designs a tracking algorithm with the meanshift framework. In order to avoid the local minimum or maximum problem, the particle filtering algorithm is applied in the optimization phase to search for the true local optimum value. In this algorithm, only a few particles are used, and the computational cost of the particle filter algorithm is greatly reduced.

The workflow of the algorithm is as the following: The sampled particle at time k is composed of two types—one is selected from the tracking result with a fixed probability, and the other one is obtained by the weighted particle at time k − 1. Then, the states of the particles at time k are predicted, and the weights of the particles are calculated. Next, the mean shift algorithm drives the tracker and obtains the tracking result. The algorithm compares the Bhattacharyya distance between the particle prediction result and the tracking result of the target model. If the prediction is better than the tracking result, move the tracker to the particle prediction position and adjust the tracking region.

2.4 Mean Shift-Auxiliary Particle Filter Algorithm

Because of the complication of the UAV reconnaissance environment and the movement feature of the tracking target, the mean shift-auxiliary particle filter (M-APF) algorithm applies the APF algorithm as the framework of the tracking algorithm. It selects the tracking region from the initial image and builds the target model. The samples are sampled from the particle collection \(\{ x_k^j,{i^j}\} _{j = 1}^N\) based on the importance density function \(q({x_k},i|{z_{1:k}})\). The mean shift algorithm is then used as deterministic optimization to move each sample particle to the local maximum of the posterior density function so that the sample particles is approaching to the true location area of the target. In this way, the application of only fewer particles can achieve the same result as the conventional APF algorithm. The real-time performance of the algorithm is then improved.

3 Simulation Analysis

In this chapter, the video obtained from the UAV payload is used for the experiment. To verify the effect of the M-APF tracking algorithm proposed by this chapter, we employ four tracking algorithms for comparison: the mean shift algorithm, the APF algorithm, the mean shift of a regional color distribution combined with the particle filter algorithms, and the M-APF algorithm. We compare the real-time performances and the robustness of the four algorithms. The APF algorithm uses N = 200 particles. The mean shift of a regional color distribution combined with the particle filter algorithms uses N = 20 particles. The M-APF algorithm uses N = 10 particles.

The hardware and software environments of this chapter are CPU: Q9500 2.83 GHz, memory: 4G, OS: Windows, development platform: Visual Studio 2008, and developing language: Open CV and C++.

3.1 Real-Time Performance

In this chapter, we use 300 frames of an image captured from the real UAV video to do the simulation. The resolution of the image is \(614 \times 467\). The tracking window is selected manually in the initial frame. In about the 220th frame, some covering shadows occur in the tracking scenario, and the target begins to be covered by the shadow. In the 255th frame, the target is severely covered. In the 275th frame, the target leaves the shadow.

Table 66.1 provides the statistic results of the time cost for the four algorithms tracking targets. We divide the 300 frames of the image into six groups: 1st to 50th frame, 51st to 100th frame, 100th to 150th frame, 150th to 200th frame, 200th to 250th frame, and 251st to 300th frame; calculate the average time cost of each group and the whole video and list them in Table 66.1.

Table 66.1 Average time cost of the four algorithms
Full size table

By the simulation result in Table 66.1, the M-APF algorithm proposed by this chapter spends less time than the APF algorithm and the mean shift of a regional color distribution combined with the particle filter algorithms for all the groups. For example, for the 151st to 200th frame, the time costs of the mean shift algorithm, APF algorithm, and mean shift of a regional color distribution combined with the particle filter algorithms are 1.69, 2.14, and 2.26 ms, respectively. Meanwhile, the algorithm proposed by this chapter spends 1.84 ms. For the whole tracking process, the first three algorithms spend 1.68, 2.13, and 2.29 ms per frame, while the M-APF algorithm spends merely 1.84 ms per frame. Comparing with the mean shift algorithm, the time cost of the M-APF algorithm is slightly higher. However, comparing with APF algorithm and the mean shift of a regional color distribution combined with the particle filter algorithms, the M-APF algorithm raises the processing speed significantly by using fewer particles. It has a better real-time performance than the APF algorithm and the mean shift of a regional color distribution combined with the particle filter algorithms. It can satisfy the real-time performance requirement of the UAV visual tracking system and is available to implement the real-time mobile object tracking for the UAV platform.

3.2 Robustness

We show the simulation result of the 1st, 187th, 255th, and 275th frames. Figures 66.1, 66.2, 66.3, and 66.4 provide the tracking result of the mean shift algorithm, the APF algorithm, the mean shift of a regional color distribution combined with the particle filter algorithms, and the M-APF algorithm. For each group of the figure sequence, figure part a is the initial frame. The region marked by a red rectangle is the selected tracking area. Figure parts b, c, and d are the tracking results of the 187th, 255th, and 275th frames of the four algorithms where the tracked region is marked with the red rectangle.

Fig. 66.1
figure 1

Tracking result of the mean shift algorithm

Full size image
Fig. 66.2
figure 2

Tracking result of the APF algorithm

Full size image
Fig. 66.3
figure 3

Tracking result of the mean shift of a regional color distribution combined with the particle filter algorithms

Full size image
Fig. 66.4
figure 4

Tracking result of the M-APF algorithm

Full size image

Comparing figure parts a and b in each figure group, the rectangles in all the figures can lock the tracking target, which implies that when the tracking target is experiencing some level of rotating and small background illumination changes, all the algorithms can track the target stably. By figure part c of all the groups, when the video moves to the 255th frame and the target is being covered, the mean shift algorithm and the APF algorithm happen to lose the target slightly. By figure part d, the mean shift algorithm and the APF algorithm lose the target completely when the target leaves the covered area. However, in Figs. 66.3d and 66.4d, the target is still tracked stably, which means that the mean shift of a regional color distribution combined with the particle filter algorithms and the M-APF algorithm are able of keeping stable track even when the target is covered.

On the basis of analysis of the real-time performance and the robustness of the four algorithms, the M-APF algorithm proposed by this chapter is slightly weaker than the mean shift algorithm in the real-time performance but is better in the robustness aspect. Comparing the APF algorithm with the mean shift of a regional color distribution combined with the particle filter algorithms, the M-APF algorithm is significantly better in real-time performance; therefore, the algorithm proposed by this chapter is superior to the rest three algorithms by the general performance. Specifically, the algorithm proposed by this chapter is capable to track the target stably and accurately when the target is covered and it has a good real-time performance. It can meet the requirement of real-time performance and robustness in the practical application of an UAV target tracking system.

4 Conclusion

With the complicated ground environment and high mobility of the target, the UAV mobile target tracking system often encounters the problem of poor real-time performance and the problem of losing a target when it is covered. This chapter proposes a real-time UAV target tracking algorithm based on the M-APF algorithm and compares it with the mean shift algorithm, APF algorithm, and mean shift of a regional color distribution combined with the particle filter algorithms. The result shows that the M-APF algorithm provides a good solution to solve these problems. It can perform in real time and track the target when it is experiencing some level of rotating, scaling, and small background illumination changes. It can also track the target stably and accurately when the target is covered by shadows. It may meet the requirements of real-time performance and robustness for the UAV platform target tracking system; however, the tracking target selected in this chapter is of small size and high mobility. In the future, we will focus on the rapidly rotating targets with a larger size.