Keywords

1 Introduction

Recently, countries and groups have increasingly expressed concern about environmental protection and low-carbon life. The trend of social development not only promotes the great development of inland river transportation, but also causes growing overloaded phenomenon of inland river ship.

With the development of information technology, video camera and high definition camera are considered as suitable sensor devices for capturing and recognizing spatio-temporal aspects of inland river structures and traffic situations, and have been introduced in many social systems for various purposes, especially transportation. It is well recognized that vision-based surveillance systems are more versatile than others for traffic parameter estimation and overloaded phenomenon identification [1].

Ship detection and ship tracking can offer us a continuous description of the vessel traffic flow. Therefore, it has been an important and challenging issue in video-based overloaded ship identification. However, there are several problems that remain unsolved in this process. On the one hand, the tracking result strongly depends on the quality of ship detection. Apparently, it augments the instability of tracking system. On the other hand, overloaded ship identification is [2] to be processed in real-time and it needs a simple and efficient approach to extract the ship feature.

In the last several years, extensive research work has been done and many traffic monitoring systems have been exploited that include road transportation2 and water transportation [3]. Researchers have developed various algorithms to extract object feature and track moving objects but most of the water transportation monitoring systems are focused on infrared cameras [4], automatic identification system (AIS) [5] or radar [6, 7], while little work has been done for optical image 8. Considering the complexity of the Kalman filter, [8] many researchers presented their own algorithms to construct imaging filters and achieve real-time operation [9, 10].

This paper presents a robust and real-time method to identify overloaded ship through a video camera and high definition camera. The proposed algorithm includes two stages: ship detection, ship tracking, and overloaded ship identification.

The remainder of the paper is organized as follows. We first introduce the process of ship detection in Sect. 67.2. The algorithm related to ship tracking is given in detail inSect. 67.3. Overloaded ship identification and its experimental results are presented in Sect. 67.4. Finally, the conclusion are drawn in Sect. 67.5.

2 Ship Detection

It is a very important step to extract the ship shape out of the river background. The ship detection method requests to automatically segment every ship so that there can be a unique tracking associated with the ship. In this phase, we will solve several problems as follows:

Extract the background image automatically from a sequence of river traffic images and update the background continually according to the change in ambient lighting, weather, etc.

Select an adaptive filter to eliminate abnormal moving object in the binary background subtraction image so that the system can be more robust.

Detect ship from the binary background subtraction image.

Apparently, it is desirable to extract the initial background image automatically from a sequence of road traffic images before background subtraction. Therefore, we propose a background extraction method based on moving object pixel detection, in which each pixel of image will be identified whether its intensity had an obvious change or not. Moving object pixels could be extracted from current input image by performing a difference on three consecutive inter-frames. First, it is needed to calculate inter-frame subtraction image of two pairs of images, i.e., the first subtraction image between (k−2) frame image and the (k−1) frame image, and the second subtraction image between (k−1) frame and (k) frame. Second, two binary images could be transformed from the two subtraction images via a dynamic subtraction threshold. Then, we apply the bitwise logical and operation to the two binary subtraction images to clarify the moving object pixels. Finally, the original background image could be obtained through patching up non-moving object pixels from a sequence of input images. The same method could be used to update the background image.

After the update process, we calculate the difference between background image and input image for each pixel, and then apply binary filter to background subtraction image to clarify the moving object region. While the binary difference image of moving ships is obtained, we apply a block filter based on statistics to filter the noises and then adopt a simple seed-growing arithmetic to detect ship. The detailed results of our proposed scheme are represented in Figs. 67.1 and 67.2.

Fig. 67.1
figure 1

An example of background estimation and updating

Full size image
Fig. 67.2
figure 2

An example of ship detection

Full size image

3 Ship Tracking

Based on the segmented ship shape, which can be represented by a simple square model, we propose a Kalman filter method to track ship.

The Kalman filter is the minimum-variance state estimator for linear dynamic systems with Gaussian noise. In addition, the Kalman filter is the minimum-variance linear state estimator for linear dynamic systems with non-Gaussian noise.

Consider the system model as follows:

$$ xk + 1 = \cdots Fxk \cdots + \cdots wk $$
(67.1)
$$ yk \cdots = Hxk \cdots + \cdots vk $$
(67.2)

where k is the time step, xk is the state, yk is the measurement, wk and vk are the zero-mean process noise and measurement noise with covariances Q and R respectively, and F and H are the state transition and measurement matrices. The Kalman filter equations are given as follows:

$$ Pk = F\xi K - 1 \cdot FT + Q $$
(67.3)
$$ Gk = Pk \cdot HT\left( {HPk \cdot HT + R} \right) - 1 $$
(67.4)
$$ x\left( {k|k} \right) = F \cdot x\left( {k - 1|k - 1} \right) + Gk\left( {yk - HF \cdot x\left( {k - 1|k - 1} \right)} \right) $$
(67.5)
$$ \xi k = \left( {I - GkH} \right) \cdot Pk $$
(67.6)

For k = 1, 2… where I am the identity matrix \( x\left( {k|k} \right) \) is the priori estimate of the state \( xk \) given measurements up to and including time \( k - 1 \) \( x\left( {k - 1|k - 1} \right) \) is the posteriori estimate of the state \( xk \) given measurements up to and including time \( k \). \( Gk \) is the Kalman gain, \( Pk \) is the covariance of the priori estimation error \( xk - x\left( {k - 1|k - 1} \right) \), and \( \xi K \) is the covariance of the posteriori estimation error xk \( x\left( {k|k} \right) \)

When the noise sequences \( \left\{ {wk} \right\} \) and \( \left\{ {vk} \right\} \) Gaussian, uncorrelated, and white, the Kalman filter is the minimum-variance filter and minimizes the trace of the estimation error covariance at each time step. When \( \left\{ {wk} \right\} \) and \( \left\{ {vk} \right\} \) are non-Gaussian, the Kalman filter is the minimum-variance linear filter, although there might be nonlinear filters that perform better. When \( \left\{ {wk} \right\} \) and \( \left\{ {vk} \right\} \) are correlated or colored, (3–6) can be modified to obtain the minimum-variance filter. Our system schematic diagram is shown in Fig. 67.3

Fig. 67.3
figure 3

The system schematic diagram

Full size image

As we know, ship feature extraction plays an important role in ship tracking. The extracted information must be robust and essential to the accurate visual interpretation of the image so that the tracking result is not dependent on parameters controlling thresholds, which is often established empirically to achieve acceptable performance. Our arithmetic extracts many ship features to characterize each ship, such as ship center of mass, average intensity, and ship speed, etc.

Now suppose it satisfies the equality constraints. Let Xk be the ship feature vector of the time k. Then:

$$ Xk = \cdots \left[ {xi\left( k \right) \cdots xvi\left( k \right) \cdots xj\left( k \right) \cdots xvj\left( k \right) \cdots xD\left( k \right)} \right] $$
(67.7)
$$ Yk = \cdots \left[ {yi\left( k \right) \cdots yj\left( k \right)} \right] $$
(67.8)

where \( xi\left( k \right) \) and \( xj\left( k \right) \) are the state values of ship center’s coordinates in the input image, \( xvi\left( k \right) \) and \( xvj\left( k \right) \) are their speed values, \( xD\left( k \right) \) is the average intensity of the whole ship, \( yi\left( k \right) \) and \( yi\left( k \right) \) are the measurement values of xi(k) and \( xj\left( k \right) \) \( Xk = \cdots \left[ {xi\left( k \right) \cdots xvi\left( k \right) \cdots xj\left( k \right) \cdots xvj\left( k \right) \cdots xD\left( k \right)} \right] \).

Based on the above definition and the system schematic diagram, we define the Kalman system model as:

$$ Xk + 1 = FXk \cdots + \cdots wk $$
(67.9)

The final result may cause the Kalman filter updates the Kalman gain Gk and the ship feature vector is recalculated. Our system tracks all these ships by implementing the above processes recursively until they have left the field of view.

$$ F = \left[ {\begin{array}{*{20}c} {\cos A} \hfill & {T\cos A} \hfill & 0 \hfill & 0 \hfill & 0 \hfill \\ 0 \hfill & 1 \hfill & 0 \hfill & 0 \hfill & 0 \hfill \\ 0 \hfill & 0 \hfill & 1 \hfill & T \hfill & 0 \hfill \\ 0 \hfill & 0 \hfill & 0 \hfill & 1 \hfill & 0 \hfill \\ 0 \hfill & 0 \hfill & 0 \hfill & 0 \hfill & 1 \hfill \\ \end{array} } \right] $$
(67.10)
$$ Yk \cdots = HXk \cdots + \cdots vk $$
(67.11)
$$ H = \left[ \begin{gathered} 1\quad 0\quad 0\quad 0\quad 0 \hfill \\ 0\quad 0\quad 1\quad 0\quad 0 \hfill \\ \end{gathered} \right] $$
(67.12)

4 Overloaded Ship Identification

The research in this step is primarily how to get the position of the waterline by digital image processing technology. In order to ensure the detection accuracy of waterline, another high definition camera will be used to shoot the ship photograph. Taking into account the difference of coordinate systems between the video camera and high definition camera, it is necessary to convert the ship center’s coordinates and obtain the data about ship length, ship speed, and navigation direction through the ship tracking result.

Our paper takes two steps to detect the waterline edge.

The first step is the edge detection approach based on the navigation direction. Compared to other operations, the canny operator gets the better results in detection. In order to find the exact waterline, the false edges are removed by navigation direction projection. Finally, the waterline line is fitted by the least square method.

The second step is the edge pick-up approach based on the Hough transform. This paper adopts a voting method to choose the possibly true edges, in which the weight values of the following attributes are large: the hue grads, the saturation grads, the intensity grads, and the times of searching edges. According to the difference in the average values between the upside area and the downside, the method selects the true edge of waterline. The horizontal line is fitted by the least square method.

Finally, the distance between the top of ship and the waterline will identify whether the ship is overloaded or not. The detailed results of our proposed scheme are represented in Figs. 67.4 and 67.5.

Fig. 67.4
figure 4

An example of edge detection

Full size image
Fig. 67.5
figure 5

The edge approach about the top of ship and the waterline

Full size image

5 Conclusion

In this paper, we present a new algorithm for ship tracking in a video-based ITS. The experiment results on real-world videos show that the algorithm is effective and real-time. The correct rate of ship tracking is higher than 85 %, independent of environmental conditions.