1 Introduction

Automobile collision warning system is through sensors, computer image processing to realize distance monitoring and other navigation functions. It is one key technology for intelligent vehicle technology. The system can protect pedestrian’s safety effectively and reduce traffic accidents. It is an intelligent vehicle system with computer vision technology. Computer vision [37, 39, 44] is to use computer to realize human vision function and achieve recognition of objective 3D world. Computer vision is one of most active fields for artificial intelligence. It can be used in industrial production automation, intelligent robots, autonomous vehicle navigation, space target docking, target tracking and various industrial inspections, medical and military applications [24, 29].

In recent years, the number of killed people in traffic accident is about three thousand and eight hundred million; disabled people number is up to about seven thousand million. A direct economic loss is about one thousand and eight hundred billion dollar, indirect loss is about six thousand billion dollar.

At present, the world has more than seven point one billion automobiles; the quantity of cars retained in China is more than six thousand eight hundred million. According to the survey the number of deaths caused by traffic accidents of China every year, more than 75 % is from road traffic accidents. The number of deaths caused by traffic accidents of China every year is also ranked first in the world. The number of deaths for million cars is six-eighteen times that of developed countries. Vehicle collision warning system can determine the relative distance and relative speed between the vehicle and the vehicle in front by monocular vision positioning technique according to the automobile license plate image captured by the camera. It can judge the danger; timely remind the driver to make the appropriate action to avoid vehicle collision. Study on the positioning technology of this system can help the driver to judge, prompting warning, improve driving safety, and has broad application prospects.

The license plate [2, 7, 18, 57] is one key visual feature on the vehicle; it is one only regular shape on the car, uniform, markers with identity recognition function. In the previous research on automotive collision warning and intelligent vehicle, less attention was paid for the vehicle license plate. Learned from literatures, we have not seen the research methods similar to this paper. Therefore, the research content of this paper enriches automobile anti-collision technology and theory of intelligent vehicle technology. It can also provide an auxiliary method for obstacle avoidance research of unmanned vehicle navigation. At the same time, the research has certain scientific significance.

1.1 Research status of detection method of target vehicle based on monocular vision sensor

At present, the whole vehicle is identified as target in research on car collision warning system of monocular vision [5, 13, 30, 33, 40, 48, 51, 53], which is to facilitate the extraction of target. Usually with some prior knowledge, such as vehicle shape and width of the vehicle, the car high proportion as constraint condition detecting vehicle edge. Document [8] is for target recognition and extraction with edge characteristics of vehicle as the constraint condition. Tracking method for vehicle is using vehicle information acquired in an image. In the current image processing, using Euclidean distance, edge density and vehicle region pixel density error based on sum of squared differences (SSD) three criteria, to match the vehicle area likely in turn, so as to detect the vehicle in front. America Nevada State University Laboratory of visual use template matching method to the feature extraction of the motor vehicle in front by using Gabor wavelet is not sensitive to characteristics translation, rotation and scale change [28].

Literature [31] identifies the multi-lanes vehicle by target sampling, motion target ranges from camera more far in the road plane, the pixel is accounted for the less in the image. Based on this principle, vehicle region of interest is established, at the same time, interval sampling is determined. This method improves the extraction rate, but also it can lead to missing because it cannot get the detail information of vehicle. The literature [19] achieves the target vehicle extraction by using vehicles have typically strong symmetry characteristics. Usually, use this method needs establish the region of interest. Such as the use of lane as the constraint condition, the vehicle is limited in the lane. Then, the pixels of interest within the region are used as the symmetry detection based on gray level. In order to ensure the reliability of detection, it can also carry out symmetry detection based on edge. We can obtain the vehicle axis of symmetry and the width of the vehicle by symmetry detection. We can obtain the boundary of the vehicle bottom by a priori knowledge of the perspective projection transform. Finally, we can determine the rectangular region of vehicle and achieve the vehicle identification on the basis of aspect to vehicle high-width ratio.

There is a defect when this method is used. Namely for the road to gray uniform, gray symmetry is usually higher than the symmetry of the rear area the vehicle, so it is prone to miscarriage of justice. The literature [20] detects the target vehicle in front based on fractal dimension calculation method of Brown motion using vehicle texture feature is distinguished from background. Literature [52] realizes target vehicle detection by using the combination of multiple features. It realized vehicle recognition mainly using vehicle shadow and texture, symmetry, edge and sequence image normalization to inertia characteristics. It achieved better effect than using single feature for recognition and extraction.

At present, the existing methods are based on the whole vehicle as targets. The measurement method presented in this paper is to use the license plate frame of goals vehicle for cooperation target. After the license plate extraction and edge extraction in visual images, then license plate frame is used as cooperative target for monocular vision measurement.

1.2 Research status of monocular vision positioning technology based on cooperative target

The theoretical basis of monocular vision positioning technology with cooperative target [9, 17, 34, 54] is based on the corresponding relationship between a specific set of geometric characteristics of known position relation with the camera image projection on the projection image, combined with the parameters of camera, position and attitude can be computed for the coordinate system of geometric characteristics in the world coordinates. The application of geometric characteristics of the model-based monocular vision positioning can be divided into point, line and advanced geometric features such as. At present the most research is based on the characteristics of point and line feature. The problem of monocular vision positioning based on point features [5, 6, 10, 11, 15, 40, 43, 48, 53, 55] is also known as Perspective-N-Points (PNP) problem, through the development and research for many years, it has made quite a number of research results, it has been successfully applied in particular cases.

Location based on line feature [12, 27, 37, 39, 44, 45] is called PNL, research is mainly concentrated in three lines position (P3L) and four line location problem (P4L). On the P3L problem, document [25] solves the optimization using the optimal region, and the P3L multi solution problem is studied. Document [3] uses iterative initial value which is close to the true value. The weak perspective projection and parallel projection perspective pose is estimated using the iterative algorithm. Through discussing the rank of geometric matrix during the middle process, some constraints are put forward on the target line layout. On the issue of P4L, Sun Fengmei discusses the object location method of single parallelogram on single image in the literature [32]. The main conclusions are: the ratio between the adjacent side, adjacent edge angle information, since they are not projective invariants, so they can not provide any additional useful information for object orientation of single image. When you know any set of side length for the parallelogram, the rotation matrix and translation vector can be uniquely determined. A method to determine the camera orientation from rectangular is presented in literature [42]. Given the case of a scaling factor, linear methods restore rectangular Euclidean metric information by Raguel’s theorem. A method of three-dimensional information extraction is proposed and implemented from a single building image using rectangular structure in 3D information from single building image [4]. We also carried out related research for the P4L problem. For plane line features with rectangular distribution, we put forward the method of pose estimation with analytical solution. This method can get one group of target estimation results [26]. Hu et al. [16, 46, 47] used VSD based technology to intelligent description of vehicles.

According to the existing research results, positioning algorithm based on point feature is more mature. The analytical solution is main for positioning algorithm based on linear feature, the algorithm itself has not high precision as iterative solution, but the view of extraction from the image point, the linear feature extraction precision is better. And analysis solution of linear feature location method is simple, less calculation, better real-time character.

1.3 Research status of present safety situation assessment and early warning model of automobile

In foreign countries, research on early warning model of safe distance on the highway has made some achievements, representative results are: Mazda model, Honda model, Beckley University model of California [48]. The idea to set up the model of CW/CA system developed by Mazda company is: When the sensor of following car finds the deceleration of before vehicle, it starts to send information to the safety distance alarm system. When the current vehicle distance is below the braking distance for following vehicle, system sends out the instruction for brake system and it began to slow down. Braking velocity can be set according to different road conditions in model.

The automobile anti-collision warning system developed by the Honda Corporation is the two alarm modes. The first alarm is call alarm; second alarm call is brake alarm. In the model, the alarm distance determination is first for pilots for several trials to get a large amount of data, and then calculate safety distance the by regression analysis for the data. Beckley University model of California used two times of alarm, alarm distance calculation used Mazda model, different thing is the brake deceleration for before and after vehicles is equal value. Document [14] has made improvement in Mazda model, Honda model, the model of California Beckley University, a safety car distance warning model based on monocular vision is proposed. The method of safety degree is put forward as determining criterion of warning time for the safety car in model with examination into the driver and road surface conditions.

1.4 Development trend analysis

At present, the overall trend of automotive collision avoidance system has two directions, one is combination of various types of sensor information, access to information of environment and objects at the greatest degree, the optimal evaluation for the safety situation is get after complex operation. On the other hand, measurement means is based on the single sensor, through improving the sensor performance, hardware processing capacity and improved algorithm to improve the recognition ability, enhance the robustness and improve the speed of computation. The advantage of the former is the target recognition capability and high reliability, but there is the problem of high prices equipment with multi-sensors and advanced data processing, thus it can not be put into the short term application. The latter is weak in the recognition ability and reliability with multi-sensors combination way. But it is more practical by the advantages of high real-time and low cost.

The idea of this paper is to apply the monocular vision positioning method based on cooperative target to the automotive anti-collision technology; therefore it could enhance the detecting ability of car collision warning system based on monocular vision. In addition to the target vehicle positioning, this method can also realize determining attitude, eliminating the restrictions of the flat and the fluctuation on the road.

This paper takes license plate frame location as the research background. A new analytical positioning method is put forward aiming at the rectangular structure characteristics of plate frame. This method does not need iteration; positioning speed is high, without known space restriction between camera coordinate system and the target coordinate system, so there is no multi-solution problem. It can be used for locating posture with license plate frame image by image processing method [1, 21–23, 35, 36, 38, 41, 49, 50, 56].

2 Scheme design of automobile anti-collision warning system of monocular vision sensor based on license plate cooperative target

Technology line is as shown in Fig. 1 for collision warning system of monocular vision positioning based on plate cooperative target. Forward looking video camera captures an image, firstly, the image is preprocessed, then positioning and extraction of license plate image, way of license plate location is 1) license plate region coarse localization according to mathematical morphology 2) the accurate location of the license plate according to texture features 3) the precise positioning combined with the color information of the license plate.

Fig. 1
figure 1

The technology route map

Full size image

License plate location is complete, if there are pluralities of license plate images, it needs to segment for different license plates. After segmentation of the license plate images, extraction of geometric features of license plate edge, which includes four lines and four angular points. On the basis of text color, background color, and the type of arrangement of words form on the license plate, plate type can be determined. Because the length and width of each type of license plate is known, it can be used to get the space position between license plate cooperative targets as geometric features based on monocular vision measurement.

According to the projection on the image, using the selected location algorithm to calculate the position and posture of the license plate in the camera coordinates. Due to rotation and translation matrix is known between camera coordinate system and body coordinate system, correspondingly, we can get position and attitude for the target vehicle in the vehicle body coordinate. For vehicle license plate of license plate image and other visual camera, it can also use the same method to get position and attitude data for the multiple vehicles before and after the car.

We can obtain the moving speed and the rotation speed (posture change rate) in the vehicle coordinate system by differential treatment for position and posture of the same target vehicle at the adjacent time. Safety assessment and early warning system obtains the detected information of the vehicle position and pose changes rate, combined with the vehicle speed and the current steering velocity, as well as the vehicle’s inherent parameters (dimensions, weight, braking performance etc.) to conduct a comprehensive analysis and evaluation, to determine the current the level of security status and operation suggestion, to provide some form of the man-machine interface for the driver of the car.

3 The key technique of monocular vision positioning based on cooperative target

This paper proposes a detection method of vehicle pose based on monocular vision with license plate, its application environment is for highway. In this method, license plate frame of the target vehicle is for cooperative target. License plate frame is the rectangle with the length and the width known. As shown in Fig. 2, the projection for plate border in the camera target plane is a plane quadrilateral. The quadrilateral can be reflected to the space. Combined with the geometric model of the plate frame, coordinate and posture can be calculated from the license plate frame in the camera coordinate system.

Fig. 2
figure 2

Schematic diagram of monocular vision location based on license plate cooperative target

Full size image

Compared to the binocular vision measurement system, monocular cooperative measurement method needs only a vision sensor, has the advantages of simple structure, fast processing speed, which can meet the real-time requirements, but also to avoid the small field of view in stereo vision, problem of stereo matching difficult.

License plate image has four edges. These four edges intersect at four points. In Fig. 3, four edges are surrounded by red frame, four vertexes are surrounded by green circles. We use four vertexes which are formed by four edges of license plate image for automobile positioning.

Fig. 3
figure 3

Four edges with four vertexes of license plate image

Full size image

Projected model for license plate frame of automobile in camera coordinate system is as shown in Fig. 4. The perspective projection model is the hypothesis and the camera parameters are known. Four edges of license plate frame areL i (i = 1 ∼ 4). Image points for four verticesP i i = (1 ∼ 4) are p i i = (1 ∼ 4) in the image coordinates. We use four corner points of four edges of license plate as cooperative target to realize monocular vision positioning. The result for monocular positioning is closed-form solution. At the same time, this method has fast calculation. We take four verticesP i i = (1 ∼ 4) for monocular vision positioning.

Fig. 4
figure 4

Monocular vision positioning of rectangular cooperative target

Full size image

Coordinates of four vertices of license plate frame in image coordinates are(x i , y i , z i )i = (1 ∼ 4). Coordinates of four vertices in the camera coordinate system are (x i t i , y i t i , z i t i ),where t i i = (1 ∼ 4) is undetermined coefficient. In perspective projection, diagonal intersection point of license plate frame is corresponding to the diagonal intersection point for projection image of license plate frame in image plane. The coordinates of the center for the image of license plate frame isp 5 = (x 5, y 5, z 5), the coordinates of the points in the camera coordinate systemP 5 = (x 5 t 5, y 5 t 5, z 5 t 5), wheret 5 is the undetermined coefficient.

In perspective projection, as shown in Fig. 4, the unit direction vector ofP 1 P 5can be represented by coordinate of two points P 1andP 5, we have:

$$ {\left({A}_1,{B}_1,{C}_1\right)}^T={\left(\frac{\left({x}_1t{}_1-{x}_5{t}_5\right)}{l},\frac{\left({y}_1t{}_1-{y}_5{t}_5\right)}{l},\frac{\left({z}_1t{}_1-{z}_5{t}_5\right)}{l}\right)}^T $$
(1)

The unit direction vector ofP 3 P 5can be expressed by coordinates of two points P 3andP 5, we obtain:

$$ \left({A}_3,{B}_3,{C}_3\right)={\left(\frac{\left({x}_5{t}_5-{x}_3{t}_3\right)}{l},\frac{\left({y}_5{t}_5-{y}_3{t}_3\right)}{l},\frac{\left({z}_5{t}_5-{z}_3{t}_3\right)}{l}\right)}^T $$
(2)

As shown in Fig. 4,P 1 P 5and P 3 P 5are collinear, so the unit direction vectors for P 1 P 5andP 3 P 5are equal, therefore we have:

$$ 2{x}_5{t}_5-{x}_1{t}_1-{x}_3{t}_3=0 $$
(3)
$$ 2{y}_5{t}_5-{y}_1{t}_1-{y}_3{t}_3=0 $$
(4)

By Eq. (3) and Eq. (4):

$$ {t}_5=\frac{x_1{y}_3-{y}_1{x}_3}{2\left({x}_5{y}_3-{y}_5{x}_3\right)}{t}_1 $$
(5)

The length ofP 1 P 5isl, we get:

$$ \sqrt{{\left(x{}_1t_1-{x}_5{t}_5\right)}^2+{\left(y{}_1t_1-{y}_5{t}_5\right)}^2+{\left(z{}_1t_1-{z}_5{t}_5\right)}^2}=l $$
(6)

Substituting Eq. (5) into Eq. (6), we have:

$$ {t}_1=\pm \sqrt{\frac{l^2}{\left({x_1}^2+{y_1}^2+{z_1}^2\right){s_1}^2+\left({x_5}^2+{y_5}^2+{z_5}^2\right)-2\left({x}_1{x}_5+{y}_1{y}_5+{z}_1{z}_5\right){s}_1}} $$
(7)

Where positive t 1 is reasonable value. By Eqs (5) and (3), values for t 5and t 3 can be calculated. Accordingly, coordinates of P 1,P 3,P 5 can be computed in the camera coordinate system.

Similarly, the unit direction vector ofP 2 P 5can be represented by coordinates of two points P 2,P 5, thus we have:

$$ {\left({A}_2,{B}_2,{C}_2\right)}^T={\left(\frac{\left({x}_2{t}_2-{x}_5{t}_5\right)}{l},\frac{\left({y}_2{t}_2-{y}_5{t}_5\right)}{l},\frac{\left({z}_2{t}_2-{z}_5{t}_5\right)}{l}\right)}^T $$
(8)

The unit direction vector ofP 4 P 5can be represented by coordinates of two points P 4,P 5, therefore we get:

$$ {\left({A}_4,{B}_4,{C}_4\right)}^T={\left(\frac{\left({x}_5{t}_5-{x}_4{t}_4\right)}{l},\frac{\left({y}_5{t}_5-{y}_4{t}_4\right)}{l},\frac{\left({z}_5{t}_5-{z}_4{t}_4\right)}{l}\right)}^T $$
(9)

The unit direction vectors for P 2 P 5 andP 4 P 5are equal, therefore it can be obtained:

$$ 2{x}_5{t}_5-{x}_2{t}_2-{x}_4{t}_4=0 $$
(10)
$$ 2{y}_5{t}_5-{y}_2{t}_2-{y}_4{t}_4=0 $$
(11)

By Eq. (10) and Eq. (11):

$$ {t}_2=\frac{x_4{y}_5-{y}_4{x}_5}{2\left({x}_1{y}_5-{y}_1{x}_5\right)}{t}_5 $$
(12)

The length ofP 2 P 5isl, we get:

$$ \sqrt{{\left({x}_2{t}_2-{x}_5{t}_5\right)}^2+{\left({y}_2{t}_2-{y}_5{t}_5\right)}^2+{\left({z}_2{t}_2-{z}_5{t}_5\right)}^2}=l $$
(13)

Substituting Eq. (12) into Eq. (13), we have:

$$ {t}_4=\pm \sqrt{\frac{l^2}{\left({x_1}^2+{y_1}^2+{z_1}^2\right){s_1}^2+\left({x_2}^2+{y_2}^2+{z_2}^2\right)-2\left({x}_1{x}_2+{y}_1{y}_2+{z}_1{z}_2\right){s}_1}} $$
(14)

Positive t 1 is reasonable value. By Eqs. (12) and (10), values for t 2and t 4 can be calculated. Accordingly, coordinates of pointsP 2andP 4can be computed in the camera coordinate system. Hereto, we calculated the coordinates for four vertices P i i = (1 ∼ 4)of plate border. The direction vectors for four edgesL i (i = 1 ∼ 4) of license plate frame can be determined.

Camera frame is acquired by rotating the object frame with rotation matrixRfirstly and then translating it with vectorT.

$$ P=\boldsymbol{R}P\hbox{'}+T $$
(15)

R can be expressed with three angles of tilt, swing and spin. Unknowns for matrix R can be obtained by solving linear equations with three 2D to 3D line correspondences. Translation vector Tcan be expressed of three independent unknowns. The translation vector of the camera coordinate system can be uniquely determined by one point correspondence.3D coordinates and attitude to cooperative target can be identified using rotating plate theory in the camera coordinate system.

Measurement of the target vehicle speed is completed based on continuous measurement of target license plate position. It is difference process for the attitude determination results of license plate at two near time. Then we get target plate translational velocity and three axis rotation speed (change rate of pose) of three-dimensional space in vehicle coordinate system. Because the position between license plate and vehicle is fixed, then we get target vehicle translational velocity and three axis rotation speed (change rate of pose) of three-dimensional space in vehicle coordinate system.

4 Experimental means and simulation experiment for collision warning positioning

In order to verify the performance of auto warning system put forward in this paper based on license plate cooperative target positioning of monocular, the paper will also carry out the related experimental study on the basis of the simulation research. We design and develop image acquisition and processing system with DSP as the core. And the license plate location and early warning system is developed with FPGA as the core. Sensor is a forward-looking monocular camera and a rear view monocular camera. Early warning system will use two methods of the voice alarm and flashing alarm. The experimental platform will be tested on ordinary roads and highways.

For collision warning location technology, we carry out simulation experiment. Camera concrete focus parameter is chosen as 9.86. Size of image is 512×512. The vehicle images are firstly captured by the camera. Figure 5 is vehicle images acquired from camera. We can locate license plate on automobile image using image processing method. Figure 6 is the license plate images obtained through image positioning, which are surrounded by red rectangle. We extract four edges of license plate. Then, we use four vertexes of license plate image edges for location. Four vertexes are surrounded by green signs. Positioning for the car license plate image is done using the license plate image vision location method introduced in this paper. Experiments show that collision warning location method can determine special information from the target vehicle coordinate to the camera coordinate system. Table 1 is automobile positioning results for license plate images. Simulation experiment for collision warning location shows that it can realize license plate image location by the visual positioning method for four vertexes of license plate image.

Fig. 5
figure 5

Vehicle image acquired by camera

Full size image
Fig. 6
figure 6

License plate image located on automobile image using image processing method

Full size image
Table 1 Automobile positioning results for license plate images
Full size table

5 Conclusions

Vehicle collision warning system can judge the danger; timely remind the driver to make the appropriate action to avoid vehicle collision. Study on this system can help the driver to judge, prompting warning, improve driving safety, and has broad application prospects. This paper proposes the design and research method of automobile anti-collision warning system based on monocular vision sensor with license plate cooperative target. In addition to the target vehicle positioning, it can also realize attitude determination. This technology eliminates the limits of road surface roughness and fluctuation. The paper presents an analytic solution of positioning method for the license plate frame image. The method uses four vertex characteristics of license plate frame image to locate and its speed is fast. At the same time, it has a unique solution. This method can be used to positioning for license plate frame. Experiment is done for collision warning location. The simulation results show that this method can locate the pose for license plate frame image.

License plate is regular shape, uniform, with identity recognition function markers on the automobile body. In the previous research on automotive collision warning and intelligent vehicle, we have not seen the research methods similar to the method introduced in this paper. The research enriches automobile anti-collision technology and theory of intelligent vehicle technology. It can also provide an auxiliary method for navigation and obstacle avoidance research for unmanned vehicle. It has certain scientific research value.