An Image Mosaic Method Based on Deviation Splitting and Transferring for Vehicle Panoramic System

Abstract

A method for look-around image mosaic is proposed, in order to solve the problem that dislocation distortion still exists in overlap region in terms of panoramic system. Firstly, the principal and construction difficulties of panoramic system are analyzed. On this basis, combined with the common panoramic image stitching method, the cause for poor image stitching is analyzed. Afterwards, with the purpose of reducing distortion in overlap region, a method for panoramic image mosaic by dividing left and right images to transfer and split original stitching deviation is proposed. In addition, the detailed implementation plan is given. The proposed stitching method is realized by MATLAB programming, and the experiment shows that the dislocation distortion is less than conventional method. The proposed panoramic image mosaic method is small-dislocation and non-blind, thus it can satisfy the requirement of human vision and contribute to panoramic system’s realization.

1 Introduction

With the development of economic globalization, the world’s car ownership has exceeded 10 billion vehicles. However, at the same time, the frequency of traffic accidents is increasing year by year. In recent years, “pilotless automobile”, “new energy”, “around view” and other technologies have become the research boom in the automotive industry. Automobile safety technology is developing towards the direction of intelligence and systematization. Among them, the car panorama system can avoid a series of road safety hidden trouble caused by the obstruction of sight and improve the safety of driving. The key to the application of the system is the stitching technology of fisheye image.

At present, there are many R & D institutions for panoramic system technology. For example: In order to achieve the full coverage of small cars, buses and construction vehicles, Chery Automobile Co., Ltd. and Microelectronics Institute of Chinese Academy of Sciences are working together to carry out the research of “vehicle 360 degree environmental reconstruction system” [1,2,3]. In the market, the main products are the following categories: Land Rover as a representative of the four-bit image split-screen display system; Nissan AVM (Around View Monitor) based on seamed stitching system; Micron, Delphi and Volkswagen as the representative of the Seamless stitching, which can eliminate the blind seam region, but the existence of splice marks and distortion dislocation is obvious. The panoramic system of Nissan X-Trail and Delphi BEV Gen1.1 as shown in Fig. 1.

Fig. 1.

The panoramic system of Nissan X-Trail and Delphi BEV Gen1.1, (a) Nissan X-Trail(2012), (b) Delphi BEV Gen1.1

Presently, in consideration of problems such as distortion in the overlapping area of the panoramic image, there are two solutions:

• (1) “Avoidance” strategy. Taking the highest market share of Nissan AVM system as an example. In order to avoid the problem of poor display of the system caused by the dislocation distortion of the overlapping area, the Nissan AVM system had set the gap in the overlapping area, and the blind spot was introduced at the same time.

• (2) “Distortion” strategy. The typical method is to use a large number of distortions in the fourth images to achieve the effect of the image in the overlap area without dislocation, such as the three spline interpolation method is used as the distortion of the calculation of the fourth images.

However, all of the above methods are Sequential splicing methods, such as: left and right images are spliced with the front image respectively, then the rear image is spliced with the left image. In the process of image mosaic and the image preprocessing stage, the deviation will cause the distortion of the fourth image splicing. In this paper, a new method is proposed to realize the image stitching by dividing the left and right images, and the experimental results are compared with the conventional stitching method.

2 Image Preprocessing

The process of image preprocessing is shown in Fig. 2. The panoramic system collects the images through the 4 fisheye cameras installed in the car’s front, rear, left and right. With image preprocessing, image mosaic and image fusion [4, 5], panoramic image can be displayed. The main purpose of image preprocessing is to restore it to ordinary perspective image and transform the ordinary perspective image into a top view image according to the splicing requirements of the 2D panoramic system.

Fig. 2.

The process of image preprocessing

The hyperboloid model proposed by Scaramuzz is adopted here to calibrate the fisheye camera and correct the perspective image [6]. The corrected perspective image is converted to the top view image suitable for subsequent image stitching by transformation matrix. In this paper, the projection transformation scheme based on two transformations [7] is adopted, the results are shown in Fig. 3.

Fig. 3.

Image preprocessing process diagram, (a) Fisheye distortion image, (b) General perspective image, (c) Top view image

3 Image Mosaic Based on Deviation Splitting and Transferring

With the purpose of reducing distortion in overlap region, a method for panoramic image mosaic by dividing left and right images to transfer and split original stitching deviation is proposed. The following is the whole process of panoramic image mosaic, including image preprocessing, image registration, image stitching and image fusion.

3.1 Extraction of Feature Points in Image Overlap Region

In this paper, Harris corner detection [8] is used to extract the corner points of the grid plate in sequence. It is shown in Fig. 4.

Fig. 4.

Feature points extraction of overlap regions diagram (Color figure online)

The red dots in the figure are the feature points detected by corner points. The pt1 and pt2 are set as the pixel coordinate matrices of the front image corner points and the left image corner points respectively. The characteristic corner pair is (pt1, pt2). The feature points of the overlap regions in the four images can be extracted by using this method, and get the characteristic corner pairs of the overlap regions: (pt3, pt4), (pt5, pt6), (pt7, pt8).

3.2 Image Segmentation and Registration

According to the formula (1), the registration matrices can be calculated separately. Its formula can be expressed as:

$$ \left\{ {\begin{array}{*{20}c} {pt1 = H1 \times pt2} \\ {pt3 = H2 \times pt4} \\ {pt5 = H3 \times pt6} \\ {pt7 = H4 \times pt8} \\ \end{array} } \right. $$

(1)

Here, H1 ~ H4 represent the registration matrices between images; pt1 ~ pt8 represent pixel coordinate matrices.

As the four-way overlooking image requires a wide range of vision coverage, so there is a strong blooming in the image boundary of the four-way overlooking image. This phenomenon limited the automatic extraction of corners, in order to solve the problem, manual extraction is adopted in this paper. An initial corner is determined by clicking the mouse, and then the corner points in the preset range are detected by the method of Harris corner detection. The experimental results show that the proposed method is suitable for fuzzy region. In this paper, the RANSAC method is used to purify the extracted corner points after corner extraction. The corner points with large deviation are eliminated, and the calculation precision of the registration matrix is improved effectively.

The left and right images are divided into two parts respectively after the completion of the registration matrix calculation, which are recorded as left_up, left_down, right_up and right_down. The registration of the upper and lower two parts can be realized according to the calculation of the corresponding registration matrix. The left_up image is registered to the front image by matrix H1, the right_up image is registered to the front image with the matrix H2, the left_down image is registered to the rear image with the matrix H3, the right_down image is registered to the rear image with the matrix H4. The effect diagram is shown in the Fig. 5(a).

Fig. 5.

The effect diagram of image segmentation and registration, (a) The schematic diagram of registration, (b) The schematic diagram of cutting seams

The seam diagram is shown in Fig. 5(b). The seams are formed by cutting the overlap regions of the registration images. The diagonals of the overlap regions are set as seams for the registration of left_up, right_up and front images. It sets the rear image height at 1/2 of the overlap region as seams for the registration of left_down, right_down and rear images, which takes into account that the left and right fisheye cameras are far away from the rear of the car, the overlap region of the left, right and rear images is fuzzy seriously. In the figure, W1 and W2 are the width of left_down and right_down images. S represents half of the height for the rear image. It sets the height and width of front, left_up, left_down, right_up, right_down, rear images as (h1,w1),(h2,w2),(h3,w3),(h4,w4),(h5,w5),(h6,w6) respectively after cutting. In addition, the pixel coordinates of the top left corner point of each cropped image are (1, 1). After cutting and image registration, the registration effect is shown in the Fig. 6(a).

Fig. 6.

The schematic of inclination level adjustment and image mosaic, (a) Schematic diagram of image registration, (b) The schematic of inclination level adjustment, (c) Ideal segmentation effect of image mosaic

Thus, the seam of the front and left_up images can be expressed as:

$$ \left\{ {\begin{array}{*{20}l} {front:v = \frac{{w_{2} - 1}}{{h_{1} - 1}} \times u + \frac{{h_{1} - w_{2} }}{{h_{1} - 1}}} \\ {left\_up:v = \frac{{w_{2} - 1}}{{h_{1} - 1}} \times u + \frac{{h_{1} - w_{2} }}{{h_{1} - 1}}} \\ \end{array} } \right. $$

(2)

The seam of the front and right_up images can be expressed as:

$$ \left\{ {\begin{array}{*{20}c} {front:v = - \frac{{w_{4} }}{{h_{1} - 1}} \times u + \frac{{w_{4} - w_{2} }}{{h_{1} - 1}} + w_{1} } \\ {right\_up:v = \frac{{1 - w_{4} }}{{h_{1} - 1}} \times u + \frac{{h_{1} \times w_{4} - 1}}{{H_{1} - 1}}} \\ \end{array} } \right. $$

(3)

The seam of the rear and left_up images can be expressed as:

$$ \left\{ {\begin{array}{*{20}l} {rear:v = \frac{{2 \times \left( {w_{3} - 1} \right)}}{{2 - h_{6} }} \times u + \frac{{2 - w_{3} \times h_{6} }}{{2 - h_{6} }}} \hfill \\ {left\_down:v = \frac{{2 \times \left( {1 - w_{3} } \right)}}{{h_{6} - 2}} \times u + 1 - \frac{{\left( {1 - w_{3} } \right)\left( {2 \times H_{3} - H_{6} } \right)}}{{h_{6} - 2}}} \hfill \\ \end{array} } \right. $$

(4)

The seam of the rear and left_down images can be expressed as:

$$ \left\{ {\begin{array}{*{20}l} {rear:v = \frac{{2 \times w_{5} }}{{h_{6} - 2}} \times u + w_{6} - \frac{{w_{5} \times h_{6} }}{{h_{6} - 2}}} \hfill \\ {right\_down:v = \frac{{2 \times \left( {w_{5} - 1} \right)}}{{h_{6} - 2}} \times u + 1 - \frac{{\left( {w_{5} - 1} \right)\left( {h_{5} - h_{6} + 1} \right)}}{{h_{6} - 2}}} \hfill \\ \end{array} } \right. $$

(5)

3.3 Horizontal Adjustment of the Inclination Angle

It can be seen from Fig. 6(a) that the cumulative deviation transfers from the fourth image of the seam to the split sections of the left and right images after division, which is mainly reflected in the tilt angle of image segmentation line relative to the horizontal line. This paper presents a method of horizontally adjusting the segmented image by decreasing the angle of rotation line by line. This technique not only enables the split image to be adjusted horizontally, but also minimizes the pixel voids caused by the rotation.

In this paper, we take the level adjustment of left_down image as an example to illustrate this problem. The principle is shown in Fig. 6(b). The steps are as follows. Firstly, the declination of the upper edge of the left_down image and the horizontal line is calculated, which is set to θ. The height of left_down and rear images are set to H2 and H1 respectively, and the vertical center line of the left_down image is the center of rotation for every line. Finally, the non-overlapping part of the left_down image with the rear image will be adjusted line by line. Taking the line of the left_down image as an example, the height of which distances from the top edge of the rear image is h. The rotation angle β is shown in formula (6).

$$ \beta = \frac{\theta }{H2 - H1} \times h $$

(6)

3.4 The Mosaic of the Registration Image

The inclined edge will be close to the horizontal line after the horizontal adjustment of the image, which can make it easily for the image mosaic. The translation distance of the upper and lower two images in vertical and horizontal directions can be calculated through the relevant image coordinates of the upper and lower two parts of the image after the horizontal adjustment of deflection angle. The ideal effect is shown in Fig. 6(c).

4 Comparative Analysis of Experiments

In this paper, the commonly used sequential splicing method and the segmentation and splicing method which based on the deviation transfer and split are realized by MATLAB programming. The ideal pixel size of a single grid in the image is 20 × 20 pixels after image preprocessing. The implementation process is as shown in Fig. 7.

Fig. 7.

The algorithm process of the comparison experiment

Figure 8(a) and (b) represent the inclination of the image before and after the horizontal adjustment respectively. It is obvious that the inclination of the right_down image is well adjusted horizontally. Figure 8(c) and (d) represent the effect diagrams of sequential stitching and image splicing method based on the deviation splitting and transferring.

Fig. 8.

The effect of stitching results comparison, (a) Before horizontal adjustment, (b) After horizontal adjustment, (c) The sequential stitching, (d) Segmentation and splicing method (Color figure online)

In this paper, the deviation of the pixels between the feature points in the red rectangle region in Fig. 8(c) and the variation of the relevant division points in Fig. 8(d) before and after the image stitching are compared. The results are shown in Table 1. It can be found that the method proposed in this paper realizes the use of the small deviation at the left and right partitions instead of the large deviations in the seam.

Table 1. Comparison of pixel deviation of two splicing methods

5 Experimental Results and Conclusions

Before the experiment, a simulation experiment platform of panoramic system is constructed. Guangzhou Zhiyuan electronic development of large wide-angle fisheye cameras are used on the platform, whose working voltage is DC 3.3 V, the image resolution is 480 × 640, the working level angle of view is 185 ± 5 degrees and the vertical is 148 ± 5 degrees.

As the acquisition of the fisheye camera is analog signal, therefore, the jovision audio and video capture card JVS-C301 which supports NTSC analog video signal and second development of program to transform analog signal into digital signal are used, the frame rate of the capture card is 25 fps. Considering that the video analog signal of the camera is weak, and it is easy to be disturbed, the coaxial cable with strong anti-interference ability and low signal weakening is adopted in the connection between the video capture card and the camera. In addition, four camera mounting brackets are designed, which can adjust the height and angle freely and simulate the camera installation and image acquisition according to the physical size of different models. The experimental hardware platform is shown below (Fig. 9).

Fig. 9.

Hardware platform of experiment, (a) Four way super wide angle fisheye camera for vehicle, (b) Video capture card, (c) Platform analog system, (d) Tilt angle adjusting mechanism

In this experiment, we use C# programming to construct the software experiment system. In the image processing section, OpenCV is used to carry on the realization of the correlation algorithm. The software system is mainly divided into algorithm test module and system real vehicle running module, which can realize the debugging, experiment of the algorithm and real vehicle running of system.

The results of panoramic image stitching experiment are shown in Fig. 10. The image fusion scheme used in this paper is the gradual and progressive weighted fusion [9].

Fig. 10.

The impression images of panoramic image stitching experimental results, (a) Fisheye distortion image in real scene, (b) Transformation diagram of overhead view, (c) The effect image of panoramic image

It can be seen from Fig. 11 that compared with the Mercedes-Benz panoramic system, the splitting method based on the deviation transferring and splitting proposed in this paper have achieved the expected target of reducing the dislocation distortion and have better visual effect.

Fig. 11.

Comparison of visual effects of panoramic system, (a) Benz 2013 annual panorama system GL, (b) Benz 2014 annual panorama system ML350, (c) The panorama system of this paper

In this paper, the results of the experiment show that the method of image mosaic based on deviation shift and split can effectively eliminate the dislocation and distortion in the panoramic image mosaic. In this method, the dislocation of the fourth image stitching is separated and transferred to the left and right images, and large distortion of the left and right images is avoided through the horizontal adjustment of the inclination. The scheme proposed in this paper achieves a better panoramic image mosaic effect.