1 Introduction

In recent years, traffic accidents occur frequently, resulting in a large number of people died every year. The China Statistical Yearbook 2017 [20] shows that only in 2016 it occurred 212,846 traffic accidents. The number of dead and injured reached to hundreds of thousands and property losses amounted to more than one billion CNY, which caused a huge threat to people’s life and property security [35]. According to the relevant statistics, part of traffic accidents are caused by the blind area of car, the main reason is that drivers cannot observe the surrounding environment completely driving which causes scraping, collision, rolling and so on [33]. To thoroughly eliminate the driver’s visual blind area, panoramic vision system is becoming more and more important [5, 7, 31, 40].

Panoramic view system involves a lot of sophisticated technology. In recent years, with the rapid development of science and technology, more and more researches about key technologies of it have presented. For image correction, Zhang [38] put forward a simple and efficient calibration method in 2000. Both the internal and external parameters of a camera are calibrated by a two-dimensional plane checkerboard, then the distortion parameters of the lens are obtained. Finally, a distorted image can be corrected. It is a very classical approach. But for fisheye lens, because of its large distortion, Zhang’s method may produce an unsatisfactory result. In 2003, Ying and Hu [36] proposed a fisheye lens correction method based on spherical perspective projection (SPP) constraint. The SPP model can be used to obtain the lens’s radial and tangential deformation parameters to correct the distortion of the fisheye image. Tu et al. [25] presented a high precision two-step calibration method combining global and local ideas, which using a quartic polynomial to fit camera global projection model, then the image is split into blocks for local linear fitting optimization to reduce the error. In [8], Lai et al. adopted a distortion correction method based on Bayer format image. According to the characteristics of the Bayer image signal, they designed a befitting Hermite interpolation algorithm to realize the distortion correction of lens, which has realized on the FPGA device, and formed a real-time prototype system. Similarly, the real-time performance of lens correction was also studied in [26]. Turturici et al. optimized the process from three aspects, which contained algorithmic design, data transfer and the implementation of hardware and software. Based on the above idea, they put forward a lens correction scheme that can be applied to different types of lens, and can run on a low power embedded device in real-time. To avoid computing too many camera parameters, Yeh et al. [34] proposed an accurate and simple method to correct the fisheye image based on the relationship between the formation distance of the image and the incident angle. Bawa et al. [1] invented a general framework, which used a non-iterative method to correct a barrel distorted fisheye lens and is able to form a top-down undistorted image. The method is fast and computable.

In terms of image mosaic and fusion, research enthusiasm has not been weakened. The SIFT algorithm proposed by Lowe DG [14] is one of the most famous image stitching algorithms. It has a high precision but with a low running speed, so as another famous fast matching algorithm, SURF was proposed in [2]. On the basis of the two classical algorithms (SIFT and SURF), a large number of image mosaic algorithms are proposed. Such as BRISK and ORB algorithm [10, 23]. Cai et al. [3] improved the SURF algorithm by using the support vector machine to predict the determinant value of the Hansen matrix, which can remove some wrong feature matching points. The perspective scale invariant feature transform and artificial bee colony were introduced in [37], which made up for the deficiency of traditional matching algorithm in processing some large viewpoint changes images. Most of the image stitching algorithms are running slowly. At the same time, the algorithms need to run on embedded devices in many scenes. So it is meaningful and indispensable to improve the image mosaic algorithm and utilize the parallel processing equipment [9, 29]. Zhou et al. [42] introduced a scheme for real-time mosaic of panoramic images. They transplanted the overall algorithm (image preprocessing, matching, stitching and fusion) to the FPGA device, whose high-efficiency calculating capability increased the performance of system greatly. In addition, Vourvoulakis et al. [28] increased the real-time performance of SIFT. They parallel processed the process of detecting and describing the feature points of SIFT on FPGA, and RANSAC algorithm was used to remove mismatching points, which guaranteeing the speed and accuracy. Dou et al. [4] reduced the SIFT extraction area by the fusion of wavelet filtering and Harris detection, which reduced the computation. In addition, He et al. [6] used Fast Fourier Transform Algorithm to convert images to frequency space, it was able to speed up the process of SIFT. The method run on GPU eventually. In [21], Dong et al. argued an image mosaic model based on BRISK algorithm. First, they got the rough matching points of image by BRISK algorithm, then the adjacent line segment matching was used, and finally acquired the exact matching of feature points by the probability statistical model. Victoria et al. [27] fused the SIFT and SURF based on fuzzy logic controller, and compared that with each original algorithm separately. It had the characteristics of low error and high noise resistance. For some complex stereoscopic images, a simple two-dimensional mosaic algorithm may not be applicable [15]. Yan et al. [32] expounded a mosaic method based on hybrid warping model. A unified homography transformation was performed on two images to be stitched, then the vertical gap between the two images was reduced through local refinement. Ultimately, an optimal stitching seam was found and stitched the images with the multi-frequency fusion algorithm. Unnatural stitching gaps may occur for images with different lighting. Zhang et al. [41] made the spliced image at the same illumination level by histogram matching. Then SIFT and RANSAC algorithms were used for matching key points roughly and removing false matching points, respectively. Ma et al. [16,17,18,19] have been conducting a series of in-depth researches on image registration methods in recent years, especially in feature point matching. They presented the locality preserving matching method, which is robust and fast. It is able to remove mismatches in feature matching, the base idea of which is the similarity of regional structure between the corresponding feature points [19]. Recently, with the advance in artificial intelligence, more and more researchers have applied it to feature extraction [11,12,13]. Rashid M et.al utilized deep convolutional neural network and SIFT point features achieve object detection and classification [22]. The neural networks methods have high degree of accuracy and also require higher time cost.

In our method, we focus on improving the key algorithms of generating vehicle panoramic image, which is able to effectively remove the serious distortion of fisheye lens and generate a panoramic image around the vehicle on embedded device compared to some existing technologies. The main contributions of this study are as follows:

  1. 1)

    The bilinear interpolation algorithm is used to optimize the SPP algorithm based on scanning line to corrected image. Meanwhile, we adopt a method by artificial marking to perform the inverse perspective mapping of the corrected image around the vehicle. The transformation matrix used is calculated by calibrating the selected points manually, which can reduce the time cost of system.

  2. 2)

    SURF, RANSAC and a weighted fusion algorithm are used for image mosaic and fusion, respectively. Which can balance the difference in the image seams caused by different brightness. Furthermore, the methods are applied to the real environment by transplanting and running on IMX6Q (an embedded device).

The structures of the rest on the paper are organized as follows. In Section 2, the proposed method is expounded in detail. Section 3 summarizes the analysis of the experimental results. Conclusions and further work are placed in Section 4.

2 Proposed method

2.1 Image correction

Due to the short focal length and wide field of view of the fisheye camera [30], it is essential to correct the fisheye image captured before stitching [24, 39]. For the SPP algorithm [38], whose efficiency and performance are optimized in this paper.

2.1.1 Efficiency optimization

A suitable ROI can greatly improve the efficiency of the subsequent algorithm. Therefore, we extract the effective part of the image to be processed by selecting the ROI. A four directions scanning method (namely ScanningMethod) is proposed to determine the ROI in fisheye image. Due to the wide field characteristic of fisheye lens, for a fisheye image, only the near circle figure in which is the effective area. The edges and corners of the fisheye image are some redundant black areas. Hence, locating the effective area of the fisheye image before correcting is a feasible means to accelerate the efficiency. The specific descriptions are as follows (taking the left boundary as an example):

  • Step1: Initiate parameters

The brightness difference set according to experience: G. The left, right, top and bottom of the fisheye image are represented by left, right, top and bottom, respectively. pixelmax and pixelmin represent the maximum and minimum pixel values of a column, respectively. pixelgray is the gray value of the current pixel. The current pixel’s RGB components are pixelR, pixelG and pixelB, respectively. pixelcenter and d are the center coordinates and radius of the fisheye image, respectively.

  • Step2: Scanning fisheye images from left to right column by column, pixelgray ← 0.3 ∙ pixelR + 0.59 ∙ pixelG + 0.11 ∙ pixelB

  • Step3: ifpixelgray > = pixelmaxthenpixelmax ← pixelgray

else ifpixelgray < = pixelminthenpixelmin ← pixelgray

  • Step4: when (Scanning a column of pixels to end) do

ifpixelmax - pixelmin > = Gthenleft ← current column number

  • Step5: In the same way, the values of right, top and bottom can be calculated.

  • Step6:pixelcenter ← ((left + right)/2, (top + bottom)/2), d ← max (left - right, top - bottom).

2.1.2 Performance optimization

For the pixel mapping module in image correction. Nearest interpolation, bilinear interpolation and bicubic interpolation are common interpolation algorithms. Considering the processing speed and effect of the algorithm, bilinear interpolation algorithm is used to obtain the final corrected image. The detailed steps are as follows:

  • Step1: Initiate parameters

Set imagewidth, imagelength and Z are the width, length and height of projection plane, respectively. Loop control variables: i and j. pixelimage is the pixel coordinate value of the fisheye image. Pixel*center and pixel*image represent the center coordinate and pixel values of undistorted image, respectively. Pixel^image is an integer value of pixel*image. α and β are the weights of bilinear interpolation.

  • Step2: Get pixelcenter and d by ScanningMethod.

  • Step3:pixel*center.x ← 1 ∙ Z ∙ pixelcenter.x / sqrt(d ∙ d - pixelcenter.x ∙ pixelcenter.x - pixelcenter.y ∙ pixelcenter.y)

pixel*center.y ← 1 ∙ Z ∙ pixelcenter.y / sqrt(d ∙ d - pixelcenter.x ∙ pixelcenter.x - pixelcenter.y ∙ pixelcenter.y)

  • Step4: forj < imagewidth && i < imagelengthdo

pixelimage.x ← i - imagelength / 2 + pixel*center.x.

pixelimage.y ← imagewidth / 2 – j - pixel*center.y.

pixel*image.x ← (d ∙ pixelimage.x / sqrt(pixelimage.x ∙ pixelimage.x + pixelimage.y ∙ pixelimage.y + Z ∙ Z)) + d.

pixel*image.y ← d - (d ∙ pixelimage.y / sqrt(pixelimage.x ∙ pixelimage.x + pixelimage.y ∙ pixelimage.y + Z ∙ Z))

  • Step5: Round down to prepare for interpolation: pixel^image ← (int)pixel*image.

  • Step6:α ← pixel*image.x – pixel^image.x

β ← pixel*image.y – pixel^image.y

  • Step7:pixel*image ← (1-α) ∙ (1-β) ∙ (pixel^image.x, pixel^image.y) + α ∙ (1-β) ∙ (pixel^image.x + 1, pixel^image.y) + (1-α) ∙ β ∙ (pixel^image.x, pixel^image.y + 1) + α ∙ β ∙ (pixel^image.x + 1, pixel^image.y + 1); end.

2.2 Inverse perspective mapping

After obtaining an undistorted image, there is a need to obtain a bird’s-eye view (i.e., inverse perspective mapping). The process of inverse perspective mapping is mainly to transform a perspective image into a bird’s-eye view image, which is also a process of homography transformation. So, it is necessary to know how an image is generated through a camera firstly. According to the image-forming principle of camera, the entire process from object to image consists of rigid transformation and perspective transformation. As shown in Eq. (1).

$$ {Z}_C\cdot \left[\begin{array}{c}u\\ {}v\\ {}1\end{array}\right]=A\cdot E\cdot \left[\begin{array}{c}X\\ {}Y\\ {}Z\\ {}1\end{array}\right],\mathrm{with}\ A=\left[\begin{array}{cccc}\raisebox{1ex}{$f$}\!\left/ \!\raisebox{-1ex}{${d}_x$}\right.& 0& {u}_0& 0\\ {}0& \raisebox{1ex}{$f$}\!\left/ \!\raisebox{-1ex}{${d}_y$}\right.& {v}_0& 0\\ {}0& 0& 1& 0\end{array}\right],E=\left[\begin{array}{cc}R& T\\ {}0& 1\end{array}\right] $$

Where A and E are the intrinsic and extrinsic parameters of the camera, respectively. As mentioned above, A represents perspective transformation matrix and B represents rigid transformation matrix. u, v are the pixel coordinates of the image, X, Y and Z are the world coordinates, dx, dy represent the physical dimensions of image pixels on the horizontal and vertical axes, respectively. ZC is the coordinate value in camera coordinate system, and u0, v0 represent the principal point of the image in pixel coordinate system. f is focal length, R and T are parameters matrices for rotation and translation, respectively.

In this way, the formula of inverse perspective mapping is able to be reasoned. The matrix named H is used instead of the complex transformation matrix in Eq. (1) to simplify it, which is called homography matrix. As shown in Eq. (2).

$$ {Z}_C\cdot \left[\begin{array}{c}u\\ {}v\\ {}1\end{array}\right]=H\cdot \left[\begin{array}{c}X\\ {}Y\\ {}Z\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& {h}_{12}& {h}_{13}\\ {}{h}_{21}& {h}_{22}& {h}_{23}\\ {}{h}_{31}& {h}_{32}& {h}_{33}\end{array}\right]\cdot \left[\begin{array}{c}X\\ {}Y\\ {}Z\end{array}\right] $$
(2)

In inverse perspective mapping, since both the original image and the result image are two-dimensional images, the value of Z coordinate in Eq. (2) can be set to zero. Since h33 in H is a known quantity as a homogeneous coordinate, H has eight unknowns, which requires eight sets of equations to be solved. A set of pixels can generate two equations, so at least four sets of pixels are needed. The specific calculation process of H will be described in detail later. After obtaining the H, it is necessary to determine the specific range of the inverse perspective mapping of the image. Because of the perspective effect, parallel lines in the world coordinate system will converge at one point in the image, namely vanishing point. According to experience, the inverse perspective mapping generally does not exceed the position of vanishing point. Therefore, the coordinates of vanishing point need to be calculated. Suppose there is a line in the world coordinate system (represented by L), whose direction is indicated by N(Nx, Ny, Nz). If Q is a point on L, whose coordinate is (XQ, YQ, ZQ), then the equation of L can be expressed as follows:

$$ F\left(\varphi \right)=\varphi N+Q $$
(3)

Where φ is the coefficient, and as φ becomes larger, F(φ) is closer to the point at infinity. Therefore, combining and calculating Eq. (2) and Eq. (3) to get Eq. (4).

$$ {Z}_C\cdot \left[\begin{array}{c}u\\ {}v\\ {}1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& {h}_{12}& {h}_{13}\\ {}{h}_{21}& {h}_{22}& {h}_{23}\\ {}{h}_{31}& {h}_{32}& {h}_{33}\end{array}\right]\cdot \left[\begin{array}{c}\varphi {N}_x+{X}_Q\\ {}\varphi {N}_y+{Y}_Q\\ {}\varphi {N}_z+{Z}_Q\end{array}\right] $$
(4)

When φ is infinite, the vanishing point coordinate (represented by (u+∞, v+∞)) will be obtained. It can be calculated according to Eq. (5).

$$ \Big\{{\displaystyle \begin{array}{c}{u}_{+\infty }=\frac{h_{11}{N}_x+{h}_{12}{\mathrm{N}}_y+{h}_{13}{\mathrm{N}}_z}{h_{31}{\mathrm{N}}_x+{h}_{32}{\mathrm{N}}_y+{h}_{33}{\mathrm{N}}_z}\\ {}{v}_{+\infty }=\frac{h_{21}{\mathrm{N}}_x+{h}_{22}{\mathrm{N}}_y+{h}_{23}{\mathrm{N}}_z}{h_{31}{\mathrm{N}}_x+{h}_{32}{\mathrm{N}}_y+{h}_{33}{\mathrm{N}}_z}\end{array}} $$
(5)

Assuming that H is known, (u+∞, v+∞) can be calculated. According to the vanishing point, the scope of the inverse perspective mapping will be determined. How to calculate H will be described below.

Homography matrix is the core of the inverse perspective mapping algorithm. As a common method, the camera calibration is able to obtain the parameters of H. However, this method depends on high accuracy when measuring external parameters. An improved inverse perspective mapping algorithm by artificial marking is carried out in this paper. First, a coordinate system model is established according to the plane where vehicle locates. As for each fisheye camera around the vehicle, four reference points are set up. Then, the coordinate values of the 16 points in the world coordinate system are manually measured multiple times, and the average value is taken as the final result. Similarly, the coordinate values in the image coordinate system corresponding to the reference points are also obtained. Finally, in each direction, the homography matrix of each fisheye camera can be obtained by the mapping relationship among the four pairs of points. Through four homography matrices, inverse perspective mappings of the four images around the vehicle will be completed.

To reduce the computation time, we build a same world coordinate system instead of creating it for each fisheye camera. The origin of the world coordinate is defined at the intersection between the line where the front and rear cameras are located and the line where the left and right cameras are located. The directions of the right camera and the front camera represent the x-axis and y-axis directions, respectively. For ease of observation and measurement during the actual operation, some special reference objects with distinct apex characteristics can be selected as reference points, such as black-and-white checkerboard. The specific implementations are as follows:

  • Step1: Initiate parameters

ps and pd are used to save pixel coordinates and world coordinates, respectively. H is the homography matrix. ib indicate bird’s-eye view image.

  • Step2:ps ← image pixel coordinates obtained by the fusion of measurement and calculation;

pd ← coordinates in the world coordinate system by artificial measurement.

  • Step3: Calculate H according to the mapping relationship between ps and pd.

  • Step4: Transform the corrected image to ib by H.

2.3 Image mosaic

For the direction of each camera, the above process is called, resulting in a bird’s-eye view around the vehicle. The SURF algorithm is applied to detect the feature points of bird’s-eye view images after this transformation. In order to achieve image mosaic, it is necessary to find the feature points which are matched between two images. Generally, Euclidean distance is used as a standard to measure similarity in the coarse matching process. In a pair of images that to be matched, one point to be processed is selected in one of them. And calculate the minimum and second minimum of Euclidean distance between this point and the point in another image. When the ratio of them is less than a threshold defined in advance (represented by r), two points are considered as a pair of rough matching points. For the mismatching that may be occurred during the rough matching process, The RANSAC algorithm based on block is used to eliminate these mismatches. In order to help speed up processing, the images to be processed are divided into blocks averagely. Only the image blocks containing the matching points are processed. A block with matching points is added to the dataset (represented by S) to be detected. RANSAC is preferentially applied to image blocks with more matching points in S. And select four pairs of points randomly to calculate a transformation matrix (represented by η). The point within a certain range (represented by ε) of the transformation matrix is called inner point. When the number of internal points reaches the maximum, η is optimal. Following this, the algorithm is used to match the four bird’s-eye images, separately. The basic steps are as follows:

  • Step1: Initiate parameters

S ← image blocks to be processed. A is an inner point set and Ag is the best case for A. η ← a transformation matrix model. n is the number of iterations. m is an iterative control variable. ε ← a certain threshold. δ indicates error.

  • Step2: select four pairs of matching points randomly from S, and calculate one model of η.

  • Step3:δ ← projection error between η and all data in S. ifδ < εthenA ← all data in S which satisfy δ < ε.

  • Step4: ifnumber of A > number of AgthenAg ← A.

  • Step5: ifm++ < nthen go to Step2

else returnη, end.

Because of the specificity of vehicle panoramic view system, the images around the car are captured from four fisheye cameras with different directions and areas. Therefore, different illumination may occur in different areas, resulting in uneven brightness of the images. In this case, there will appear obvious brightness differences in the fusion area. The weighted average method is used to deal with the case. The main idea is to determine the pixel values of the join points by the pixel values of the two images to be stitched together. It can greatly increase the correlation between stitching images. Usually, a coefficient (represented by K) is necessary, K ∈ [0, 1]. So how to calculate the pixel value (represented by Pixel) in the fusion area is shown in Eq. (6).

$$ Pixe l=K\cdot Pixe{l}_1+\left(1-K\right)\cdot Pixe{l}_2 $$
(6)

Where Pixel1 and Pixel2 represent the pixel values of position at the seam on the two images to be stitched, respectively. K = S1 / (S1 + S2), S1 and S2 represent distances between the point in the fusion area and images to be stitched, respectively.

3 Experiment and analysis

To verify the proposed method, all experiments are programmed on computer with 3.4 GHz Intel Core i7 CPU and 16GB RAM using Ubuntu 12.04.

Fig. 1 shows the correction result of fisheye image. Figure 1 (a) is the original image captured by fisheye camera. Figure 1 (b) and Fig. 1 (c) are the undistorted images by Zhang’s algorithm and our method, respectively. It can be seen that the correction effect in Fig. 1 (b) is not excellent, and there is still obvious distortion at the corners of the corrected image. Moreover, compared with the original image, Fig. 1 (b) greatly reduces the field of vision of the fisheye image. It will seriously affect the adjacent overlapping areas of the overlooking image after the inverse perspective transformation, and thus may increase the inaccuracy of image mosaic. Our method based on SPP is used to correct fisheye image, and optimize the corrected image through the bilinear interpolation. Figure 1 (c) has a wider view than Fig. 1 (b). Compared with Zhang’s algorithm, our method has better effect and the result image is smoother.

Fig. 1
figure 1

Comparison between original image and undistorted image. a Original image. b The result of Zhang’s algorithm. c The result of our method

Full size image

In order to calibrate the fisheye camera, we select an alumina chessboard as calibration board (size: 200 mm * 200 mm, square array: 12 * 9, side length of a square: 15 mm, precision error: ±0.01 mm). Figure 2 is a part of the images taken by fisheye camera, which are to be calibrated.

Fig. 2
figure 2

Original images

Full size image

The result of pixel error analysis for the calibration results is shown in Fig. 3, where Fig. 3 (a) represents the overall mean error. We can set the threshold to about 0.16 pixels, the images with a mean error of more than threshold are deleted as shown in Fig. 3 (b).

Fig. 3
figure 3

The result of pixel error analysis. a The original overall mean error. b Delete the image with too much error

Full size image

Fig. 4 shows the position relationship between the fisheye camera and calibration board. Figure 4 (a) and (b) are the extrinsic parameters visualizations centered on the camera and calibration board, respectively.

Fig. 4
figure 4

The position relationship between the camera and calibration board. a Camera is the center. b Calibration board is the center

Full size image

Camera parameters are obtained through calibration, as shown in Table 1.

Table 1 Camera parameters
Full size table

The distortion correction of fisheye image is realized by using radial distortion and intrinsic matrix composed of focal length and principal point. In addition, the distortion of fisheye lens which are used in our system is too large, so the third-order radial distortion coefficient is employed. Compared with the general second-order coefficient, which can reduce the calibration errors and has better correction effects.

Then, we use the improved inverse perspective mapping algorithm to transform the corrected image. The resolution of corrected image is 720 * 576, so the points in the world coordinate system need to be processed according to Eq. (7).

$$ \left[\begin{array}{c}{X}_{new}\\ {}{Y}_{new}\\ {}1\end{array}\right]=\left[\begin{array}{ccc}1& 0& \raisebox{1ex}{$720$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\\ {}0& -1& 576\\ {}0& 0& 1\end{array}\right]\cdot \left[\begin{array}{c}{X}_{old}\\ {}{Y}_{old}\\ {}1\end{array}\right] $$
(7)

Where (Xnew, Ynew) is the new coordinate, (Xold, Yold) is the old world coordinate. After processing, the scale is reduced to half and final result is shown in Fig. 5 (c).

Fig. 5
figure 5

Comparison between undistorted image and bird’s-eye view image. a Original image. b Initial result of bird’s-eye view image. c Improved result of bird’s-eye view image

Full size image

As shown above, the improved inverse perspective transform algorithm has a good overlooking effect. Compared with the manual measurement method based on internal and external parameters by camera calibration, the proposed method only calculates the homography matrix directly. It does not need to fix the camera on the car and use the instrument to measure the camera height, pitch angle, yaw angle and other parameters artificially. Therefore, the measurement process of the improved method is simple and the error sensitivity is greatly reduced.

After overlooking transformation, it enters the image splicing stage of the whole system. In this paper, the method of controlling variable is used to compare the mainstream image stitching algorithm. And analyze them from three aspects: image blurring, illumination condition and algorithmic efficiency.

3.1 Image blurring

Manual processing of an image to change its blur strength, and recognize the feature points in the image through several popular algorithms. As shown in Fig. 6. Figure 6 (a) is a multi-level blurring process for the same image. From Fig. 6 (b) ~ (d), it is clear that SURF and SIFT algorithms can still accurately identify all kinds of feature points for a relatively blurred image, while BRISK algorithm can recognize most of the feature points, but it also loses some checkerboard corner points.

Fig. 6
figure 6

The detection of feature points by different algorithms. a Five different blur strengths of an image. b By SURF. c By SIFT. d By BRISK

Full size image

3.2 Illumination condition

The image is processed with different brightness, and the rest is the same as above. Figure 7 shows the result. Figure 7 (a) shows the brightness processing of the same image at different levels. From Fig. 7 (b) ~ (d), it can be clearly seen that SIFT algorithm recognizes the most feature points for a dark image, followed by SURF, and it can still accurately identify all kinds of feature points. As for BRISK algorithm, some checkerboard corner points are not recognized.

Fig. 7
figure 7

The detection of feature points by different algorithms. a Five different luminance of an image. b By SURF. c By SIFT. d By BRISK

Full size image

3.3 Algorithmic efficiency

In the vehicle panoramic view system, algorithmic efficiency is quite significant. It is not easy to achieve real-time effect when run all algorithms in an embedded system environment. Therefore, the algorithm with high speed should be preferred under the premise that stitching effect is acceptable. Additionally, for the three aspects above, the contrast results of different methods are shown in Fig. 8.

Fig. 8
figure 8

Comparison of the different algorithms. a Image blurring. b Illumination condition. c Algorithmic efficiency

Full size image

Where Fig. 8 (a) and (b) represent the number of feature points detected by different algorithms for the same image at different degrees of blur and brightness. Figure 8 (c) is the running time of the three algorithms. It can be seen from the results that the number of feature points detected will decrease with the decrease of image blurring. But at the same blurring, the number of feature points identified by SIFT and SURF is much higher than BRISK. When the brightness of image is changing, SIFT, SURF and BRISK still detect a certain number of feature points and have a good stability. In terms of algorithmic efficiency, SIFT has the biggest number of feature points, but the execution speed is the slowest. SURF has a relatively large number of feature points and runs faster compared with SIFT.

Considering the generality of experiment, we used these methods to conducted experiments on the set of images with different features respectively. Analysis results are listed in the form of tables and graphs. The number of feature points detected by SIFT, SURF and BRISK are shown in Table 2.

Table 2 The number of feature points detected by different algorithms
Full size table

For a more intuitive comparison of several methods, Fig. 9 shows the detection results of every methods, where horizontal axis is the number of images and ordinate represents the number of feature points. Black, red and blue lines in Fig. 9 represent the detection results by SIFT, SURF and BRISK, respectively.

Fig. 9
figure 9

Detection results of the different algorithms for different images

Full size image

From the above analysis, SURF is selected as the image mosaic algorithm on the basis of the comprehensive consideration of the detection effect and system time. In order to better reflect the effectiveness and practicability of the proposed methods, we have implemented the algorithms on PC and embedded platform, respectively. As shown below:

The results of the operation on PC are shown in Fig. 10. Four fish-eye images from the front, left, right and back of the vehicle form a 360-degree panoramic image through distortion correction, inverse perspective mapping, stitching and so on. Figure 10 (e) can clearly show the road information around the vehicle. Figure 11 shows the final result of vehicle panoramic view system which runs on IMX6Q. Display screen is divided into two parts, the left is the original image captured by fisheye camera and the panoramic image is on the right.

Fig. 10
figure 10

System running on PC. a Front camera. b Left camera. c Rear camera. d Right camera. e panoramic image

Full size image
Fig. 11
figure 11

System running on IMX6Q. a Front camera & panoramic image. b Rear camera & panoramic image. c Left camera & panoramic image. d Right camera & panoramic image

Full size image

It is easy to see from Figs. 10 and 11 that the proposed methods can be well applied to the vehicle panoramic view system. Furthermore, since vehicle panoramic view system contains abundant environmental information around the car, it is possible to integrate various functions based on which. Such as lane departure warning system, parking assistant system and so on. If the panoramic image includes lane lines and parking spaces, by the extraction and recognition of which, a highly integrated and low-cost driving assistant system will be formed. It can greatly increase the safety of driving.

4 Conclusion

An improved vehicle panoramic image generation algorithm is present in this paper. There are some improvements for image correction, inverse perspective mapping and image stitching. Statistics and analysis show that it has high accuracy and feasibility. Moreover, the proposed method has good reliability and robustness under different blurring and illumination. The method is transplanted and running on IMX6Q, which can accurately and effectively display the panoramic view around the vehicle for driver to improve the driving safety. In the future, we will enhance the real-time performance of the method and apply it in the real environment.