Keywords

1 Introduction

Fundamental matrix plays an important role in epipolar geometry since it contains all geometric information about the relative transformation between two images. Fundamental matrix estimation is based on solving a homogeneous linear system in which each linear equation is formed by a pair of correspondence feature points. When the data is free of outliers, the nonlinear seven-point method [1] or linear eight-point method [2] is used to recover the fundamental matrix from the linear system via least squares. In practice, however, the outliers or large measurement errors are inevitable due to the inconsistency in feature extraction and matching process. Therefore, a robust algorithm that is resilient to outliers is vital for fundamental matrix estimation.

A large number of robust estimation approaches have been proposed to alleviate the influence of outliers to the fundamental matrix estimation. The M-estimator method [1, 3] reduces the effect of outliers by applying weight functions to transform the problem to a weighted least squares problem. However, the approach needs a good initial estimation and only works under low percentages of outliers. LMedS [4, 5] evaluate each estimation in terms of the median symmetric epipolar distances of the point correspondences and choose the one which minimizes the median error. The method does not need to know the percentage of outliers, but it is very time-consuming. RANSAC is a very popular robust algorithm for fundamental matrix estimation [6, 7]. The algorithm use minimal points set to estimate an initial guess. Then, the confidence of the estimation is established by testing each point correspondence against the hypothesized model; and an inliers set is determined by choosing points that have error below a given threshold. Next, a new fundamental matrix is estimated by the inliers set. Iteratively, the RANSAC algorithm attempts to find a solution that maximize the amount of the inlier set. In the last two decades, several RANSAC based algorithm have been proposed.

PROSAC algorithm [8], by taking into account additional information of the quality of the errors of the point matches, largely reduces the number of iterations. The MLESAC algorithm [9] maximizes a likelihood which is a mixture model of normal distribution for inliers and uniform distribution for outliers. The parameter of the model is estimated by expectation maximization. MAPSAC [10] maximizes the posterior estimation of the fundamental matrix and matches. Feng et al. [11] proposed a robust estimation method that measure the point matches by means of 2D reprojection error. The algorithm uses the mixture models of Gaussian and Uniform distributions. Huang et al. [12] improved the RANSAC algorithm by means of constructing a voting array for all the point correspondence pairs to record the consistency votes for each pair from a number of fundamental matrix estimations to better identify the outliers. Carro et al. [13] proposed a new robust method by combining the PROSAC and LMedS algorithms. All the above RANSAC-based approaches basically concentrate on the evaluation criterion of the estimation instead of the iteration step. Although these methods can achieve a better estimation of the fundamental matrix, the time cost issue is still not solved. The iterations increase greatly with the increase of outlier percentages, as a result, much more computation time is required.

In this paper, we adopt reprojection error, rather than the widely used algebraic error, to evaluate the confidence of the fundamental matrix. By assuming Gaussian image noise, it is shown that the reprojection error of point correspondences can be described by a chi-square distribution, and the outliers usually yield very large reprojection errors. Thus, the outliers can be simply eliminated using the 3-sigma principle. Based on this observation, a fast and robust algorithm is proposed for the fundamental matrix estimation. With this strategy, the inlier set can be robustly established in only two steps. Compared to other robust algorithms, the proposed technique is not only very efficient, but also extremely accurate. The algorithm is validated by extensive experiment using both synthetic and real image data.

2 Robust Fundamental Matrix Estimation

2.1 Eight-Point Linear Algorithm

Fundamental matrix is estimated from a set of point correspondences between two images. Given an image pair \({\text{ I }}\) and \({\text{ I }}'\), suppose \(\mathbf {x}_i \in \text{ I }\) and \(\mathbf {x}'_i \in \text{ I }'\) are a pair of corresponding homogeneous points between the two images. Then, the fundamental matrix \(\mathbf {F}\) satisfies the following equation.

$$\begin{aligned} \mathbf {x}'^T_i \mathbf {F} \mathbf {x}_i=0 \end{aligned}$$
(1)

where the fundamental matrix is a \(3\times 3\) homogeneous matrix defined up to scale. Each pair of point correspondence yield one linear constraint the entries of \(\mathbf {F}\). Thus, the fundamental matrix can be linearly estimated from eight point pairs. When more correspondences are available, the fundamental matrix can be estimated via least squares.

Fig. 1.
figure 1

(left) The histogram distribution of the real added noise and outliers. (right) The histogram distribution of the reprojection errors.

Full size image

2.2 Algebric Error and Reprojection Error Evaluation

After obtaining an estimation of the fundamental matrix, an error measure can be evaluated for each pair of point correspondence. The most commonly used criterion is the algebraic error defined as \(e_{a}(i)=\mathbf {x}'^T_i \mathbf {F} \mathbf {x}'_i\). This definition is simple, however, it does not have any geometric meaning.

Based on the initially estimated fundamental matrix, a pair of camera matrices can be recovered, and thus, a perspective 3D reconstruction of all corresponding points is obtained via triangulations [14]. Then, the reconstructed 3D points can be reprojected back to the two images via the camera matrices. Suppose \(\mathbf {\hat{x}}_i\) and \(\mathbf {\hat{x}}'_i\) are the reprojected images of point i, the 2D reprojection error of the corresponding point is defined as

$$\begin{aligned} e_{r}(i) = \frac{1}{2} \sum \Vert \mathbf {x}_i-\mathbf {\hat{x}}_i\Vert ^2_F + \Vert \mathbf {x}'_i-\mathbf {\hat{x}}'^2_i\Vert _F, \;\;\;s.t.\;\;\; \mathbf {\hat{x}}'^T_i \mathbf {F} \mathbf {\hat{x}}_i=0 \;\;\; \forall i \end{aligned}$$
(2)

The 2D reprojection error is proven to be more superior to other geometric errors. Optimal triangulation [14] is a linear triangulation method which converts the least-square function to a one parameter function and finds a global optimal solution.

2.3 Outlier Detection Strategy

The image noise is normally modeled by Gaussian distribution. Under this assumption, it can be verified that the reprojection error should follow chi-square \(\chi ^2\) distribution, as shown in our simulation result Fig. 1.

In Fig. 1, the added noise is Gaussian, while the added outliers are some random points with large standard deviations. As shown in the figure, the points located at the leftmost and rightmost areas are added outliers. Through extensive simulations, we found that the reprojection errors of outliers are usually greatly larger than those of inliers. This result is also support by our early study on structure from motion [16]. As a result, these outliers can be identified using 3-sigma principle. Points with reprojection errors larger than the triple variance of all the reprojection error can be classified as outliers. Based on robust statistics [15], we can obtain a robust standard deviation of the reprojection errors by the following equation.

$$\begin{aligned} \sigma = 1.4826 \Big (1+\frac{5}{n-q}\Big )\text {median}_i|e^r_i| \end{aligned}$$
(3)

The above equation is the median absolute deviation (MAD) scale estimate [15]. The first number is obtained from the inverse of the cumulative normal distribution, and the term \((1+\frac{5}{n-q})\) is the finite sample correction factor with the total number of parameters \(q=8\) and n the total number of features. The details of the derivation can be found in [15]. According to the distribution model, we distinguish the inliers from their reprojection errors of each pair of corresponding points. The points whose reprojection errors are less than 3\(\sigma \) are deemed as inliers, since 99.14% of the data points lies within 3\(\sigma \) under the assumption of the Gaussian distribution error model.

Fig. 2.
figure 2

Evaluation results from synthetic data. (left) Outlier detection rates; (middle) reprojection errors; and (right) computation time (second) by different algorithms.

Full size image

2.4 Outline of the Proposed Approach

Based on the above discussion, the implementation details of proposed approach is summarized as below.

  1. 1.

    Normalize the coordinates of all matching points;

  2. 2.

    Estimate an initial fundamental matrix using eight-point linear algorithm;

  3. 3.

    Compute the reprojection error and determine an outlier threshold;

  4. 4.

    Re-estimate the fundamental matrix using the inliers detected in step 3;

  5. 5.

    Repeat the steps 3 and 4 one time to refine the inlier set;

  6. 6.

    Estimate the optimal fundamental matrix using the inliers obtained in step 5.

3 Evaluations on Synthetic Data

The proposed algorithm was evaluated on synthetic data and compared with previous algorithms. During the simulation, 200 space points were randomly generated with a cube of [10, 10, 10], and two images were produced from these points. The image size is \(800 \times 800\); and the focal lengths of the two cameras are set at 800. Gaussian noise with zero mean and 2 pixels standard deviation is added to each pixel. Outliers are simulated as Gaussian noise with large standard deviation (greater than 8 pixels in the test); and they are randomly added to part of the image points. The percentage of outliers varies from 5% to 30% in a step of 5%.

We evaluated and compared the performance of the proposed algorithm with three popular previous algorithms proposed in [12, 13], and [11], which are named as Ransac1, Ransac2, and Ransac3, respectively. The evaluation criteria include outlier detection rate, final reprojection error, and computational cost. 200 independent trials are carried out under each configuration in order to yield a more meaneaingful statistical result. Figure 2 shows the experimental results, from which we can see that the proposed algorithm obviously outperforms all other three approaches in terms of the outlier recall precision and the reprojection error. The computational cost of the proposed algorithm is also noticeably lower than the Ransac1 and Ransac3 algorithms.

4 Evaluations on Real Images

The proposed algorithm has been evaluated using extensive real images, and only one result is reported here due to limited space. Two images from the “Model House” sequence (http://www.robots.ox.ac.uk/~vgg/data1.html) are used in the experiment, as shown in Fig. 3. The points marked in red circles denote the point correspondences between the two images. We randomly select different ratios of the matched points and add large random noise onto them to simulate the outliers. Figure 4 shows the outlier detection rates, final reprojection errors, and computational cost with respect to different percentage of outliers by different algorithms. We can see from Fig. 4 that the results are similar to those on synthetic data. The proposed algorithm yields obviously better results than other three approaches in the real image test.

Fig. 3.
figure 3

Two images of a model house with matching result shown in red circles.

Full size image
Fig. 4.
figure 4

Evaluation results on model house images. (left) Outlier detection rates; (middle) reprojection errors; and (right) computation time (second) by different algorithms.

Full size image

5 Conclusion

In this paper, we have proposed a new robust algorithm for fundamental matrix estimation based on reprojection errors. Compared with previous algorithms, we adopted a more meaningful error criterion to evaluate the confidence of the estimated fundamental matrix. With the new outlier detection strategy, the outliers can be identified from all pairs of point correspondences in two steps, leading to a more robust and more accurate estimation of the fundamental matrix. Compared to the RANSAC-based algorithms, the proposed algorithm can find the optimal solution in two steps.