Intrinsic Calibration of a Camera to a Line-Structured Light Using a Single View of Two Spheres

Abstract

This paper proposes a novel approach to calibrate the intrinsic camera parameters from a single image, which includes the silhouette of two spheres and two ellipses generated by the intersection between the line-structured laser light and the two spheres. This approach uses the vanishing line of a plane and its normal direction to calculate the orthogonal constraints on the image of absolute conic (IAC). And this plane is formed by the camera center and two sphere centers. In addition, the pair of the circular points of the light plane is calculated from the generalized eigenpairs from the intersection between the light plane and the spheres. The intrinsic parameters of the camera can then be recovered from the derived orthogonal constraint and the pair of circular points on the IAC. Furthermore, the 3D positions of these two sphere centers under the camera coordinate can be recovered from the camera intrinsic matrix and then used to evaluate the accuracy of the camera intrinsic matrix. Experiment results on both synthetic and real data show the accuracy and the feasibility of the proposed approach.

1 Introduction

Camera calibration is a vital step for the task of 3D reconstruction in computer vision or photogrammetry in machine vision. The camera intrinsic parameters indicate the projective geometry from 3D objects under the camera coordinate to a 2D plane under the image coordinate system. The camera extrinsic parameters describe the rotation and translation information of a camera relative to the world coordinate system. Many approaches have been developed according to the dimension of the calibration objects [1], such as 3D objects [9, 10, 14,15,16], 2D plane [5, 13, 20], one-dimensional objects [1], or even without calibration objects [10], i.e. self-calibration. Due to its high accuracy and convenience, the most widely used calibration object is the 2D chessboard plane [13]. However, since a single plane can only provide two linearly independent constrains on calibration, pictures of multiple poses are required to be taken to fully calibrate the camera intrinsic parameters. Such procedures make the process of calibration manually cumbersome and time consuming. In order to overcome these problems, some common objects such as spheres or surfaces of revolutions (SOR) were proposed to replace the calibration board. In [16], the camera intrinsic parameters were calibrated with a single image which includes at least three spheres. In [10], the surface of revolution was used to recovering the intrinsic of a camera. Compared with the 2D calibration patterns, the images of 3D objects such as spheres or SOR can provide more constraints for intrinsic calibration.

Recently, these 3D calibration objects have been employed in calibrating the system with line-structured light vision sensors [2, 4, 18]. In [18], four balls placed on a flat board are used to calibrate the structured light plane. In [2], a method with a single object was proposed to calibrate the line-structured system. However, these methods generally involve calibrating the camera extrinsic parameters. A 2D calibration board was required to be firstly used to recover the intrinsic matrix in a separated step. 3D calibration objects were then introduced to estimate the light plane parameters [2].

This paper proposes a method to calibrate the intrinsic parameters from the intersection of a laser light plane with two spheres. Inspired by [16], the vanishing line of a plane and its normal direction were derived from the eigenvectors of the dual of the conic homograph and conic homograph generated from two sphere silhouettes, which formed the orthogonal constraint on the IAC. Besides, the intersection of the two spheres and the light plane form two co-planar circles. From generalized eigenvectors of their images can be broken down into constraints on the dual conic of the circular points from generalized eigenvectors. Together with the orthogonal constraints from the sphere silhouettes, the camera intrinsic parameters can then be recovered. From the camera intrinsic, the position of the sphere center can also be obtained.

The paper is organized as follows. Section 2 presents the fundamental theory of the image of the absolute conic. Section 3 introduces the orthogonal constraints from the silhouettes of two spheres. Section 4 relates the pair of circular points to the images of the intersection circles between the line-structured light plane and the spheres. The camera intrinsic parameters can then be estimated. Section 5 explains the process of recovering the 3D position of the sphere center. Experimental results and conclusions are then given in Sects. 6 and 7, respectively.

Throughout this paper, we use italic font (e.g., f) to represent a scale value, and use bold font (e.g., v, M) to represent a vector or a matrix.

2 The Image of the Absolute Conic

The absolute conic (AC) was firstly introduced in camera calibration by Faugeras et al. [6], and its projection in image is IAC which is an imaginary points conic, without real points lying on it. IAC is a 2D projection on the plane at infinity which is invariant under Euclidean transformations and can be written as

$$\begin{aligned} \mathbf {\varvec{\omega } = K^{-T}K^{-1}}= \left( \begin{array}{cccc} \omega _{1} &{} \omega _{2} &{} \omega _{3}\\ \omega _{4} &{} \omega _{5} &{} \omega _{6} \\ \omega _{7} &{} \omega _{8} &{} \omega _{9} \end{array} \right) , \end{aligned}$$

(1)

where K is the camera intrinsic matrix

$$\begin{aligned} \mathbf {K} = \left( \begin{array}{cccc} \alpha f &{} s &{} u_{0}\\ 0 &{} f &{} v_{0}\\ 0&{}0&{}1 \end{array} \right) . \end{aligned}$$

(2)

Here f is the focal length, $\alpha$ is the aspect ratio, ($u_0,v_0$) is the principle point and s is the skew. Generally, in order to calibrate the camera intrinsic, the image of the absolute conic (IAC) was firstly recovered, and then the camera intrinsic matrix K can be easily obtained by the Cholesky decomposition [7] of IAC.

3 Orthogonal Constraint from the Silhouettes of Two Spheres

The setup of a camera and a line-structured light system is shown in Fig. 1. The camera locates outside of the light plane. Two spheres, as the calibrate objects, are placed in front of the camera so that they are both visible to the camera. The line-structured light hits on both spheres and the intersection curves are partially visible to the camera. In Fig. 1, the solid part of the intersection curves are the visible outline to the camera, and the dashed curves are invisible part to the light source and cannot be seen by the camera.

Let the spheres be $\mathbf x ^T\,\mathbf B _i\mathbf x = 0$ (i = 1, 2), x = [x y z 1]$^T$ is any point on the spheres. $\mathbf B _i$ is a $4\times 4$ matrix

$$\begin{aligned} \mathbf {B} _{i} = \left( \begin{array}{cc} \mathbf {I} _{3\times 3}&{} \, \, \, \, \, \, -\mathbf {o} _{i}\\ -\mathbf {o} ^{T}_{i}&{} \, \, \, \, \, \, \mathbf {o} ^{T}_{i}\mathbf {o} _{i}-{r}^{2}_{i} \end{array} \right) , \end{aligned}$$

(3)

where the radius of the sphere is $r_i$ and the sphere center is $\mathbf o _i$. After projection, we can get two conics $\mathbf C _1$ and $\mathbf C _2$ in the image (see Fig. 2 for details), which are the outlines of these two spheres. These two conics are both $3 \times 3$ matrices and satisfied $\mathbf m ^T\mathbf C _i\mathbf m = 0$ (i = 1,2), where m = [u v 1]$^T$.

The vanishing line l of the plane formed by the camera and the two sphere centers can be recovered as one of the eigenvectors of the “dual of the conic homograph” $\mathbf H _{c}$ [3]

$$\begin{aligned} \mathbf {H}_\mathbf {c} = \mathbf {C}_\mathbf {2} \mathbf {C}^{-1}_\mathbf {1}, \end{aligned}$$

(4)

which passes through both sphere silhouettes $\mathbf C _1$ and $\mathbf C _2$ (see Fig. 2 for details). The other two eigenvectors are discarded since they only pass through one conic. Note that the eigenpairs of $\mathbf H _c$ are as same as the generalized eigenpairs of ($\mathbf C _1^{-1}$, $\mathbf C _2^{-1}$). Similarly, the vanishing point v of the plane normal can be obtained directly as one of the eigenvectors of the “conic homograph” $\mathbf H _{d}$ [3]

$$\begin{aligned} \mathbf {H} _\mathbf {d} = \mathbf {C}^\mathbf {-1}_\mathbf {2} \mathbf {C}_\mathbf {1}, \end{aligned}$$

(5)

Fig. 1.

The system setup. The two spheres $\mathbf B _1$ and $\mathbf B _2$, as the calibration objects, are placed in front of the camera so that they are both visible to the camera. The line-structured light hits on both spheres. The solid part of the intersection circles are the visible outline to the camera, and the dashed curves are invisible to the light source and cannot be seen by the camera.

which is lying outside of both conics. The other two eigenvectors are discarded since they lie within both of the silhouettes. The orthogonal constraints between the vanishing point v and the vanishing line l provides the pole-polar relationship with respect to the IAC $\omega$ [3]

$$\begin{aligned} \mathbf {l = \varvec{\omega } v}. \end{aligned}$$

(6)

Here the point v is the pole and the line l is the polar. Thus, two linear independent constraints on the IAC are provided by an image of two spheres. To fully calibrate the camera we still need to find other constraints from the system.

4 Circular Points from the Intersection Curves

Let the plane of the line-structured light in the world coordinate frame be

$$\begin{aligned} \mathbf {l}_{light}^{T}\mathbf {p} = 0, \end{aligned}$$

(7)

where $\mathbf l _{light}= [a b c d ]^{T}$ is the coefficient vectors of the light plane, $\mathbf p = [x y z \,1]$ is any point on this plane. The projection of the laser line onto the two spheres is the front half of the intersection between the light plane and the two spheres, which can be fitted to two complete conics $\mathbf x ^{T}{} \mathbf D _1\mathbf x = 0$ and $\mathbf x ^{T}$ $\mathbf D _2\mathbf x = 0$ on the image, where $\mathbf D _1$, $\mathbf D _2$ are two $3 \times 3$ matrices. Since $\mathbf D _1$ and $\mathbf D _2$ are the projection of two co-planar circles, and intersect the vanishing line $\mathbf l _{\infty }$ of the light plane at two invariant conjugate complex points, which are the circular points i and j [8] (see Fig. 3 for details). In generally, when we know the conic $\mathbf D _1$ and $\mathbf D _2$, we can get these two intersection points by solving quadratic equations. However, the process of solving two quadratic equations is time consuming.

Fig. 2.

The vanishing line l of the plane, which is pass through the camera center and the center of two spheres. The vanishing point v of the plane normal and vanishing line l, satisfied the pole-polar relationship and provide two linear independent constraints on the IAC.

Fig. 3.

Co-planer conics $\mathbf D _{1}$ and $\mathbf D _{2}$ intersect the vanishing line $\mathbf l _\infty$ at two imaged circular point i and j. The pair of circular points provide the other two linear independent constraints on the IAC.

In order to get the circular points i, j in a linear way, the generalized eigenpairs ($\lambda _i,\mathbf v _i$) (i = 1, 2, 3) of conic $\mathbf D _1$ and $\mathbf D _2$ are firstly obtained, where $\lambda _i$ are the eigenvalues, $\mathbf v _i$ are the eigenvectors. Similar to Sect. 3, being regarded as points, one of the eigenvectors (let it be $\mathbf v _3$) lies outside of both conics, on the vanishing line $\mathbf l _{\infty }$. Its corresponding eigenvalue $\lambda _3$ has the smallest absolute value. The other two eigenvectors ($\mathbf v _1$ and $\mathbf v _2$) lie inside of either one of the conic $\mathbf D _1$ and $\mathbf D _2$. Secondly, each of the degenerated conic $\mathbf L _i = \mathbf D _1 - \lambda _{i}$ $\mathbf D _2$ $({i} = 1, 2, 3)$ composes two lines which can be obtained by singular value decomposition (SVD) [17]

$$\begin{aligned} \mathbf {L} _{i} = \mathbf {U} _{i}\mathbf {S}_{i}\mathbf {U} ^{T}_{i}= \left( \begin{array}{ccc} \mathbf {u} _{i1}&\mathbf {u} _{i2}&\mathbf {u} _{i3} \end{array} \right) \left( \begin{array}{ccc} \mathbf {s} _{i1}&{}0&{}0\\ 0&{}\mathbf {s} _{i2}&{}0\\ 0&{}0&{}0 \end{array} \right) \left( \begin{array}{c} \mathbf {u} ^{T}_{i1}\\ \mathbf {u} ^{T}_{i2}\\ \mathbf {u} ^{T}_{i3} \end{array} \right) , \end{aligned}$$

(8)

where the diagonal matrix $\mathbf S _i$ is with the singular values $s_{i1} > s_{i2} \geqslant 0$, and the $\mathbf U _i \in \mathbf R ^{3\times 3}$ is a orthogonal matrix. From each $\mathbf S _i$ and $\mathbf U _i$, two vectors [$\mathbf x _{i1}$, $\mathbf x _{i2}$] $\equiv \mathbf U _i$ $\mathbf S ^{1/2}_i$ [$\mathbf e _1$ $\mathbf e _2$] can be derived, where e$_1$=[1 0 0]$^T$ and e$_2$=[0 1 0]$s^T$. For the degenerated conics $\mathbf L _3$, we can get two lines from $\mathbf x _1$, $\mathbf x _2$ passing through $\mathbf v _3$, i.e., $\{ \mathbf m _3$, $\mathbf n _3\} = \{ \mathbf x _1 + \mathbf x _2$, $\mathbf x _1 - \mathbf x _2\}$ [17]. The vanishing line $\mathbf l _{\infty }$ can be selected from $\{ \mathbf m _3$, $\mathbf n _3\}$ using the fact that the two points $\mathbf v _1$ and $\mathbf v _2$ lie on both side of $\mathbf l _\infty$. Since the conic $\mathbf C ^{*}_\infty$ ($\mathbf C ^{*}_\infty = \mathbf{ij }^T + \mathbf{ji }^T$) [6] dual to the imaged circular points i and j is a degenerate line conic (rank 2) consisting of the two circular points on the vanishing line $\mathbf l _\infty$ , $\mathbf l _\infty$ lies in the null space of $\mathbf C ^{*}_\infty$, i.e.,

$$\begin{aligned} \mathbf {C ^{*}_\infty }\mathbf { l _\infty } = 0. \end{aligned}$$

(9)

This provides two constraints on $\mathbf C ^{*}_\infty$. Besides, each of the other two degenerated conics $\mathbf L _1$ and $\mathbf L _2$ composes two complex conjugate lines passing through the circular points, which provides further constraints on $\mathbf C ^{*}_\infty$ [17],

$$\begin{aligned} \mathbf {x^{T}_{i1}C ^{*}_{\infty }x_{i2} = 0} \, \, \, and \, \, \, \mathbf {x^{T}_{i1}C ^{*}_{\infty }x_{i1} - x^{T}_{i2}C ^{*}_{\infty }x_{i2}}= 0\, (where\,\, i = 1, 2, 3). \end{aligned}$$

(10)

Finally, from (9) and (10), the conic $\mathbf C ^{*}_\infty$ dual to the imaged circular points can be recovered and it can then be broken down to the circular points i, j by SVD.

The circular points i, j are lying on $\mathbf l _\infty$ in pairs (see Fig. 3 for details), and are also lying on the IAC [3], i.e.,

$$\begin{aligned} \mathbf {i}^{T}\varvec{\omega } \mathbf { i} =\mathbf { j} ^{T}\varvec{\omega }\mathbf { j} = 0. \end{aligned}$$

(11)

This provides another two linear independent constraints on IAC additional to (6). The system resulting from (6) and (11) provides four linear constraints on $\varvec{\omega }$, hence a natural camera with only four parameters, i.e., a zero skew camera, can be recovered.

5 The Recovery of Sphere Center 3D Position

The position of the sphere center can be estimated from the camera intrinsic matrix. In conventional methods, triangularization is generally used to reconstruct the position from two images captured at two different viewpoints. This method is accurate, but exists ambiguity when decomposing the essential matrix E, and needs at least two images and known some previous information.

To obtain the 3D position of the sphere center just from one image, we can use the eigenpairs from the right-circular cone of spheres’ conic under camera coordinate system. From Sects. 3 and 4, we have already know the intrinsic matrix K and detected two spheres’ outlines $\mathbf C _{i}$ (i = 1, 2) from image. Then, we can recovery the right-circle cone $\mathbf Q _{i}$ for each sphere circle from back perspective projection model.

$$\begin{aligned} \mathbf {Q}_{i} = \mathbf {K}^{T}\mathbf {C}_{i}\mathbf {K}. \end{aligned}$$

(12)

We apply matrix decomposition on the $\mathbf Q _{i}$, and get eigenvectors ($\mathbf e _{i1}$, $\mathbf e _{i2}$, $\mathbf e _{i3}$) together with the corresponding eigenvalues ($\lambda _{i1}$, $\lambda _{i2}$, $\lambda _{i3}$), $\lambda _{i1}$, $\lambda _{i2}$ > 0, $\lambda _{i3}$ < 0, where i = 1, 2. Once we get the measured sphere radius R, we can get the sphere center position under the camera coordinate system from [19]

$$\begin{aligned} \left[ X_{i} \, Y_{i} \, Z_{i}\right] ^{T} = \mathbf {\lambda }_{i}^{\prime }\left[ \mathbf {e}_{i3} \right] , \end{aligned}$$

(13)

where

$$\begin{aligned} \mathbf {\lambda }_{i}^{\prime } = R\sqrt{\frac{\left| \lambda _{i1} \right| (\left| \lambda _{i2} \right| +\left| \lambda _{i3}\right| )}{\left| \lambda _{i3}\right| (\left| \lambda _{i1}\right| +\left| \lambda _{i3}\right| )}}, (i = 1, 2). \end{aligned}$$

(14)

6 Experiments and Results

6.1 Synthetic Data

The synthetic camera has fixed intrinsic parameters, with focal length f = 1000, $\alpha$ = 1.2, s = 0, and principle point ($u _0$, $v _0$) = (320,240). The points on the silhouette of each sphere and each laser ellipse are corrupted with Gaussian noises at different levels from 0 to 5 pixels with 0.1 pixel step, and then two conics are fitted on the noisy points [11].

The synthetic experiment is to calibrate the camera intrinsic parameters using our proposed method under different noise levels. For each level, 100 independent tests are carried out. And, then computing the RMS error in each noise step. Figure 4 shows the relative error between the ground truth and RMS (root mean square) errors of the focal length f, the aspect ratio $\alpha$, and the principle point ($u_0$, $v_0$). From Fig. 4, it can be seen the noise is slightly effect the four parameters in intrinsic matrix. Even for the worst error of the principle point $v_0$ is less than 15%.

Furthermore, we compare the proposed method with the orthogonal method mentioned in [16]. Under the same condition, the average intrinsic matrix parameters is showed in Table 1. It can be seen that the result of proposed method is better than the orthogonal method for a natural camera. Besides, the orthogonal method requires two images, which contain only the silhouettes of the two spheres to calibrate the camera intrinsic parameters.

Fig. 4.

The relative error between the RMS errors and ground truth at different noise levels below 5 pixels with 0.1 pixel step. And for each step, 100 independent tests are carried out. (a) The relative errors for the focal length f. (b) The relative errors for the aspect ratio $\alpha$. (c) The relative errors for $u_0$. (d) The relative errors for $v_0$.

To verify the correctness of the estimated camera intrinsic parameters, we can obtain the positions of the two sphere centers and compare the estimated distance with the ground truth value. In the synthetic experiment, the sphere radius and the distance of sphere centers are given. Then, we also can use the same way to assess the stability of intrinsic matrix under different noise levels. Figure 5 shows the RMS error of the distance of two sphere centers under the noise level below 5 pixels. We can find the distance is weakly effected by the noise. The biggest RMS error is still below 1.00%, which means the evaluating the accuracy of camera intrinsic parameters by sphere center distance is reliable.

Table 1. Estimated camera parameters by the proposed method and the orthogonal method under the noise lever below 5 pixels.

Fig. 5.

The relative errors between RMS error and ground truth for the sphere center distance at each noise level. The noise level is below 5 pixels with 0.1 pixel step. And for each step, 100 independent tests are carried out.

6.2 Real Scene

In the real scene experiment, an image of two connected ceramic balls is taken by an industrial camera. The setup of the real system is shown in Fig. 6(a). The two spheres are hit by the laser light and the detected light are fit to two conics (see Fig. 6(b) for details). The distance between these two sphere centers is 74 mm. The radius of both spheres is 12.7 mm. The image resolution is $1280 \times 960$. The Canny edge detector [12] is firstly applied to find the points on the silhouette of the spheres to which conics are fitted with a least square approach [11]. The intersection conics between the light plane and the spheres are obtained by the following steps. First, the Hessian matrix was used to compute the center curve of the laser light rays in image, and then the center curve was fitted to an ellipse with a least square approach [11]. By using proposed methods, the estimated intrinsic parameters are then obtained and listed in Table 2. For the purpose of comparison, the result from the classical method of Zhang [13] is taken as the ground truth. From Table 2, we can see that the result of proposed method is much close to the ground truth, except the vertical coordinate of the principle points. This may due to the errors induced from the vanishing point in (6). Compared with Zhang’s method [13], this proposed method can just use one single image to recovery the camera intrinsic parameters linearly and get the result almost accurate, which means the proposed method is correct and reliable. Besides, we cannot use orthogonal method [16] to obtain the camera intrinsic matrix from one image which include only two spheres. Two images can provide four linear independent constraints on the camera intrinsic parameters. Table 2 also shows the results of the proposed method and the orthogonal method based on two same images. From the result, we can see the focal length is better than that of the proposed method. The principle point is worse own to the silhouettes extraction. The result of the orthogonal method is worse and own to the vanishing point in (6) derived far away.

Fig. 6.

The real scene system setup and captured image. (a) is the real scene system setup. (b) is the captured image with extracted silhouette circles, in which the spheres and laser light can all be seen.

Table 2. Comparison of intrinsic results in three methods

After we get the intrinsic matrix and the sphere conics from the image, we can use the method mentioned in Sect. 5 to estimate the distance between the sphere centers. Please note, at this moment, the sphere center is under the camera coordinate system. And based on the projection property, the distance between two sphere centers will not change. Hence, we can compare the estimated distance with the ground truth to assessment accuracy of the proposed method. The results of the distance of the sphere centers are list in Table 3. The camera intrinsic matrix calibrated by the proposed method is accepted. The difference from the ground truth is caused by the silhouette and extracted laser center line.

Table 3. Comparison of the distance of sphere centers between the estimated value and ground truth

7 Conclusion

This paper has proposed an approach to calibrate the camera intrinsic parameters using a single image which includes the outline of two spheres and two ellipses generated by the intersection between the line-structured laser light plane and the two spheres. Then, the distance of two sphere centers can be recovered using the estimated intrinsic matrix K. For the intrinsic matrix calibration, a plane is formed by the camera center and two sphere centers. The orthogonal constraint on the silhouettes of two spheres can be generated from the vanishing line of the plane and its normal direction. Besides, the pair of the circular points of the light plane is calculated from the generalized eigenpairs from the intersection between the light plane and the spheres. Four linear constraints derived from the orthogonal constraint and the pair of circular points. Thus, the camera intrinsic parameters K can be recovered. Furthermore, the 3D positions of these two sphere centers under the camera coordinate can be recovered from the camera intrinsic matrix. After that, on the strength of the distance invariability under the camera coordinate system, we can further obtain the distance of two sphere centers. And then using this distance to assess the accuracy of the camera intrinsic matrix. Both synthetic and real experiments show the feasibility of the proposed approach.