Keywords

1 Introduction

A typical coded structured light system consists of a projector and a camera, of which the encoded fringes are distorted on the curved surface. Focused on the demand of encoding, decoding, multi-view registration and system calibration, we expect to obtain the undistorted fringes from the image captured by the camera.

Raskar is the first to study on the projecting distortion calibration on quadric surfaces like spherical surface, cylindrical surface and paraboloid. When studying on multi-projector imaging, he used interpolation to make up for the differences among each projecting images. Besides, within in a small scale, Raskar’s research is also representative [1]. Yuichiro tried to calibrate the projecting distortion on irregular surface [2], which has a positive result as a whole but not good in detail, and then Michael proposed a distortion calibration approach which implements block mapping to the 2D projecting images. This approach does not need to calibrate the camera but it has a disadvantage that the calibration precision relies on the intensity of the image blocks. With the improvement of the calibration and iteration algorithm, the accuracy and robustness of calibration approaches are improved [3, 4]. Zhang divided the projected surface in equal angle under the polar coordinates and achieved the distortion calibration of the image projected on the curved surface with the texture mapping [5]. This approach is fast but the precision need to be improved. Liu used the corresponding image coordinates under the two cameras of a certain point on the projected surface to get the conversion matrix [6]. The precision is improved but the complexity of the system is increased. Yang uses cubic Beizer to solve the conversion matrix [7], which is suitable for complex curved surfaces. However, it reduces the calculation accuracy and speed.

Recently, the projecting distortion calibration approaches mainly focus on normal curved surfaces and emphasizes the visual effect. In this paper, the research focuses on the structured light 3D measurement and simultaneously considers the encoded fringe distortion calibration approach, which is able to improve the accuracy of the system calibration and measurement.

2 Coded Structured Light and Projecting Distortion

In this paper, we use Infocus 82 projector and HV-F22F camera to construct the coded structure light sytem, which is shown in Fig. 1. The projector projects the patterns orthographically and the camera captures images from the side. According to the projecting range of the projector, the measuring range in z direction is 120 mm and the measuring ranges in x and y directions are respectively 350 mm and 250 mm.

Fig. 1.
figure 1

Coded structured light measuring device

Full size image

In order to fit the measurement of the surfaces with strong reflection and various colors, we use the binary encoding/decoding principle, which is shown in Fig. 2. Combing the line-shifting fringe center and the Gray code fringe edge to encode and decode is an effective way to improve the sampling density when guaranteeing the decoding accuracy and resolution.

Fig. 2.
figure 2

Encoding and decoding principle by combining Gray code and line shift fringe (Color figure online)

Full size image

In the process of encoding/decoding, multi-view registration and system calibration, the Gray code or line-shifting fringe will distort when projected to the measured surface, which is shown in Fig. 3. If we want to obtain the undistorted image, we need to project pre-calibrated patterns. The following will analyze the projecting distortion calibration approach that is suitable for coded structured light system.

Fig. 3.
figure 3

Encoded pattern distortion

Full size image

3 Practicability Analysis of Projecting Distortion Calibration Approach in Coded Structured Light

Typical approaches mainly include control point calibration approach and fitting surface calibration approach.

3.1 Control Point Calibration Approach

The control point calibration approach sets grid to the projector image, determines the corresponding grid intersection of the projector and camera images and fit the conversion matrix of them. Finally, according to the undistorted camera image, calculate the pre-calibrated projector image through the conversion matrix. When projecting the pre-calibrated image, the camera is able to capture the undistorted image [8, 9].

The conversion matrix can be presented with a bivariate polynomial, which is shown in (1). To quickly get the conversion matrix, we can let n = 1, namely, linear conversion, which is shown in (2).

$$ \begin{aligned} x^{\prime} = \sum\limits_{i = 0}^{n} {\sum\limits_{j = 0}^{n - i} {a_{ij} x^{i} y^{j} } } \, \hfill \\ y^{\prime} = \sum\limits_{i = 0}^{n} {\sum\limits_{j = 0}^{n - i} {b_{ij} x^{i} y^{j} } } \hfill \\ \end{aligned} $$
(1)
$$ x^{\prime} = a_{00} + a_{10} x + a_{01} y\,y^{\prime} = b_{00} + b_{10} x + b_{01} y $$
(2)

In the equations, a and b represent the conversion coefficients of the projector image and the camera image respectively. x, y, x′ and y′ are the coordinates of the projector and camera image before and after the conversion.

To accurately obtain the conversion matrix, we can take advantage of higer-degree polynomial such as Bezier surface, which is shown in (3).

$$ P(u,v) = \sum\limits_{i = 0}^{N} {\sum\limits_{j = 0}^{M} {P_{ij} B_{i}^{N} (u)B_{j}^{M} (v);} } \,u,v \in [0,1] $$
(3)

In (3), (u, v) is the image coordinates, P is the conversion coefficient of the image and B is the Bezier function. When solving, we determine the Bezier surface with (N+1) × (M+1) control points. Each point on the surface has a corresponding u and v value. When the projected surface is a complex surface, to describe the surface more accurately, we need to increase the degree of the Bezier surface, which consequently increases the calculation work.

3.2 Fitting Surface Calibration Approach

In the fitting surface calibration approach, The converting relation T pc of the projector and camera coordinate systems to the projected surface Q can be presented with function Ψ, which is a 4 × 4 symmetric matrix. Q can be obtained according to cloud point X measured with the structured light, namely, X T QX = 0. Then, for a point x on Q, the relation between its projector coordinate x p and its camera coordinate x c can be denoted by (4).

$$ x_{p} \cong (B - eq^{T} )x_{c} \pm \sqrt {x_{c}^{T} (qq^{T} - Q_{33} )x_{c} } \cdot e $$
(4)

In (4), B and e are the rotation matrix and transformation matrix of T pc , which are obtained through calibration. Q 33 and q can be obtained according to \( Q = \left[ {\begin{array}{*{20}c} {Q_{33} } & q \\ {q^{T} } & g \\ \end{array} } \right] \). The image coordinate system is a special case of the world coordinate system. Therefore, function Ψ can also be treated as the relation between the projector image and observation image, and consequently, we can calculate the projector image.

The control point calibration approach has a simple process and the control point calibration is accurate. However, those which are not control points are obtained through fitting, so they are not accurate enough. Despite the fact that we can use higher-degree Beizer surface to improve the fitting accuracy, for a complex surface, it is still hard to achieve accurate calibration with one Beizer expression. We need multiple Beizer surfaces to achieve that, which lower the calibration speed.

The fitting surface calibration approach comprehensively considers the projected surface. Points in and out of the point cloud are all accurately calibrated. But for complex non-quadric surfaces, the fitting error is large, especially in complex part and edges.

Combining the features of the two approaches, we propose a calibration scheme based on simplified encoded structured light.

4 Calibration Scheme Based on Simplified Coded Structured Light Model

According to the coded structured light model which is shown in Fig. 4, the world coordinate (X w, Y w, Z w) and the camera image coordinate (m c, n c) of the spatial point P has a converting relation which is shown in (5).

Fig. 4.
figure 4

Spatial converting relation

Full size image
$$ Z^{\text{c}} \left[ {\begin{array}{*{20}c} {n^{\text{c}} } \\ {m^{\text{c}} } \\ 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {({{N^{\text{c}} } \mathord{\left/ {\vphantom {{N^{\text{c}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{1}^{\text{c}} } & {(\tan \alpha \cdot f_{1}^{\text{c}} )/d_{1}^{\text{c}} } & {n_{0}^{\text{c}} } \\ 0 & {({{M^{\text{c}} } \mathord{\left/ {\vphantom {{M^{\text{c}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{2}^{\text{c}} } & {m_{0}^{\text{c}} } \\ 0 & 0 & 1 \\ \end{array} } \right] \cdot \, \left[ {R^{\text{c}} ,T^{\text{c}} } \right] \cdot \, \left[ {\begin{array}{*{20}c} {X^{\text{w}} } \\ {Y^{\text{w}} } \\ {Z^{\text{w}} } \\ 1 \\ \end{array} } \right] $$
(5)

In the equation, tan \( a \cdot f_{1}^{\text{c}} \) indicates the skew angle α caused by the incomplete orthogonality of the principle optic axis and CCD image surface. Furthermore, n c and m c are the corresponding column order and row order of the spatial point P in the camera image. \( d_{1}^{\text{c}} \) is the width of a single pixel of the CCD. N c and M c are the column and row numbers of the CCD. Besides, \( \beta_{1}^{\text{c}} \) and \( \beta_{2}^{\text{c}} \) are half of the horizontal and vertical field angles. (\( n_{0}^{\text{c}} ,\,\,m_{0}^{\text{c}} \)) is the pixel position of the principle point on CCD image surface. R p and T p are the rotation matrix and translation matrix, respectively. If we ignore the two position error between the principle optic axis and the CCD image surface, (5) can be simplified as (6).

$$ Z^{\text{c}} \left[ {\begin{array}{*{20}c} {n^{\text{c}} } \\ {m^{\text{c}} } \\ 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {({{N^{\text{c}} } \mathord{\left/ {\vphantom {{N^{\text{c}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{1}^{\text{c}} } & 0 & 0 \\ 0 & {({{M^{\text{c}} } \mathord{\left/ {\vphantom {{M^{\text{c}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{2}^{\text{c}} } & 0 \\ 0 & 0 & 1 \\ \end{array} } \right] \, \cdot \left[ {R^{\text{c}} ,T^{\text{c}} } \right] \cdot \left[ {\begin{array}{*{20}c} {X^{\text{w}} } \\ {Y^{\text{w}} } \\ {Z^{\text{w}} } \\ 1 \\ \end{array} } \right] $$
(6)

Likewise, the world coordinate (X w, Y w, Z w) and the projector image coordinate (m p, n p) has a converting relation which is shown in (7).

$$ Z^{\text{p}} \left[ {\begin{array}{*{20}c} {n^{\text{p}} } \\ {m^{\text{p}} } \\ 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {({{N^{\text{p}} } \mathord{\left/ {\vphantom {{N^{\text{p}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{1}^{\text{p}} } & 0 & {n_{0}^{\text{p}} } \\ 0 & {({{M^{\text{p}} } \mathord{\left/ {\vphantom {{M^{\text{p}} } 2}} \right. \kern-0pt} 2}) \cdot {\text{c}}\tan \beta_{2}^{\text{p}} } & {m_{0}^{\text{p}} } \\ 0 & 0 & 1 \\ \end{array} } \right] \, \cdot \left[ {R^{\text{p}} ,T^{\text{p}} } \right] \cdot \, \left[ {\begin{array}{*{20}c} {X^{\text{w}} } \\ {Y^{\text{w}} } \\ {Z^{\text{w}} } \\ 1 \\ \end{array} } \right] $$
(7)

In (7), \( m_{0}^{p} \) of the principle point coordinate (\( n_{0}^{p} ,\;m_{0}^{p} \)) is not zero and even goes beyond the DMD chip. Through connecting (6), (7) and the encoding/decoding principle (8), we can establish the correspondence among the projector image point, the camera image point and the projected surface point and then build the calibration conversion matrix.

$$ n^{\text{p}} = {\text{g}}[\Upphi {\text{v}}(m^{\text{c}} ,n^{\text{c}} ),N^{\text{p}} ] $$
(8)

In (8), ΦV(n c, m c) is the encoding value of the camera image point.

Compared with the control point calibration approach, the fitting of the proposed approach is more accurate because the number of the point cloud obtained by coded structured light measurement is much more than the number of the control point. Moreover, we simplified the minor parameters of the system and therefore the calculation work is reduced. Then, compared with the fitting surface calibration approach, the proposed approach has no fitting error in complex part and edges of the surface and consequently increase the accuracy of the calibration.

5 Projecting Distortion Calibration Experiments on Encoded Fringes

5.1 Visual Effect Evaluation

To verify the effectiveness of the proposed approach, we use 3dsmax to build the coded structured light system and implement the projecting distortion calibration experiment.

Figure 5(a) shows a standard projecting image. The distortion situation of projecting the image to the sphere is shown in Fig. 5(b). What’s more, the pre-calibrated image obtained with the proposed approach is shown in Fig. 5(c) and the image captured by the camera is shown in Fig. 5(d). From the figure, we can see that the chessboard is effectively calibrated and that is equivalent to the fact that there is no visual effect distortion in Fig. 5(a).

Fig. 5.
figure 5

Visual effect on sphere

Full size image

Furthermore, the same calibration effect is also obtain on complex surface. Figure 6(a) and (b) are the camera images before and after the calibration.

Fig. 6.
figure 6

Visual effect on complex surface

Full size image

5.2 Quantitative Evaluation

In order to verify the accuracy of the proposed approach, we implement projecting distortion calibration experiment on the sphere with both the proposed approach and the fitting surface calibration approach.

Theoretically, the edges of the encoded fringes in the camera image are parallel and equal-width straight lines. Therefore, we evaluate the calibration accuracy by judging the straightness and equal-width degree of the fringe edges after the distortion calibration. First of all, we use the gray curve intersection method to locate the edges [10]. Then, we fit the discrete edge points into edge lines and use the discrete degree to evaluate the straightness of the edge. What’s more, we calculate the average interval according to each interval of the adjacent edges and use the maximum difference between the average interval and each edge interval to evaluate the equal-width degree.

The encoded fringe of which the width is 64 pixels is shown in Fig. 7. The captured camera image after the calibration is shown in Fig. 7(a) and the fitted edge is shown in Fig. 7(b). Taking the largest point straightness error of each edge as the straightness error of that edge, which is shown in Fig. 7(c). Also, take the largest difference between the average interval and each edge interval as the equal-width error, which is shown in Fig. 7(d).

Fig. 7.
figure 7

Calibration result of 64 pixel-width fringe

Full size image

The encoded fringe of which the width is 8 pixels is shown in Fig. 8.

Fig. 8.
figure 8

Calibration result of 8 pixel-width fringe

Full size image

The distortion calibration data of the 64 pixels and 8 pixels fringes are shown in Table 1. From the table, we can see that the fitting surface approach has a larger error than that of the proposed approach.

Table 1. Error of projecting distortion calibration
Full size table

6 Conclusion

Focused on the demand of the structured light system distortion calibration, we propose a calibration scheme based on simplified coded structured light system and according to the control point approach and the fitting surface approach. Moreover, we analyze the accuracy and sampling density advantage of the proposed approach to the complex surface. Besides, we verify the effectiveness of the proposed approach through visual effect. We also design the encoded fringe distortion calibration evaluating scheme based on the straightness and equal-width degree. In the experiments of 64-pixel width fringe and 8-pixel width fringe, the straightness error and equal-width error of the proposed approach are all smaller than those of the fitting surface approach, which proves that the proposed approach has a smaller comprehensive error. In the future, we will focus on the fringe distortion calibration of highly complex surface and improve the measuring accuracy of the system.