Keywords

1 Introduction

Laparoscopic surgery is a popular clinical procedure because of its minimal invasiveness and shorter postoperative hospital stays relative to open surgery. However, this procedure is always limited by restricted field of view and lack of tactile feedback. To overcome these limitations, image-guided surgery system [1] and surgical robotic [2] have been developed. Hand-eye calibration is a well-known term in robotics literature; for laparoscopic surgery navigation, “hand” denotes the reflective passive markers (RPM) attached to the laparoscope, and “eye” denotes the laparoscopic camera. Hand-eye calibration determines the position and orientation of the laparoscopic camera relative to the (RPM), is an important procedure for implementing a navigation system for laparoscopic surgery.

The hand-eye calibration can be formulated as a linear system of equations as:

$$ {\mathbf{AX}} = {\mathbf{XB}} $$
(1)

where A denotes the relative motion of laparoscopic camera, B is the motion of the RPM, and X represents the hand-eye transformation. Tsai and Lenz [3] have proven that at least two motions are required to solve Eq. (1).

Many methods have been proposed to estimate the hand-eye transformation. It can be decomposed into a rotation matrix and a translation vector. The most commonly used method was realized by computing the rotation and the translation separately [4], while the rotation and the translation can also be jointly estimated [5]. Horaud et al. [6] proved that the camera intrinsic calibration is not independent from hand-eye calibration. Malm and Heyden [7] investigated a joint solution for a camera intrinsic calibration and hand-eye calibration using an unfixed planar calibration object for robot vision. Malti et al. [8] proposed a similar hand-eye calibration approach, they estimated the optimal camera parameters as well as hand-eye transformation based on the minimal number of the calibration object images.

Although hand-eye calibration has been well-studied within robotic field, it is challenging to transfer these methods from robotic literature into laparoscopic surgery directly. Because different situations existed in laparoscopic surgery, the proposed method is required to be compatible with sterility constrains and not to interrupt the surgical workflow. Recently, Thompson et al. [9] proposed a novel calibration method for laparoscopic surgery navigation. In their work, an invariant cross-hair point was imaged by a stereo laparoscope with different views; positions of the invariant point were automatically calculated from laparoscopic images, and the Levenberg-Marquardt (LM) optimizer was used to estimate an optimal hand-eye transformation based on extracted invariant points and poses of RPM. However, the accuracy of their method would be compromised with inaccurate cross-hair detection or outlier data input.

Inspired by the work of Thompson et al. [9], we proposed a progressive strategy based on an invariant point for hand-eye calibration. In this strategy, the hand-eye transform of the i-th iteration was initialized with the results of i-1 iteration, and updated based on the accumulation data buffer of the i-th iteration. Different from the invariant point used in Thompson’s work, a fixed small dot acting as the invariant point was applied in our method. To evaluate accuracy of the proposed method, two errors were calculated at each iteration. With the iterative evaluation, the operator can set a desired error prior to calibration and terminates the hand-eye calibration when the procedure gives an acceptable value.

2 Methods

2.1 Hardware

The hardware used in this study is shown in Fig. 1. A Polaris Spectra optical tracking system (NDI, Waterloo, Ontario, Canada) was employed to record the pose of RPM. Because the clinical stereo laparoscope is not available, two Ultra Mini CMOS Color Cameras (MISUMI) were assembled to make a stereo laparoscope (Fig. 1A), its spatial resolution is 640 × 480 and baseline is 6 cm. The RPM is fixed on the distal end of the stereo laparoscope. To improve the accuracy of the invariant point detection, we did not use the cross-hair as an invariant point, any sterile paper marked with one dot can be used as a calibration object in our work. To guarantee the accuracy of invariant point extraction, the diameter of the dot was set as 1 mm.

Fig. 1.
figure 1

(A) Custom-made stereo laparoscope and (B) hand-eye calibration theatre setting.

Full size image

2.2 Progressive Hand-Eye Calibration

The stereo laparoscopic camera was calibrated prior to hand-eye calibration. To obtain the camera intrinsic and extrinsic parameters, the widely used checkboard method [10] from OpenCV was applied directly. Because the focal length is fixed in custom-made laparoscope, camera calibration can be performed prior to the clinical procedure and saved to a file for the usage at any time. During the procedure of hand-eye calibration, the laparoscope camera was kept above the dot calibration object, and the RPM was tracked by the optical positional tracker at the same time. We rotated the laparoscope within a cone to acquire laparoscopic images of the invariant point (Fig. 2); in this cone, the vertex is roughly located at the dot, and the rotation angle is about 60°.

Fig. 2.
figure 2

Rotate the laparoscope within a cone, its vertex is roughly located at the dot, and its angle is about 60°.

Full size image

We proposed a strategy that laparoscopic images of the calibration object were iteratively acquired and processed to get an optimal hand-eye transformation. Figure 3 shows the workflow of the proposed method. After capturing the left/right images of the variant dot, both images were converted into grayscale format and smoothed, the blob detection algorithm was applied to detect 2D image coordinates of the dot centroid. Based on previously determined camera intrinsic and extrinsic parameters, the 3D coordinate of the dot centroid in the left camera coordinate system was computed by triangulation. The 3D coordinate is defined as X L_i , where i denotes the number of iterations. Meanwhile, the optical position tracking system captured the pose of RPM, \( ^{\text{tracker}} T_{{{\text{hand}}\_i}} \) , at the same time.

Fig. 3.
figure 3

The workflow of the proposed method for hand-eye calibration.

Full size image

Considering that the ith iteration, the coordinate of the invariant point defined in the left camera space, X L_i , can be transformed to the optical tracker coordinate system (see the relationship of transformations as shown in Fig. 2), X world_i , according to the following equation,

$$ X_{{{\text{world}}\_i}} = \,^{\text{tracker}} T_{{{\text{hand}}\_i}} \times \,^{\text{hand}} T_{\text{eye}} \times X_{{{\text{L}}\_i}} $$
(2)

Meanwhile, we denote the 3D coordinate of invariant point defined in optical tracker system space as X tracker_point = (x t , y t , z t ). Therefore, there are two vectors, X world_i and X tracker_point, representing coordinates for the same invariant point. Theoretically, these two vectors should be identical or the residual error E between them is as small as possible,

$$ E_{j} = X_{{{\text{world}}\_i,j}} \,{-}\,X_{{{\text{tracker}}\_{\text{point}}\_j}} $$
(3)

where j represents the x, y, z components of X tracker_point.

The hand-eye transformation can be represented as a matrix, \( \left[ {\begin{array}{*{20}c} R & t \\ {\vec{\varvec{0}}} & 1 \\ \end{array} } \right] \), where R denotes a rotation with three components, and t is a translation which is also can be divided into three scalars. Thus, there is a total of nine parameters, including R, t, and three components of X tracker_point to be optimized. Levenberg-Marquardt optimizer algorithm was employed to optimize the unknown parameters based on Eq. (3), which is initialized with X tracker_point = (0, 0, 0) and an identity matrix of \( ^{\text{hand}} T_{\text{eye}} \).

A strategy that progressively adding new frames of the invariant dot into the workflow to optimize the obtained hand-eye transform was used. We acquired one frame data of the variant dot from the laparoscopic space and the tracking system space, respectively. After adding the newly acquired data into the data buffer obtained before, a new accumulated data buffer was set as the input for LM algorithm to re-optimize the hand-eye transform.

The calibration error was evaluated at each iteration. Before the calibration, we located the invariant dot in the optical tracking system with a tracking probe from the Polaris Spectra system, then the 3D coordinate of the invariant dot, X tracker_point, was obtained. Based on the calculated hand-eye transform, the 3D coordinate of the invariant dot in the optical tracking system can be mapped into the 2D image coordinate system according to the following equation,

$$ X_{{{\text{L}}\_{\text{project}}\_i}} = \,\mathop {^{\text{hand}} T_{\text{eye}} }\nolimits^{ - 1} \times \,\mathop {^{\text{tracker}} T_{{{\text{hand}}\_i}} }\nolimits^{ - 1} \times X_{{{\text{tracker}}\_{\text{point}}}} $$
(4)

where X L_project_i denotes the mapped 2D image coordinate; the calibration error is evaluated as the distance (in pixel) between the mapped invariant dot and the corresponding ground truth in the laparoscopic image. We evaluated the calibration error iteratively. The iteration terminates once the error meets the input tolerance, otherwise; hand-eye calibration program will progressively collect the new frames to perform the next iteration until the error is acceptable.

3 Experiments

The ground truth data were acquired before calibration to evaluate accuracy of the proposed method. Through the blob detection, the 2D image coordinate of the variant dot, X L_base, was computed from laparoscopic images. At the same time, a probe from Polaris Spectra system was used to locate the position of invariant dot in the tracker coordinate system, X tracker_base. The X L_base and X tracker_base were used as the ground truth data for the method evaluation. The proposed hand-eye calibration approach was implemented based on an open-source toolkit of MITK [11] as shown in Fig. 4.

Fig. 4.
figure 4

The graphic user interface of the hand-eye calibration implemented using MITK.

Full size image

The forward and backward errors were calculated to assess the calibration accuracy. Based on the obtained hand-eye transform after each iteration, a 2D image coordinate of the variant dot X L_test, can be mapped into the optical tracker coordinate system to get a 3D coordinate X tracker_test, by applying Eq. (2). The forward error is defined as the distance (in mm) between X tracker_test and the ground truth X tracker_base, on the contrary, by applying Eq. (4), the point coordinate estimated from LM optimization, denoted as X tracker_estimated, can be mapped into 2D laparoscopic image coordinate system. The backward error is defined as the distance (in pixel) between mapped 2D coordinate, donated as X L_estimated, and the corresponding ground truth X L_base.

4 Results

A total of 70 frames data were collected to evaluate calibration accuracy. The forward and backward errors are depicted in Figs. 5 and 6, respectively.

Fig. 5.
figure 5

Forward errors, including the total error and its three components were calculated for 60 iterations.

Full size image
Fig. 6.
figure 6

Backward errors, including the total error and its two components were calculated for 60 iterations.

Full size image

The accuracy of the proposed method can be guaranteed when the outlier data was collected, because the procedure can be reinitialized with a reasonable value from a previous iteration without outlier data, and the error would fall into an acceptable range during the next few iterations. For instance, the forward error of the 5th iteration is 41.95 mm. Obviously, it is an outlier input because of a wrong localization of RPM’s pose, while the error can be decreased into a reasonable range after several iterations. At the same iteration, the projected error is 39.77 pixels, but the error is also decreased during the following iterations. The minimum errors were obtained at the 8th iteration with forward error of 1.32 mm and the backward error of 0.86 pixels, respectively.

5 Conclusion

In this paper, we presented a progressive strategy based on an invariant dot to accomplish the hand-eye calibration for laparoscopic surgery navigation. The calibration accuracy was evaluated by forward and backward errors in the laboratory experiments. The results show that the high calibration accuracy can be obtained with a few progressive iterations. In addition, the proposed method provided a way for operators to monitor the calibration procedure; they can stop the calibration when the procedure feedbacks an acceptable accuracy.