Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes

Abstract

Purpose

The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed.

Methods

We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed.

Results

The proposed calibration method was validated using a conventional $30{^{\circ }}$, 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4–8.5 pixel, with image resolution of $1280 \times 720)$ for rotation angles ranged from $-40.3{^{\circ }}$ to $174.7{^{\circ }}$.

Conclusions

We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.

Introduction

Computer-assisted surgery (CAS) is increasingly an integral part of modern patient care. A key capability to enable and expand CAS approaches in endoscopy is the calibration of the rigid endoscope, a process that includes camera calibration and hand-eye calibration (a concept that originates from robotics). Through camera calibration, intrinsic parameters (focal length, principal point, etc.) and distortion coefficients of the camera are determined [1,2,3,4], whereas hand-eye calibration produces the rigid transformation between the camera lens and the tracking device attached to the camera [5]. Endoscope calibration is critical to many CAS applications, including the emerging augmented reality (AR) application, a topic of great interest to our team. In AR, virtual models or tomographic images are overlaid on live endoscopic video to enhance intraoperative visualization. To accomplish this, most reported AR systems rely on external tracking, such as optical tracking [6,7,8] and electromagnetic (EM) tracking [9, 10].

Fig. 1

Left imaging tips of forward- and oblique-viewing rigid endoscopes. Right components of a conventional laparoscope

Two types of rigid endoscopes are common: (1) the forward-viewing endoscope that has a flat lens relative to the camera and (2) the oblique-viewing endoscope that has an angled ($30{^{\circ }}$ or $70{^{\circ }})$ lens relative to the camera (Fig. 1). An angled endoscope has the advantage of offering a much larger field of view through the rotation of its telescope relative to the camera head. However, this rotation also creates a rotation offset between the actual object shown in the camera image and the projected object obtained using the calibrated parameters before rotation (i.e., initial calibration). Although numerous calibration methods for standard camera have been reported, only a countable few groups have developed methods to calibrate oblique-viewing endoscopes and update the initial calibration result after a rotation. A key step in such methods is to track the relative rotation between the two moveable parts of the endoscope. Yamaguchi et al. [11] attached one optical marker on the telescope and another optical marker and a rotary encoder on the camera head to track this rotation. They treated the optical marker on the camera head as the reference, and, in their calibration method, the image plane was fixed. Wu et al. [12] improved the Yamaguchi method by removing the rotary encoder and treating the optical marker on the telescope as the reference. In their method, the hand-eye calibration is preserved during rotation, but the camera image rotates about the rotation center in the image plane. Similarly, De Buck et al. [13] and Feuerstein et al. [14] attached two optical markers on the telescope and the camera head to determine the rotation axis and the rotation angle. De Buck et al. further extended the standard camera model by incorporating functions that accounted for the rotation of the endoscope. The parameters of the functions were obtained by interpolation of a set of previously calculated parameter values. Different from the approaches that rely on external markers, Melo et al. [15] presented a method that used information in the camera image to calculate the rotation center and the rotation angle.

There are several limitations associated with the aforementioned approaches. First of all, optical tracking is not ideal for all clinical applications. The 6-degrees-of-freedom (DOF) optical marker usually has a relatively bulky cross- or star-shaped rigid body with several infrared reflective spheres (or LEDs) mounted on its corners. Both markers on the endoscope maintaining a line-of-sight with the optical camera during the rotation require a bulky configuration of the assembled endoscope, as well as a large physical space for surgeons to perform the rotation. This makes optical tracking challenging in a clinical setting. Second, to obtain the initial calibration, most prior approaches relied on conventional camera calibration methods [1,2,3,4], which require acquisition of multiple (typically 15 or more) images to achieve an acceptable calibration result. This lengthy calibration procedure limits their use in the operating room (OR). The only exception, to our knowledge, is the work of Melo et al. [15], in which the authors used a new method, called single-image calibration (SIC) [16], to initialize the calibration of oblique-viewing endoscopes. Although clinically feasible, the Melo method focused only on calibrating camera intrinsic parameters and distortion coefficients. Therefore, it cannot be used directly in applications such as AR because of the missing hand-eye calibration and external tracking. Third, there is not a universally accepted method to estimate the rotation center in the camera image. Wu et al. [12] assumed the principal point to be the rotation center in the image. However, this is generally not true, as demonstrated by Melo et al. [15], who estimated the rotation center using image features such as the circular image boundary contour and the triangular mark on the image boundary of an oblique-viewing arthroscope. However, these image features are not universally available. For example, most camera images produced by conventional laparoscopes do not show these features.

The purpose of this work was to develop a fast calibration method suitable for OR use for oblique-viewing rigid endoscopes. In our earlier work [17], we developed a fast calibration method for forward-viewing endoscopes using the SIC method [15, 16] and EM tracking. EM tracking reports the location and orientation of a small ($\sim$1 mm diameter) wired sensor inside the magnetic field (i.e., working volume) created by the tracking system’s field generator (FG) [18]. We have extended our previous work to oblique-viewing endoscopes here by mounting two EM sensors, one on the telescope and another on the camera head, thus creating an overall compact configuration. We further developed a SIC plate that can be sterilized for clinical use. In addition, we extended the work of Wu et al. [12] by incorporating SIC and a new method to estimate rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed. Finally, the proposed calibration method was evaluated in a simulated clinical setting.

Materials and methods

Related work

We first briefly review the approach of Wu et al. [12], on which our calibration method is based. Wu et al. attached two optical markers on the endoscope, one on the telescope $(\hbox {OM}_{1})$ and another on the camera head $(\hbox {OM}_{2})$. They used $\hbox {OM}_{1}$ as the reference because it has a fixed geometric relation with the scope lens, so that the hand-eye calibration is preserved during the rotation. Let $p_{\mathrm{OM}_{2}}$ be an arbitrary point in $\hbox {OM}_{2}$’s coordinate system. Its coordinates in $\hbox {OM}_{1}$’s coordinate system can be expressed as

$$\begin{aligned} p_{{\mathrm{OM}}_{1}} ={}^{{\mathrm{OM}}_{1}} T_{\mathrm{OT}} \cdot {}^{\mathrm{OT}} T_{{\mathrm{OM}}_{2}} \cdot p_{{\mathrm{OM}}_{2}} \end{aligned}$$

(1)

where OT refers to the optical tracker and ${}^\mathrm{B} {T}_{\mathrm{A}}$ represents the transformation from A to B.

When rotating the two optical markers relative to each other, $p_{{\mathrm{OM}}_{2}}$’s corresponding coordinates in $\hbox {OM}_{1}$’s coordinate system at various time points are recorded, forming a set of points $P_{{\mathrm{OM}}_{1}} =\left\{ {p_{{\mathrm{OM}}_{1}}^{{t}_{1}},p_{{\mathrm{OM}}_{1} }^{{t}_{2}},\ldots ,p_{{\mathrm{OM}}_{1}}^{{t}_{n}}} \right\}$ where ${t}_{i}$ is the ith time point. The points in $P_{{\mathrm{OM}}_{1}}$ should reside on a circle centered at the rotation center in $\hbox {OM}_{1}$’s coordinate system $O_{{\mathrm{OM}}_{1}}$. For any three points in $P_{{\mathrm{OM}}_{1}}, O_{{\mathrm{OM}}_{1}}$ can be estimated based on the geometric formulas provided in [12]. To improve accuracy and robustness, Wu et al. introduced a RANSAC algorithm to, repetitively, select three random points in $P_{{\mathrm{OM}}_{1}}$ and calculate $O_{{\mathrm{OM}}_{1}}$. For each iteration, the distances between the calculated $O_{{\mathrm{OM}}_{1}}$ and all points (except the three points used to calculate $O_{{\mathrm{OM}}_{1}})$ in $P_{{\mathrm{OM}}_{1}}$ were calculated. Among all iterations, the $O_{{\mathrm{OM}}_{1}}$ that generates the smallest variance of those distances is chosen as the optimum rotation center. Once $O_{{\mathrm{OM}}_{1}}$ is obtained, it is relatively straightforward to calculate the rotation angle given the tracking data before and after the rotation.

EM tracking mounts

As mentioned before, optical tracking-based approaches may not be practical for clinical use. As shown in Fig. 2a, b, the 6-DOF optical marker is relatively bulky in size, and maintaining a line-of-sight with the infrared camera for both markers is challenging and even impossible for certain rotation angles.

In this work, we used a conventional 2D laparoscopic camera (1188 HD, Stryker Endoscopy, San Jose, CA, USA) with a $30{^{\circ }}$ 5-mm telescope, and an EM tracking system with a tabletop field generator (FG) (Aurora, Northern Digital Inc., Waterloo, ON, Canada) and two 6-DOF EM sensors. The tabletop FG is specially designed for OR applications. It is positioned between the patient and the surgical table and incorporates a shield that suppresses distortions to the magnetic field caused by metallic materials below the FG. A recent study showed that the tabletop arrangement could reduce EM tracking error in a clinical environment [19]. As shown in Fig. 2c, we designed and 3D printed (using ABS material) two EM tracking mounts, which can be tightly snapped on the camera head and the telescope/light source, respectively.

In our design, we tried to place the two sensors as far away from the camera head as possible because the camera head causes greater distortion error in EM tracking compared with the telescope. Please refer to our previous works [10, 20] for detailed measurement of EM tracking accuracy when placing the sensor near the laparoscope. During an actual surgery, the camera head is usually positioned higher than the scope lens relative to the patient. Therefore, our designed sensor locations are close to the FG as much as possible without touching the patient. As shown by Nijkamp et al. [21], this configuration yields the best tracking accuracy and stability. Because the sensors and the mounts are sterilizable and will be kept outside the patient’s body, the safety issues associated with clinical use can be managed relatively easily. Compared with optical tracking-based approaches, our approach has the advantages of a much more compact configuration, no concerns regarding a loss of the line-of-sight, and a greater range of possible rotation. It should be noted that there is a small inaccessible rotation angle range $(33.5{^{\circ }}$ out of $360{^{\circ }})$ in our design, which is caused by the light source cable physically blocking the tracking mount on the camera head (please refer to the “Discussion” section for more details on this subject).

Fig. 2

a, b Optical tracking-based configurations in [12, 13], respectively (reproduced with permission). c EM tracking mounts for a conventional $30{^{\circ }}$ 5-mm laparoscope, placed within a 5-mm trocar

Clinical fCalib

A contribution of this work is that we have incorporated the SIC method into the calibration framework for oblique-viewing endoscopes. In our previous work [17], we developed a fast calibration method called fCalib for forward-viewing endoscopes. The method combines the SIC method [15, 16] (Perceive3D, Coimbra, Portugal), which estimates camera intrinsic parameters and distortion coefficients, and the conventional hand-eye calibration so that the complete calibration can be achieved by acquiring a single image of an arbitrary portion of the target pattern. In our earlier work [17], we glued a calibration pattern printed on paper on a plastic plate as a temporary solution. In this work, we have developed a clinical fCalib plate by laser-marking the calibration pattern on Radel$^{{\circledR }}$ polyphenylsulfone, a type of heat and chemical-resistant polymer. As shown in Fig. 3, a 6-DOF EM sensor was permanently embedded in the plate. A tube phantom was fixed on the plate and was registered with the sensor. The tube is used for quick visual evaluation of the calibration accuracy by overlaying a virtual tube model on the camera image and comparing it with the actual tube shown in the camera image. Please refer to [17] for details of the calibration and evaluation methods associated with fCalib. The clinical fCalib plate can be sterilized using autoclave, which is necessary for fast endoscope calibration in the OR.

Fig. 3

Clinical fCalib plate

Fig. 4

Relative rotation of approximately $180^{\circ }$ between the telescope and the camera head

Calibration steps

When the telescope is stationary and only the camera head is rotated (this can be achieved by translation between coordinate systems according to Eq. 1), the camera image rotates about a point in the image plane, i.e., the rotation center in the image $O_{\mathrm{IMG}}$. Wu et al. [12] assumed the calibrated principal point $C=\left( {{C}_{x},{C}_{y}} \right)$ to be $O_{\mathrm{IMG}}$. However, this is not generally true as explained in [15], in which the authors showed that the principal point also rotates about $O_{\mathrm{IMG}}$ while rotating the camera head relative to the telescope. Let $C({0^{\circ }})$ be the principal point calibrated at an initial state. A generic estimation of $O_{\mathrm{IMG}}$ would be the midpoint of the line segment connecting $C({0^{\circ }})$ and $C({180^{\circ }})$, i.e.,

$$\begin{aligned} O_{\mathrm{IMG}} =\frac{C\left( {0^{\circ }} \right) +C\left( {180^{\circ }} \right) }{2} \end{aligned}$$

(2)

where $C({180^{\circ }})$ is the principal point estimated after a relative rotation of $180^{\circ }$ from the initial state (Fig. 4). With the use of fCalib, this can be achieved relatively fast and easily. Based on this estimation of $O_{\mathrm{IMG}}$, we developed the following calibration method:

• (1) Obtain the rotation center in EMS $_{1}$ ’s (EM sensor on the telescope) coordinate system $O_{\mathrm{EMS}_{1}}$. This can be achieved using the Wu method [12] as described in the “Related work” section. In particular, we recorded EM tracking data at a frequency of 12 Hz for 15 s while rotating the camera head relative to the telescope. This yielded a total of 180 sample points located on a circle centered at $O_{\mathrm{EMS}_{1}}$ (after applying Eq. 1). We calculated $O_{{\mathrm{EMS}}_{1}}$ using the RANSAC algorithm with 2000 loops. The net calculation time for calculating $O_{\mathrm{EMS}_{1}}$ was <0.5 s.

• (2) Obtain the first calibration using fCalib and record the current poses of the two sensors (Pose 1). Calibration results include camera intrinsic parameters, distortion coefficient, and extrinsic parameters (results of the hand-eye calibration). Root-mean-square (RMS) re-projection error associated with the calibration results was recorded. As reported in [17], the average time of calibration using fCalib was 14 s.

• (3) Rotate the endoscope $180{^{\circ }}$ from Pose 1 (Fig. 4). Given $O_{\mathrm{EMS}_{1}}$ obtained in Step 1 and Pose 1 obtained in Step 2, any relative rotation angle $\theta$ from Pose 1 can be calculated. Because it is not possible to manually rotate exactly $180{^{\circ }}$ from Pose 1, we considered $\theta \in \left[ {175^{\circ },185^{\circ }} \right]$ to be a good candidate, which can often be achieved through one or two adjustments of the rotation.

• (4) Obtain the second calibration using fCalib, record Pose 2, and calculate $O_{\mathrm{IMG}}$ according to Eq. 2. This completes the calibration. Between the two calibrations, the one with the smaller RMS re-projection error will be set as the Initial Calibration and its pose will be set as the Initial Pose.

After the calibration, the rotation angle $\theta$ can be calculated based on $O_{\mathrm{EMS}_{1}}$ and the Initial Pose. The camera matrix can then be updated based on $\theta , O_\mathrm{IMG}$, and the Initial Calibration. The calibration method was implemented using C$++$ on a laptop computer with 4-core 2.9 GHz Intel CPU and 8 GB of memory. OpenCV (Intel Corp., Santa Clara, CA, USA) functions and Perceive3D’s SIC software were incorporated into the calibration software. It should be noted that the extrinsic parameters and distortion coefficient are not supposed to change with rotation.

Updating the camera matrix

In this section, we describe the formulas for updating the camera matrix with respect to clockwise (i.e., generating a clockwise rotation in the camera image) and counterclockwise rotations. Let $({{x}_{\mathrm{d}},{y}_{\mathrm{d}}})$ be the normalized pinhole projection after lens distortion, and $({{x}_{\mathrm{p}},{y}_{\mathrm{p}}})$ be its corresponding pixel coordinates in the image. We have

$$\begin{aligned} \left[ {{\begin{array}{c} {{x}_{\mathrm{p}}} \\ {{y}_{\mathrm{p}}} \\ 1 \\ \end{array}}} \right] =K\left[ {{\begin{array}{c} {{x}_{\mathrm{d}}} \\ {{y}_{\mathrm{d}}} \\ 1 \\ \end{array}}} \right] \end{aligned}$$

(3)

where K is the camera matrix and can be simplified as

$$\begin{aligned} K=\left[ {{\begin{array}{c@{\quad }c@{\quad }c} {{f}_{x}}&{} 0&{} {{C}_{x}} \\ 0&{} {{f}_{y}}&{} {{C}_{y}} \\ 0&{} 0&{} 1 \\ \end{array}}} \right] \end{aligned}$$

(4)

where ${f}_{x}$ and ${f}_{y}$ are the focal lengths and C is the principal point. We assume the camera is skewless, which is often the case for endoscopes. Let $O_{\mathrm{IMG}} =({O_x,{O}_{y}})$ be the rotation center in the image, and $R_\theta ^+$ be the conventional counterclockwise rotation matrix (we defined the counterclockwise rotation to be positive). Thus, the corrected projection after a counterclockwise rotation of $\theta$ about $O_{\mathrm{IMG}}$ can be expressed as

$$\begin{aligned} \left[ {{\begin{array}{c} {{x}_{\mathrm{c}}} \\ {y_{\mathrm{c}}} \\ 1 \\ \end{array}}} \right]= & {} R_\theta ^+ \left[ {{\begin{array}{c} {{x}_{\mathrm{p}} -O_{x}} \\ {{y}_{\mathrm{p}} -O_{y}} \\ 1 \\ \end{array}}} \right] +\left[ {{\begin{array}{c} {O_{x}} \\ {O_{y}} \\ 1 \\ \end{array}}} \right] \\= & {} \left[ {{\begin{array}{c@{\quad }c@{\quad }c} {\cos \theta }&{} {-\sin \theta }&{} 0 \\ {\sin \theta }&{} {\cos \theta }&{} 0 \\ 0&{} 0&{} 0 \\ \end{array}}} \right] \left[ {{\begin{array}{c} {{f}_{x} {x}_{d} +{C}_{x} -{O}_{x}} \\ {{f}_{y} {y}_{d} +{C}_{y} -{O}_{y}} \\ 1 \\ \end{array}}} \right] \\&+\left[ {{\begin{array}{c} {{O}_{x}} \\ {{O}_{y}} \\ 1 \\ \end{array}}} \right] \\= & {} \left[ {{\begin{array}{c@{\quad }c@{\quad }c} {\cos \theta }&{} {-\sin \theta }&{} {\left( {1-\cos \theta } \right) {O}_{x} +\sin \theta \cdot {O}_{y}} \\ {\sin \theta }&{} {\cos \theta }&{} {-\sin \theta \cdot {O}_{x} +\left( {1-\cos \theta } \right) {O}_{y}} \\ 0&{} 0&{} 1 \\ \end{array}}} \right] \\&\cdot \left[ {{\begin{array}{c@{\quad }c@{\quad }c} {{f}_{x}}&{} 0&{} {{C}_{x}} \\ 0&{} {{f}_{y}}&{} {{C}_{y}} \\ 0&{} 0&{} 1 \\ \end{array}}} \right] \left[ {{\begin{array}{c} {{x}_{\mathrm{d}}} \\ {{y}_{\mathrm{d}}} \\ 1 \\ \end{array}}} \right] \\= & {} R_{\theta ,O_{\mathrm{IMG}}}^+ K\left[ {{\begin{array}{c} {{x}_{\mathrm{d}}} \\ {{y}_{\mathrm{d}}} \\ 1 \\ \end{array}}} \right] \end{aligned}$$

Similarly, the rotation matrix for the clockwise rotation can be expressed as

$$\begin{aligned} R_{\theta ,O_{\mathrm{IMG}}}^- =\left[ {{\begin{array}{c@{\quad }c@{\quad }c} {\cos \theta }&{} {\sin \theta }&{} {\left( {1-\cos \theta } \right) {O}_{x} -\sin \theta \cdot {O}_{y}} \\ {-\sin \theta }&{} {\cos \theta }&{} {\sin \theta \cdot {O}_{x} +\left( {1-\cos \theta } \right) {O}_{y}} \\ 0&{} 0&{} 1 \\ \end{array}}} \right] \end{aligned}$$

(5)

For implementation, it is straightforward to use the above formulas for correcting rotation offset, i.e., by multiplying $R_{\theta ,O_{\mathrm{IMG}}}^+$ or $R_{\theta ,O_{\mathrm{IMG}}}^{-}$ on the left of the initial camera matrix.

Let $p_{\mathrm{EMS}_{2}}$ be an arbitrary point in $\hbox {EMS}_{2}$’s (EM sensor on the camera head) coordinate system, and $p_{\mathrm{EMS}_{1}}^{\mathrm{initial}}$ be its corresponding coordinates in $\hbox {EMS}_{1}$’s coordinate system at the Initial Pose. After a new rotation, the corresponding coordinates of $p_{\mathrm{EMS}_{2}}$ in $\hbox {EMS}_{1}$’s coordinate system changes to $p_{\mathrm{EMS}_{1}}$. The direction of rotation (clockwise or counterclockwise) can be determined according to

$$\begin{aligned} \mathrm{sgn}\left( {\left[ {\left( {O_{\mathrm{EMS}_{1}} -p_{\mathrm{EMS}_{1}}^{\mathrm{initial}} } \right) \times \left( {O_{\mathrm{EMS}_{1}} -p_{\mathrm{EMS}_{2}}} \right) } \right] _z} \right) \end{aligned}$$

(6)

where $O_{\mathrm{EMS}_{1}}$ is the obtained rotation center in $\hbox {EMS}_{1}$’s coordinate system.

Experiment 1

We first evaluated our method to obtain the rotation center in the image $O_\mathrm{IMG}$. After attaching the EM tracking mounts, a team member repetitively performed five freehand calibrations following the described calibration steps (Steps 1–4), and the results of these calibration trials were recorded and analyzed. The starting relative angle between the two parts of the laparoscope was gradually increased approximately $30{^{\circ }}{-}40{^{\circ }}$ between two consecutive calibration trials. When acquiring images, the distance between the laparoscope lens and the center of the fCalib plate ranged from 7 to 9 cm, which fell in the typical distance range when using such a laparoscope clinically.

Fig. 5

Simulated clinical environment for validation of calibration accuracy

Experiment 2

In Step 3 of our calibration method, we require a rotation of $180{^{\circ }}\pm 5{^{\circ }}$ from Pose 1 to yield Pose 2. To investigate the influence of violation to this rule, we performed additional calibration trials, in which the rotation angles between Pose 1 and Pose 2 were approximately $170{^{\circ }}, 160{^{\circ }}$ and $150{^{\circ }}$.

Experiment 3

We subsequently validated the static calibration accuracy of the proposed method. Because EM tracking accuracy is susceptible to the presence of metallic and conductive materials, we performed experiments in a simulated clinical environment, as shown in Fig. 5. The tabletop EM FG was placed on a surgical table. A plastic laparoscopic trainer that simulates patient’s abdomen was placed on the FG. Two common laparoscopic surgical tools, one grasper and one pair of scissors, were inserted into the trainer through trocars. To simulate the use of a laparoscope in an ultrasound-based AR system, a laparoscopic ultrasound (LUS) probe was inserted into the trainer. The LUS probe was connected to the ultrasound scanner, and the scanner was kept on throughout the experiments. The fCalib plate was placed inside the trainer and was used only for the corner point detection purpose in these experiments. The laparoscope was inserted into the trainer through a 5-mm trocar and was held in place using a stand made of LEGO$^{{\circledR }}$ bricks to eliminate hand tremor.

We used the calibration results ($O_{\mathrm{EMS}_{1}}, O_{\mathrm{IMG}}$, Initial Calibration, Initial Pose) from one of the five freehand calibration trials in Experiment 1. We slightly rotated the camera head relative to the telescope, a few angles at a time, both clockwise and counterclockwise. After each rotation, a picture of the fCalib pattern was acquired and corner points were automatically detected from the picture. The rotation angle $\theta$ was calculated based on $O_{\mathrm{EMS}_{1}}$, the Initial Pose, and the current pose (tracking data of the two sensors). The rotation-corrected projection of corner points ($p_{\mathrm{cor}})$ were obtained based on $O_{\mathrm{IMG}}, \theta$, and the Initial Calibration, and were compared with the detected corner points $(p_{\mathrm{det}})$ using the RMS re-projection error, which is defined as

$$\begin{aligned} \mathrm{error}=\sqrt{\frac{1}{N}\sum \nolimits _{i=1}^N d\left( {p_{\mathrm{cor}}^{i},p_{\mathrm{det}}^i} \right) ^{2}} \end{aligned}$$

(7)

where N is the number of detected corner points and $d\left( {\cdot ,\cdot } \right)$ is the Euclidean distance in pixel. It is worth mentioning that the SIC method [15, 16] will detect as many corner points as possible in any visible part of the calibration pattern.

Experiment 4

To further evaluate dynamic calibration accuracy for practical use, we visually examined the virtual tube overlay using the fCalib plate. A feature of fCalib is the ability to immediately check the calibration accuracy by overlaying a virtual tube model on the camera image [17]. We used this feature and overlaid the rotation-corrected virtual tube on the image. A good visual agreement between the virtual and the actual tubes shown in the image suggests accurate calibration and rotation correction.

Table 1 Results from Step 1 of the five freehand calibrations

Fig. 6

Comparison of the rotation center in the image $O_{\mathrm{IMG}}$ (triangle), the principal points (square), and the image center (star). Each $O_{\mathrm{IMG}}$ was calculated using two principal points of the same color. The image resolution was $1280 \,\times \, 720$ pixels.

Table 2 Distance from $O_{\mathrm{IMG}}^{\mathrm{ref}}$ to $O_{\mathrm{IMG}}$ (with various $\theta$) and image center

A team member performed calibration of the oblique laparoscope following the described procedure. The laparoscope was then inserted into the trainer along with two other surgical tools. The fCalib plate was also placed inside the trainer so that the tube could simulate a target structure such as blood vessel or bile duct. The team member rotated the two parts of the laparoscope by random angles both clockwise and counterclockwise. During the rotation, the team member held the telescope relatively stable and rotated the camera head such that the tube stayed in the field of view. The virtual tube model, generated before and after rotation correction, can be visualized in the video. To trigger rotation correction, the team member pressed a button using a foot pedal. After rotation correction, the team member moved the laparoscope around to visually assess the accuracy of the overlay between the actual and the virtual tubes shown in the video.

Results

Experiment 1

It took an average of 2 min and 8 s (range 1 min and 50 s–2 min and 25 s) to complete one calibration. Table 1 lists the results from Step 1 of the five calibration trials. It indicates the estimated $O_{\mathrm{EMS}_{1}}$ are consistent. Figure 6 shows the estimated principal points (two in each calibration), the calculated $O_{\mathrm{IMG}}$, and the image center [i.e., (640, 360) in our case] as a reference. The actual rotation angles between the First and the Second Calibrations (the ideal angle is $180{^{\circ }})$ ranged from $176.5{^{\circ }}$ to $179.7{^{\circ }}$. As can be seen, the calculated $O_{\mathrm{IMG}}$ was stable and differed considerably from the respective principal points and less so from the image center. Except for the varying principal points, other calibrated parameters such as the focal length and the distortion coefficient were consistent among different calibration trials and comparable to our previous results reported in [17]. The 10 SICs (two in each calibration) yielded $1.2\pm 0.2$ pixel RMS re-projection error.

Fig. 7

RMS re-projection error comparing the rotation-corrected projection of corner points and the detected corner points. The red solid line is our method, whereas the black dashed line refers to the method using the image center as the rotation center in the image

Fig. 8

Left rotation-corrected projection of corner points superimposed on the original image. The image also shows the overlay of the rotation-corrected virtue tube (a series of rings) and the actual tube. Right close-up view showing the rotation-corrected projection of corner points (red dot) and the detected corner points (yellow triangle). As a reference, the physical size of the edge of each square in the fCalib pattern is 3.2 mm

Fig. 9

Three snapshots of the submitted video clip. The video shows handling of the laparoscope as well as the overlay of the virtual and the actual tubes. a The initial state before rotation. b The laparoscope was rotated, and rotation correction in the image was not applied (before pressing the button). c Rotation correction in the image applied (after pressing the button)

Experiment 2

Based on the results from Experiment 1, it is reasonable to assume the ground truth $O_{\mathrm{IMG}}^{\mathrm{ref}}$ to be the average of the five calculated $O_{\mathrm{IMG}}$. Let $\theta$ be the rotation angle between Pose 1 and Pose 2. Table 2 shows the distances from $O_{\mathrm{IMG}}^{\mathrm{ref}}$ to: (1) $O_{\mathrm{IMG}}$ of the five calibrations in Experiment 1, (2) the image center, and (3) $O_{\mathrm{IMG}}$ of the three additional calibration trials in Experiment 2. The results suggest a rotation of $180{^{\circ }}\pm 5{^{\circ }}$ between Pose 1 and Pose 2 is necessary.

Experiment 3

Figure 7 shows the RMS re-projection error using our method (range 2.4–8.5 pixel). As a reference, we also showed the re-projection error with the approach using image center as the rotation center in the image. Combined with Table 2, it can be seen that more distance from $O_{\mathrm{IMG}}^{\mathrm{ref}}$ yielded worse RMS re-projection error.

As a qualitative evaluation of our method’s static accuracy, we superimposed rotation-corrected projection of corner points on the original images at three different rotation angles (Fig. 8). In the close-up views of Fig. 8, we showed the detected corner points and the rotation-corrected projection of corner points.

Experiment 4

A video clip showing handling of the laparoscope as well as the overlay of the virtual and the actual tubes has been supplied as supplementary multimedia material. Three sample snapshots of the video are shown in Fig. 9. In general, there is good spatial agreement between the actual tube and the rotation-corrected virtual tube, and the correction came into effect in near real time after the button was pressed.

Discussion and conclusions

In this paper, we have reported a fast calibration method for oblique-viewing rigid endoscopes. In contrast to previous approaches, we used EM tracking as the external tracking hardware to improve the compactness and practicality of the system configuration. To enable OR use, single-image calibration was incorporated into the method and a sterilizable plate, laser-marked with a calibration pattern, was developed. Furthermore, we proposed a universally accepted method to estimate the rotation center in the camera image without relying on image features that may or may not be present. The proposed method was validated both qualitatively and quantitatively in a simulated clinical environment.

One major advantage of our method is that it is a fast and easy calibration procedure that has the potential to be performed by the clinical staff in the OR. The process is expected to add about 2 min to the existing clinical workflow; however, it can be performed in parallel with the preparation of the CAS system that employs the oblique-viewing endoscope. Our future work will include training the clinical staff to perform and test our calibration method through animal and human experiments to examine the learning curve, the ease of use, the actual time spent, and the accuracy of the method in practical situations.

Our method constitutes an improvement on the Wu method [12] with a better estimation of the rotation center in the image $O_{\mathrm{IMG}}$. It is worth mentioning that, other than using Eq. 2, it is possible to calculate the exact $O_{\mathrm{IMG}}$ based on the two principal points and the rotation angle. However, solving this problem mathematically would yield two possible solutions. One may consider using the distance to the image center as a criterion to choose from the two solutions. However, the differences among the principal point, the image center and $O_{\mathrm{IMG}}$ could vary from endoscope to endoscope. Investigating more types of endoscopes in the future would help us better understand these differences. Currently, our method is general enough for stable and accurate estimation of $O_{\mathrm{IMG}}$.

For an approximate comparison of re-projection error reported in Fig. 7, Yamaguchi et al. [11] achieved less than 5 pixel re-projection error for a rotation in the $\left[ {0^{\circ },140^{\circ }} \right]$ range. Wu et al. [12] achieved a similar accuracy for angles within $75^{\circ }$; however, their re-projection error increased to 13 pixels when the angle increased to $100^{\circ }$. It should be noted that the image resolution in both these studies was less than or equal to $320\, \times \, 240$, whereas the image resolution in our study is $1280\, \times \, 720$, i.e., a 12-fold denser pixel matrix. In addition, our results compared more corner points on image periphery whereas the results in the above two studies compared only corner points close to the image center because their calibration pattern was the conventional checkerboard.

Wu et al. reported that the re-projection error increased greatly when the rotation angle increased beyond $75^{\circ }$. We observed a similar trend in our results: the error approximately doubled when the rotation angle exceeded $80^{\circ }$. A similar pattern was also presented in the results of the Melo work [15]. All the three methods are based on rotating the image plane to correct for the rotation offset. Thus, the error in the estimation of $O_{\mathrm{IMG}}$ appears to be one major contribution to the re-projection error. Another source of error is under- or overestimation of the rotation angle, which is calculated based on the estimated rotation center in 3D space $O_{\mathrm{EMS}_{1}}$ and the positions of the two EM sensors. As shown in Table 1, although x- and y-coordinates of $O_{\mathrm{EMS}_{1}}$ are quite stable, there is more variation in the estimated z-coordinate of $O_{\mathrm{EMS}_{1}}$, which could cause small error in the estimation of $O_{\mathrm{EMS}_{1}}$. Both of the errors in $O_\mathrm{IMG}$ and $O_{\mathrm{EMS}_{1}}$ will have a greater impact on larger rotation angles than smaller ones, causing the angle-dependent re-projection error as shown in Fig. 7. Nevertheless, based on the results shown in Fig. 8 and the accompanying video (Fig. 9), we do not anticipate rotation correction to fail using our method for larger rotation angles. In the unlikely event that the error turns out to be unacceptable after a large rotation, re-calibration using fCalib to reset the Initial Calibration and the Initial Pose continues to be an option. For example, in ultrasound-based AR applications, the top edge of the overlaid ultrasound image is expected to align with the bottom edge of the imaging elements of the laparoscopic ultrasound transducer shown in the video (from certain viewing perspectives). One can re-initialize calibration if the alignment deteriorates significantly after a large rotation.

In addition to rotation-related errors, there are also errors resulting from EM tracking and the Initial Calibration. As shown in the video (Fig. 9), the overlay error starts to change direction (relative to the center line of the actual tube) when the tube crosses the center of the camera image. This could be an effect caused by both dynamic EM tracking error [18] and the error in radial distortion estimation, which causes extra inward or outward displacement of a point from its ideal location. It should be noted that this phenomenon also occurs for forward-viewing endoscopes.

In our current design of EM tracking mounts, $33.5{^{\circ }}$ out of $360{^{\circ }}$ of rotation angles cannot be reached because the tracking mount on the camera head blocks the light source cable. An alternative option is to shorten the tracking mount on the camera head so nothing restricts a complete 360-degree rotation. However, this option will place the EM sensor closer to the camera head and further from the FG, which could create unacceptable EM tracking error. We favored better accuracy over greater flexibility at this stage. Nevertheless, the majority of rotation angles are still accessible and we anticipate the small inaccessible angle range would not affect practical use of the oblique-viewing endoscopes.

In conclusion, demonstrated in the laboratory setting, the presented method enables fast and accurate calibration of oblique-viewing rigid endoscope. Our work has the potential to make such fast calibration possible in the actual clinical setting as well and thus support clinical translation of many emerging endoscopic CAS systems.