Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Trans-esophageal ultrasound is a manually controlled imaging modality widely used for diagnosing heart disease and guiding cardiac surgical procedures [1]. The on-site operation of the probe usually requires operators standing for long periods of time and wearing heavy radiation-protection shielding when X-ray is utilized during the surgery [2]. Apart from the inconvenience and tedium of the manual control, the need for highly specialized skills is always a barrier for reliable and repeatable acquisition of ultrasound. Accordingly, there is a need for an automatic TEE system and method to acquire the desired imaging based on the user’s request. Though numerous works have been presented for robotic ultrasound as reviewed in [3], there is no solution for automatic heart scanning with TEE produced so far due to the complexity of heart imaging and the unavailability of a robot specifically designed for this task.

A recently developed robotic system for TEE has made remote control possible [4] and we have subsequently proposed an automatic acquisition workflow (as shown in Fig. 1) [5] using this robot based on a view-planning platform for path-panning and an ultrasound-to-MR registration method [6] for locating the probe position when applying feedback. However, in the workflow described in [5], the probe tracking method based on registering 3-D echo images to pre-scanned MR models requires a close estimate of the probe pose. This was estimated based on the robotic kinematics, which could result in failures because the mechanical performance of the probe driven by the robot mechanism within the real esophagus could be different to the kinematics in the simulation environment. Therefore, a method of more reliable tracking of the probe is a key component for automatic acquisition to be clinically transferable. As an alternative option, EM tracking systems are widely used for medical device tracking and have been reported for tracking the TEE probe [7]. However, registering the EM tracking coordinates to the patient coordinates is required which is difficult to achieve using the EM tracking system on its own and the accuracy of EM tracking could also be influenced by the electromagnetic environment.

Fig. 1.
figure 1

Overview of the TEE add-on robotic system and the automatic acquisition workflow.

Full size image

To solve the problem of reliable tracking, we introduce a method to combine the EM tracking system with image-based registration for probe tracking and integrate this method into the workflow as shown in Fig. 1. This tracking method is tested with a phantom experiment in which the tracking information provides feedback for the robot. The performances of the combined tracking method and the automatic acquisition are analyzed and discussed. In this paper, the robotic system and the view-based motion planning are briefly reviewed in Sect. 2.1. Details of the new probe tracking method are presented in Sect. 2.2. The utilization of this new tracking method in feedback position control, relying on the inverse kinematics, is introduced in Sect. 2.3. Based on these methods, experiments, results, discussion and conclusions are presented in the subsequent sections.

2 Materials and Methods

2.1 Robotic System and View-Based Motion Planning

An overview of the robotic system is shown in Fig. 2(a). The add-on TEE robot holds the probe handle and manipulates four degrees of freedom (DOFs) that are available in manual handling of the probe, including the rotation about and translation along the length of the TEE probe and additional manipulators with 2-DOFs to steer the probe head. The remote operation of the probe is via Bluetooth communication. More details of the design of the robotic system can be found in [4].

Fig. 2.
figure 2

(a) Overview of the mechanical design of the robotic TEE system with its mechanisms shown. (b) The view-planning platform with the function of the platform and an example defined view shown.

Full size image

In the view-planning platform (Fig. 2(b)) described in [5], an automatically segmented heart 3-D model from the pre-scanned MR image, the corresponding manually segmented esophagus center line, and the virtual model of the TEE probe head can be loaded and viewed intuitively. The forward kinematics of the probe is modeled and the corresponding virtual 2-D ultrasound images are displayed based on the given robotic parameters [4]. By defining targeted views based on the virtual ultrasound image outputs, sets of robotic parameters, along with planed paths for the robotic movements, can be obtained.

In addition, the view-planning platform has the capability of auto-patient adaption, in which case standard TEE views of patient-specific data can be automatically obtained based on registration and optimization methods. This function allows rapid motion planning of the acquisition if standard TEE views in the protocol are required as targets. Details of the auto-patient adaption method can be found in [5].

2.2 Probe Calibration and Tracking Method

The proposed probe tracking method uses the combination of image-based registration and an EM tracking system. The registration [6] takes a 3-D full-volume ultrasound image, registers to a pre-scanned MR image and obtains the probe pose as the result. This registration can provide an accurate probe pose but requires a close initial estimation. The method has previously been shown to have a capture range of 9 mm [9]. We therefore introduce EM tracking to the workflow, which typical has accuracy within this range, to provide the initial estimation and ensure the success of the registration.

Spatial Calibration.

In the workflow, EM tracking is done using the Aurora Electromagnetic Measurement System (Northern Digital Inc, Waterloo, Canada). An EM sensor has been mounted onto the tip of the probe. Spatial calibration of the ultrasound image to the EM tracker was by a simple registration-based method using a phantom comprising several crossed wires [8]. The experimental setup and workflow for the calibration are shown in Fig. 3(a). 3-D images of the wires were acquired at different positions and orientations, and the straight lines and crossing points of the wires were extracted manually. The calibration transformation \( {}^{{{\mathbf{tracker}}}}{\mathbf{T}}_{{{\mathbf{US}}}} \) was then solved for iteratively to minimise the misalignment of the extracted wires in each position. From this the targeted transformation \( {}^{{{\mathbf{tracker}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \) was obtained using a prior known transformation from the probe coordinates to the ultrasound image coordinates. Using this spatial calibration, a measured pose of the EM tracker in the EM coordinates (defined by the Aurora field generator) can be converted to the pose of the probe in the EM coordinates \( {}^{{{\mathbf{EM}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \).

Fig. 3.
figure 3

(a) Experimental setup and workflow for the calibration of the probe. (b) Flow diagram of the tracking method used in the TEE automatic acquisition.

Full size image

Initialization.

In order to provide the probe pose in the MR coordinates, another calibration between the EM coordinates and the MR coordinates must be obtained. This can be done at the time when the probe is manually inserted into the esophagus at a random starting pose with the probe pointing towards the heart. By visually looking at the first output ultrasound image, a similar view can be manually selected from the view-planning platform, giving an estimate of the current probe pose in the MR coordinates \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{initial}}} \right) \). The ultrasound image is then registered to the MR image starting from this estimation, giving an accurate probe pose in the MR coordinates \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{reg}}} \right) \). The current probe pose \( {}^{{{\mathbf{EM}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \) is also measured from the EM system. Therefore, the transformation from the EM coordinates to the MR coordinates \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{EM}}}} \) can be obtained. It is important to understand that this is a once-only manual operation for each TEE scan of a patient. Clinically, this could be achieved relatively easily by an experienced TEE operator by requiring that they position the probe to one of the standard views when it is first inserted into the esophagus.

Combined Tracking.

After initialization, the probe pose \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}}} \right) \) can be tracked automatically in any position inside of the esophagus by the EM tracking system using \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{EM}}}} \) and \( {}^{{{\mathbf{EM}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \). To further eliminate the influence of the environment on EM tracking and the inaccuracy of the initial calibration \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{EM}}}} \), registration of the current ultrasound image to the MR image is performed from the current \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}}} \right) \). As described in [6], an ultrasound-like image is generated from the MR using the acoustic property information and an ultrasound imaging model. This is then registered to the real US image using a monogenic phase similarity measure. The optimization method attempts to maximize this similarity measure to find the true pose of the TEE probe relative to the MR coordinates \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}} - {\mathbf{reg}}} \right) \), which is used to provide feedback for the robot control. The overview of the initialization and combined tracking are shown in Fig. 3(b).

2.3 Inverse Kinematics and Feedback

The tracking result is used as feedback information for the robotic system to adjust the parameters required to obtain the targeted positions. In order to find the robotic parameters’ offsets between the current pose and the targeted pose, an inverse kinematic model is proposed using a gradient decent search strategy based on the forward kinematics reported before. The search strategy defines a single objective function in order to optimize the robotic parameters \( p = \left( {{\rm{x, }} \uptheta, \upalpha, \upbeta } \right) \), where x is the translation parameter, θ is the axial rotation parameter, and α, β are the bi-directional bending parameters. The forward kinematics, denoted as F, gives the transformation from the probe coordinates to the MR coordinates:

$$ {}^{{\rm{MR}}}{{\rm{T}}_{{\rm{probe}}}} = {\rm{F}}\left( {\bf{p}} \right) = {\rm{F}}\left( {{\rm{x, }}\uptheta , \upalpha ,\upbeta } \right) $$
(1)

Detailed information on the forward kinematics is given in [4]. The objective function uses the four corners of the probe transducer face as reference points, denoted as \( {\mathbf{R}}_{{\mathbf{i}}} \). The current pose of the probe is denoted as \( {}^{\text{MR}}{\text{T}}_{{{\text{probe}}^{ * } }} \), and the objective function f(p) used for optimization is defined as follows:

$$ {\text{f}}\left( {\mathbf{p}} \right) = - \frac{1}{4}\sum\nolimits_{i = 1}^{4} {\left\| {{}^{\text{MR}}{\text{T}}_{{{\text{probe}}^{ * } }} {\text{R}}_{\text{i}} - {\text{F}}\left( {\mathbf{p}} \right){\text{R}}_{\text{i}} } \right\|} $$
(2)

During the search and step approach, the parameter pi in the parameter space p which gives the maximum partial derivative will be selected as the step parameter to be updated. The best step direction \( {\text{d}}_{\text{i}} = - \nabla {\text{f}}\left( {{\text{p}}_{\text{i}} } \right)/\left\| {\nabla {\text{f}}\left( {{\text{p}}_{\text{i}} } \right)} \right\| \) in the step direction space d is the forward direction of the selected parameter. The step size, \( \upsigma \), is initially defined based on the dimension scale of each parameter and then reduced after each convergence when \( \nabla {\text{f}}\left( {{\text{p}}_{\text{i}} } \right) = 0 \). A new parameter set \( {\mathbf{p}}_{{\mathbf{ + }}} \) is of the form:

$$ {\mathbf{p}}_{ + } = {\mathbf{p}} +\upsigma \cdot {\mathbf{d}} $$
(3)

The search strategy starts from p = 0 and ends when f(p) reaches its minimum preset value. The final parameter set \( {\mathbf{p}}^{ * } \), representing the current pose of the probe, is the output of the inverse kinematics. With the tracking information and the inverse kinematics, a simple feedback position controller is designed as shown in Fig. 4. Based on the result from the previous work [5], the cycle of measurement and adjustment is executed only one time, which has been shown to effectively improve the accuracy.

Fig. 4.
figure 4

Feedback position controller based on the tracking and kinematics of the TEE robot.

Full size image

2.4 Automatic Acquisition Experiment

A phantom experiment was designed to test the proposed tracking method and its performance in automatic TEE acquisition. A custom phantom was built in order to provide a simulation environment for the TEE approach. This phantom includes a silicone tube representing the esophagus and a commercial ultrasound/MRI heart phantom (Computerized Imaging Reference Systems, Incorporated (CIRS), USA.) used for imaging. The heart and silicone tube model of the phantom were extracted from the pre-scanned MR image and loaded into the view-based robot planning platform. Based on featured structures (chambers, valves, vessels) shown in either long-axis view or short-axis view, five views were defined and the corresponding probe poses and robotic parameters were recorded. Mechanically, a special link mechanism was designed in order to lead the endoscopic portion of the TEE probe translating into the phantom. The experimental setup is show in Fig. 5.

Fig. 5.
figure 5

Experimental setup for the automatic acquisition using the TEE robot, custom heart-esophagus phantom, and the EM tracking system.

Full size image

During the experiment, the initial calibration between the EM coordinates and the MR coordinates was performed at the very beginning using the method described in Sect. 2.2. After initialization, the probe was tracked by the proposed tracking method. Based on the pre-planned poses of the probe, the robotic system was actuated, driving the probe towards the targeted poses. When the probe arrived, the tracked probe pose was used as feedback information with the inverse kinematics method described in Sect. 2.3. The adjustments of the probe parameters were calculated and performed, and the ultrasound image recorded. The experiment was repeated three times with different random initial poses of the probe.

For post-processing, to understand the improvement in accuracy using the combined tracking method over the EM tracking method, the tracked probe poses reported by the EM tracking system \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}}} \right) \) and the combined tracking method \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}} - {\mathbf{reg}}} \right) \) were compared. Root sum square (RSS) of the differences between the X-, Y-, and Z-axes rotation and translation components were calculated after decomposing each matrix. To understand the need for using the EM tracking system to provide the initial estimate of registration for tracking, we used robot kinematics as an alternative initialization for the registrations and compared the success rate with the proposed combined tracking method. The accuracy of automatic acquisition was quantified by comparing the final probe pose determined by registration \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{EM}} - {\mathbf{reg}}} \right) \) with the planned probe pose \( {}^{{{\mathbf{MR}}}}{\mathbf{T}}_{{{\mathbf{probe}}}} \left( {{\mathbf{planned}}} \right) \). 60 marker points were defined in the ultrasound image field of view \( \left( {90\deg \,*\,90\deg {\text{cone}}} \right) \) at a depth of 5–6 cm where most structures of clinical interest during cardiac procedures are located, including major valves and the septum. The locations of corresponding marker points in the MR coordinates were obtained and compared. Additionally, the acquired real ultrasound images were compared with the planned views in the view planning platform visually.

3 Results

Results from the experiments indicate that the proposed tracking method is suitable for the robotic-based automatic TEE acquisition. Figure 6(a) shows one of the tracking examples where the EM tracking provided a close estimation and the image-based registration calculated an accurate probe pose. Visual examination of the registration results found that the combined registration result could not be improved by manual adjustments, whereas the EM-only registration had some clear misalignment. Quantitatively, the proposed combined tracking method has a relatively high tracking accuracy with a median registration error of 2.9 mm. This has been shown previously from the investigation of the registration method itself in [9]. Therefore, the combined tracking result is used as the reference to compare with the EM tracking result. The error in the EM tracking method compared to the combined result indicates an improvement of 7.96 mm and 3.41° in tracking accuracy. The success rate is 11 out of 15 (73%) using the kinematics to provide the initial estimate of registration while all registrations succeed (100%) using the EM tracking result as the initial estimate of registration. For the performance of the automatic acquisition, the ultrasound image was overlaid on the MR segmentation data in order to intuitively compare the acquired ultrasound views with the views originally defined in the view planning software (example views are shown in Fig. 6(b)). The results show that all planned structures were in the 3-D field of view and most of the center slices of the obtained ultrasound images align with the original slice planned in the view planning platform. Quantitatively, the overall error of marker points defined in the ultrasound field of view at the depth of interest over all three experiments is 10.44 ± 2.30 mm (mean ± standard deviation). A histogram of this error is shown in Fig. 6(c).

Fig. 6.
figure 6

(a) Example of tracking with target probe pose (black), EM tracked pose (green), and EM-registration tracked pose (white) shown. (b) Examples of automatic acquisition results with planned view (top row) and acquired real ultrasound images (bottom row) shown. (c) Histogram of the error at 5–6 cm depth. (Color figure online)

Full size image

4 Discussion and Conclusions

The proposed combined tracking method using the image-based registration and an EM tracking system together enables a more accurate tracking performance than using the EM sensor alone. This is because the EM sensor could be influenced by the metallic environment and an inaccurate calibration. Compared with using kinematics as the initial estimate of registration, the EM tracking system provides a more reliable estimate and ensures the success rate of the tracking. Therefore we believe the proposed combined tracking method is suitable for the automatic TEE acquisition in terms of both accuracy and reliability. It should be noted that our gold standard for the error measurement of the tracking method was to run the registration from a good initial alignment, and then to manually correct any visible alignment errors, although in the experiment there was almost no visible misalignment after this registration. Therefore, while the gold standard is not truly independent, we are confident that it is accurate, and certainly shows that registration is better than using EM tracking alone.

As for the accuracy of probe positioning for automatic acquisition, the error in the ultrasound space at the depth of clinical interested due to the probe positioning error is similar to the amount of movement and deformation of the beating heart (1 cm). With this range of error, most of the desired anatomies are still very likely to remain in the field of view in either 2-D or 3-D mode. However, such a deviation might still cause significant challenges for the 2-D mode if a small structure is required in the view plane. In that case, a precision of a few millimeters might need to be achieved. There are a number of error sources contributing to the overall error, including the error from tracking, inaccuracy of the inverse kinematics in the constrained environment of the silicone tube, and mechanical movement. The rigidity of the silicone tube may mean that the feedback is less effective in the phantom than in a human esophagus. Additionally, the probe is constrained to move along the esophagus center line in the view planning platform, which in reality might be different in the silicone tube or real esophagus.

In this paper, we have proposed a method of probe tracking using an EM tracking system and image-based registration to work with a TEE robot. This method is particularly developed for the application of automatic TEE acquisition. Results from the experiment demonstrated the feasibility of the tracking method and proved the new concept of automatic TEE acquisition in a phantom. To further evaluate the method in the human body, specially preserved cadavers using the Thiel embalming method [10] will be employed and the whole workflow will be evaluated in a more realistic clinical scenario. The accuracy requirements for the automatic TEE acquisition workflow in this less rigid environment can be re-evaluated based on a qualitative study judging whether the planned structures are successfully obtained in a real human body. In addition, further developments in automation, particularly in the initialization, will be necessary for clinical translation of the workflow.