Introduction

Every 43 s, an incident occurs in the USA [1], when someone undergoes percutaneous coronary intervention (PCI) procedure. The PCI procedure starts by making an incision in the femoral or radial artery to make a way for the catheter (a thin tube) followed by the guidewire (a thin wire) to the affected location, by looking at two X-ray images (angiogram and fluoroscopy) taken by the C-arm X-ray machine. The guidewire delivers a small balloon to the blocked section of artery, which is then inflated to widen the affected portion of artery, while in the case of stent, it is left permanently to resume the free blood flow [2, 3]. Depending on the curvature of the coronary vessels and degree of the stenosed portion, guidewire’s effective manipulation sometimes accounts for a substantial portion of the operation completion time and thus surgeon skillset is crucial. A typical operating room for conventional coronary intervention procedure, shown in Fig. 1a, may be replaced by a tele-robotic system shown in Fig. 1b, where surgeon remotely manipulates catheter/guidewire sited on a slave device, through the motion of a handheld master device. A lot of work is being carried out for employing robots in the in vivo surgical operations for their potential applications and benefits [4, 5]. Robot assistance [6] may help surgeon, but it has potential to provide haptic feedback as well. Haptic feedback is one of the many attributes whose importance in robot-assisted surgery is openly acknowledged by several research groups [7]–[9], and many companies such as Hansen Medical [10] and Mentice [11] are developing equipment in this field. The development of haptic interfaces in the cardiac surgery is an active area of research [5, 12]–[14] because of their immersive nature. One aspect of haptic feedack is the rendering of the contact forces between the instruments’ tip and the vessel walls, which provide important clues to cardiologist for safe and efficient intervention. A robot-assisted catheterization system is designed to allow the physicians feel the resistance on guidewire proximal-end during the remote operation [15]. A tele-robotic system focuses on avoiding the puncture of potential blood vessel due to excessive forces, via contact force rendering to the operator’s hand. A 6-DOF force–torque sensor is attached at the tool’s tip for feeding back force–torque vectors [16]. Meiß developed a guidewire with a miniature piezo-resistive sensor at the tip to distinguish between soft and hard surfaces of plaques [17]. Another work for navigating catheter inside heart using 3DOF robot was done in [18]; however, they require special electromagnetic equipment to estimate the position of distal shaft of the catheter as in [19]. The development of tiny sensor, as commercially available contact force catheters [20], its integration with the guidewire tip and carrying the signal out of patient’s body through microwires are crucial for interventional procedures. The difficulties and added expense of mounted sensors at guidewire/catheter tip limit the feasibility of above-mentioned approaches in real scenario. Moreover, the contact forces can be sensed only after the contact happened and some processing time lapse for sensor signals, which may damage the blood vessel before taking preventive action. The virtual fixtures (VFs), on the other hand, are computer generated artificial constraints that can help operators approaching a target, which have a potential to eliminate the need of sensorized surgical tools in many cases [21]. Many researchers exploit the effectiveness of haptic virtual fixtures (VFs) on various scenarios, not limited to medical applications such as cutting, palpation and cardiac interventions [22]–[26]. The studies related to VFs in cardiac surgeries require a 3D cardiac model from CT or MRI machine, which is expensive, and these scans are not taken during the whole time of surgery. The haptic using virtual fixtures was utilized by [27, 28], where author generated forbidden region virtual fixtures (FRVFs) with 2D digital optical camera, showing both the phantom blood vessel structures and the catheter tip simultaneously, inside the vessel. However, the approach is solely limited to prevent puncture of blood vessel, and not used for guidance purpose, as in [29]. Moreover, no user studies were presented showing the benefit of the above approach in terms of its effectiveness in real-time operation.

Fig. 1
figure 1

a C-arm X-ray machine for coronary intervention, (b) master–slave robotic system [9]

Full size image

This paper proposes a cardiac X-ray image-based haptic guidance method for robot-assisted coronary intervention [30]. Unlike most of the research, which require expensive tools, e.g., sensorized guidewire/catheter and/or time-consuming and expensive preoperative procedures, e.g., 3D modeling or 3D imaging, the proposed methodology can generate haptic guidance through torque, force or vibration display directly from the available two 2D cardiac X-ray images (angiographic and fluoroscopic images) obtained from the C-arm X-ray machine. During the PCI procedure, vasculature is not visible but only the guidewire, and hence, lot of practice and skillset is required to manually register angiogram with fluoroscopic images. Moreover, contrast enhancement medium is frequently fed to localize guidewire with respect to vasculature. Novelty of this work lies (a) in the utilization of guiding and forbidden VFs, (b) compatibility with conventional equipment in the operation theatre and (c) only one angiogram is required at the start of experiment and VFs are drawn over it. During the procedure, no contrast enhancement medium is required to be injected. Hence, VF-based haptic feedback simplifies and speeds up the procedure. The feasibility of the proposed method is validated through experimentation on phantom coronary artery model by comparing operator’s performance for different haptic modalities.

Methods

Proposed methodology is based upon information that is currently available from angiogram and real-time fluoroscopic images. Since proposed methodology does not require any force–torque or position sensor on the tools and is dependent on X-ray images for haptic feedback, there are some premises which need to be obeyed.

Basic premises/assumptions

The proposed haptic guidance method relies only on the two available X-ray images in 2D; therefore, following three assumptions are made: (i) Two X-ray images should be obtained from the same pose (position and orientation) of the C-arm and should provide optimal 2D view of both vasculature and guidewire. This can be easily performed in the current PCI by cardiologists (operators). (ii) Heart during the operation follows a regular heartbeat which means the waveform of the cardiac cycle does not include arrhythmia. (iii) The image rotation due to beating/breathing heart is not significant; therefore, the use of only one static angiographic image can be justified. With these assumptions, to show the proof of concept of the proposed method, the algorithm is developed with validation on phantom blood vessel model in the experiment section. However, heartbeat and breathing can be compensated with the registration of angiogram onto the real-time fluoroscopy image using the method explained in [31], where angiogram with one complete heartbeat cycle will allow to update VFs location in real time on X-ray images.

Proposed haptic guidance system

This paper presents a haptic guidance method in tele-robot cardio intervention by using only two readily available X-ray images: (1) angiogram for showing coronary vessels taken at the start of experiment and (2) fluoroscopic image showing the guidewire. The proposed haptic guidance requires to draw VFs on angiographic images either manually in the preoperative procedure or automatically, during real-time procedure. Haptic feedback is rendered on two degrees of freedom (2DOF—translational and rotational DOFs) master device. Following steps explain proposed haptic guidance system:

Step (i) Generation of VFs on angiogram.

The purpose of the VFs is to create virtual constraints in 2D angiographic images. This can help operator navigate the guidewire in the direction of the VF vectors into the target branches or, otherwise, prevent the guidewire from entering non-targeted branches. The VFs can be created either manually in the offline process or automatically in the online process. The automatic creation of VFs can be done by the following steps: (i) obtaining at least one regular cycle of heart beating from angiogram, which may also include patients breathing motion. (ii) Finding bifurcation points for each branch by a series of image processing techniques such as Frangi vesselness filtering [32], thresholding, contour detection [33] and skeletonization [34]. (iii) Drawing VF vector starting from the bifurcation point towards the targeted branch. (iv) Drawing a circle with a radius 5 mm having an origin at the bifurcation point being denoted as center of field circle.

A single set of VF composes of: (1) a line for guiding the direction of guidewire tip and (2) a circle, either GVF or FRVF, one at a time, further referred as field circle, for guiding the push/pull (translational motion) relative to the bifurcation point. Also, the two modes of VFs are proposed as: (i) The repulsive VFs (black color in Fig. 2) account for prevention of the guidewire from entering non-target branches and (ii) the attractive VFs (white color in Fig. 2) for pulling the guidewire into the target branch.

Fig. 2
figure 2

Virtual fixtures on angiogram

Full size image

Step (ii) Image processing on fluoroscopic images.

Once the VFs are drawn in the angiographic images, an algorithm will find both the guidewire’s tip position and tip bend angle (tangent at the guidewire tip) in the fluoroscopic X-ray images in real time. The fluoroscopic images, without the contrast medium, contains only the guidewire/catheter and the dense bone structures with background noise; therefore, the segmentation is comparatively easier. The detection of guidewire tip and tangent at the tip location is computed as follows: (i) segmentation of guidewire using Frangi vesselness filtering, and (ii) skeletonization is performed after noise removal followed by end of curve detector algorithm, (Fig. 3b).

Fig. 3
figure 3

Image processing results, a guidewire inside phantom model—analogous to angiogram, b guidewire segmentation and tip detection—zoomed image shows guidewire tip bend tangent

Full size image

Step (iii) Computation of tangent line at the guidewire tip.

The tangent at the guidewire tip must be computed as a pre-requisite for torque feedback in the case of twist motion guidance, as explained in step (iv). This tangent is computed by cropping a window of 63 pixels × 63 pixels centered at the guidewire tip and then performing quadratic curve fitting, shown in Fig. 3b.

Step iv) Haptic rendering.

To provide useful haptic feedback for manipulating the guidewire in the vasculature, we propose two-step haptic feedback algorithm:

  1. (i)

    While moving toward the target branch during intermittent push motion of the guidewire, the relative position of guidewire tip from the bifurcation point, approximately where field circle center is located, is continuously estimated. If the guidewire tip is outside the field circle, then there is no haptic feedback. If the guidewire tip reaches boundary of the field circle, i.e., near the branch entrance, then depending upon haptic feedback modality, a vibration–torque or force–torque feedback is rendered:

  1. a.

    Force feedback in case of targeted branch: Forces are rendered proportional to the distance of guidewire tip from the center of field circle. A push force is rendered to master device assisting user to locate guidewire tip at the bifurcation point (acting as a GVF). If the guidewire passes beyond the center of the field circle, a pull force is rendered to master device, assuring that guidewire tip stays at the bifurcation point (acting as a FRVF).

  2. b.

    Force feedback in case of non-targeted branch: The only difference from case (a) is that if the guidewire has entered the field circle, it will render the forces as to push the guidewire pass through the bifurcation point (acting as a GVF). If guidewire tries to enter in non-targeted branch, the VF will act as FRVF.

  3. c.

    Vibration feedback in case of targeted branch: Vibration feedback is modulated on frequency and amplitude. If the guidewire tip has just crossed field circle boundary, vibration of lower amplitude and frequency (2 mm and 30 Hz, respectively) is rendered. If the guidewire tip has crossed the bifurcation point, then a strong vibration of amplitude and frequency as 7 mm and 50 Hz, respectively, is rendered.

  4. d.

    Vibration feedback in case of non-targeted branch: If the guidewire tip has just crossed field circle boundary, vibration of lower amplitude and frequency (2 mm and 30 Hz, respectively) is rendered. If guidewire is about to enter in the non-targeted branch, a strong vibration of amplitude and frequency as 7 mm and 50 Hz, respectively, is rendered. No haptics is rendered if guidewire crosses bifurcation point.

  1. (ii)

    Once the guidewire tip is inside the field circle, operator may start adjusting the tip orientation through rotational DOF of master device clockwise (CW) or counterclockwise (CCW). A guidance torque or vibration is provided on the rotational DOF until the guidewire tip orientation is either maximum aligned with the VF vector (in the case of attractive VF) or maximum out of alignment (in the case of repulsive VF). Inner product is metric used to measure guidewire alignment/out of alignment with respect to VF. Torque magnitude is proportional to inner product, whereas force magnitude and vibration modulation are proportional to Euclidean distance between guidewire tip and center of field circle.

figure a

Experimental setup

The proposed method is validated on an experimental setup shown in Fig. 4. Figure 4a shows the computer software developed (C# with Qt library). In-house designed master device is shown in Fig. 4b, the slave robot in Fig. 4c and the phantom heart model in Fig. 4d. To validate our results in the laboratory, we used a web cam as image acquisition device and a phantom heart model as vasculature test bed. The master device in Fig. 4b, with 2DOF named translation and twist, is being controlled by master digital signal processor. It reads motor positions and displays haptics, i.e., synthetic forces and torques, through electrical DC motors, in response to various measurements. The translational link has stroke length of 54.6 mm with the home position at the middle of the total stroke length i.e., at 27.3 mm, allowing the operator to move the link (hence the guidewire) in either direction: forward (push) or backward (pull). This device is also equipped with a touch sensor working as clutch. While the operator is holding the master device, every movement of master device is mimicked by the slave robot. Upon releasing the clutch, the translational link comes back to its home position while during this time slave robot does not move. The twisting DOF is responsible to twist the guidewire in CW/CCW direction while trying to adjust its orientation according to the desired blood vessel entrance. The slave device in Fig. 4c is controlled by slave digital signal processor and maneuvers the guidewire in 3D with the 2DOF actuation. Master and slave robots are in position–position control scheme. Bad mechanical transmissions are naturally inherited in systems concerning guidewire manipulation using slave robot. In proposed methodology, VFs are drawn at bifurcation points, and if the guidewire is outside of field circle, distortions due to bad mechanical transmissions do not matter. However, inside the field circle, X-ray image-based haptics would assist to compensate for the distortions in the transmissions. Moreover, master device is friction compensated; hence, master-level movements with high fidelity are ensured at the slave robot. The phantom heart model in Fig. 4d is supported by two Falcon haptic devices to simulate heart beating, up-down motion which is orthogonal to the C-arm image plane. Since drawing of VFs can occlude the view of blood vessels; therefore, another image showing angiogram was used for drawing VFs during the preoperative step as shown in Fig. 4a. A static angiogram image, which shows the guidewire in real time and the target location, without the vasculature, is chosen as the reference image for the subject. Schematic of experimental setup is presented in Fig. 5.

Fig.4
figure 4

a Experimental setup, b Master device, c Slave device, d Phantom heart model setup

Full size image
Fig. 5
figure 5

Experimental setup schematics

Full size image

The experiment starts with the VFs being generated manually with a static angiographic image by a mouse click and drag from a start to an endpoint representing a VF vector. Figure 6a shows the overall haptic guidance steps based upon the inner product between the tangent vector at the guidewire tip and VF, for an attractive VF. In this example, initially force feedback is provided to bring guidewire tip at bifurcation point and then torque feedback is provided so as to maximize inner product. Figure 6b shows the progression of the guidewire tip toward the target location; Fig. 6b shows the white line directing the target branch with the red dot as the target location, whereas the dotted red line represents the tangent line at the tip of the guidewire which is initially outside the field circle (marked as a white circle).

Fig. 6
figure 6

a Haptic rendering algorithm, b algorithm helped to position guidewire in targeted branch

Full size image

Results

For the experiment, ten subjects (8 males, 2 females, age 22 – 27) participated. The 7/10 subjects were not familiar with the kinesthetic haptic feedback and the PCI manipulation. Therefore, as the training session they were given 30 min to have hands-on-experience of the given system by performing the task of positioning guidewire at the target location. Before the experiment, guidewire tip is appropriately bent in accordance with the phantom vasculature. The subjects were asked to navigate guidewire to five different target locations one by one as in Fig. 6, while their task completion time was recorded. At the start of experiment, subjects took the angiogram image which clearly shows the vasculature. Adjacent to the angiogram, a live streaming of guidewire was shown upon the static angiogram. Each session contained three trials: (1) without haptic feedback mode, (2) force–torque feedback mode and (3) vibration–torque feedback mode, in random order. Each subject was required to perform the task of reaching the target locations one by one, from branch 1 to branch 5.

The time was recorded for positioning the guidewire tip at each target location as shown in Fig. 7a. From the results, it is evident that branch 4 is the most difficult branch because of narrow opening and difficult entry angle, while branch 1 is easiest and branches 2, 3 and 5 are of intermediate difficulty levels. Without haptic feedback, subjects took more time to complete their task and showed more variability in task completion time as compared to haptic feedback. Figure 7b clearly shows significant decrease in task completion time, i.e., the time elapsed from location 0 to location 5, when haptic feedback (force–torque or vibration–torque) is provided to the operator. In case of no feedback, average task completion time is 241.7 ± 42.21 s. This time is reduced to 157.05 ± 28.12 s for force–torque feedback, whereas it is 152.47 ± 27.02 s (36.91% decrease) for vibration–torque feedback, which is quantitatively better than force–torque feedback. However, analysis of variance (ANOVA) shows that there is no significant difference between the force–torque feedback and vibration–torque feedback. In terms of qualitative measurements, 8/10 subjects voted for vibration–torque feedback is better and 1/10 voted for force–torque feedback is better. However, only one subject reported that no feedback is better commenting that in the current phantom setting, guidewire manipulation is easier without haptics.

Fig. 7
figure 7

Experimental result

Full size image

Conclusions and future works

In this work, a feasibility study of a novel haptic guidance procedure for the percutaneous coronary intervention is presented. The haptic feedback is generated by drawing the virtual fixtures using only 2D X-ray images in real time. The phantom experiments showed that the proposed haptic guidance for coronary intervention is helpful to improve the performance of the operators in terms of task completion time where an approximately 40%-time reduction is observed. The haptic rendering has been found most efficient for the most difficult branches, while, for the other branches, no substantial difference has been observed. The results show that the vibration feedback quantitatively provides efficient haptics and can replace force feedback for translational DOF. Also, the vibration actuator is comparatively lesser expensive; therefore, the cost of haptic setup can be minimized. As future works, image processing techniques will be implemented for automatically extracting the VFs on real-time X-ray images, only providing the target locations. Furthermore, two dynamic X-ray images in 2D will be combined for the animal experimentation in the operation room.