Keywords

1 Introduction

Augmented reality (AR) becomes popular in medical applications where minimally invasive surgery (MIS) is becoming common in practice [1, 2]. In MIS, video images of the surgical scene recorded by an endoscopic camera are presented to surgeons through a display monitor to guide their operation maneuvers. Although MIS has many benefits for patients, such as small wound size and shorter recovery time, it presents significant challenges to surgeons since the targets (e.g. tumor or vessels) are usually located behind the visible surface. AR techniques can be used to enhance the surgeon’s perception of 3D spatial relationship by combining the virtual structures and the endoscopic image in visualization. AR visualization can guide surgeons during the surgery, hence promising to reduce surgical time and increase surgical precision.

Augmented reality has been applied to kidney surgery [3–6]. An entire AR system usually includes tracking, registration, reconstruction and visualization. However, at present time, AR visualization techniques for MIS have not received sufficient attention in the previous publications, and have a lot of room for improvement. Depth perception, which means to obtain the correct spatial relationship between the virtual and the real structures, is the main issue in AR visualization. For more knowledge about medical AR visualization, please refer to [7, 8].

To the best of our knowledge, only [9, 10] explored different visualization techniques in medical AR. Therefore, medical AR visualization is worth further exploring and it is the focus of this paper.

General AR visualization can be classified into (1) the video-based method; (2) the see-through method; and (3) the projection-based method [8]. Considering the characteristics of MIS where only a small incision is available, the video-based method is more suitable than others. Although many visualization strategies are proposed in general video-based AR, transparent overlay is most widely used in medical AR [3, 4, 6] due to its simplicity. However, it easily results in a common problem that the virtual structure seems to be located above the real surface rather than below it [11]. Virtual window [12] is an alternative way to observe virtual structure occluded by the real surface. However, it partly covers the real surface which weakens the user’s perception of the real scene. Random-dot mask [13] is then proposed to achieve the minimal destruction of the real surface. This method creates a feeling of observing the virtual structure through many small holes on the real surface. The ghosting method [14] is another popular visualization strategy. It assigns different transparency for each pixel by analyzing the importance of the camera image. However, according to these publications, random-dot mask and the ghosting method are rarely applied to medical applications.

In this paper, we apply different visualization techniques (modes) to our medical AR system to evaluate their characteristics. The modes we developed and tested in our AR platform include: (1) transparent overlay; (2) virtual window; (3) random-dot mask; and (4) the ghosting method. We choose these modes because they are representative and commonly used in general AR. We also introduce the depth-aware ghosting method to further enhance perception of spatial relationship. Our AR system concentrates on the ablation of tumor inside the kidney. We conducted both simulated and in vivo experiments in our study. The visualization modes are presented, evaluated and compared with each other. In the next section, we will briefly introduce the components of our AR system and the implementation of each visualization mode.

2 Methods

In this section, we first introduce the components of our AR system and then the technical approach in each visualization mode.

2.1 Overview of Our AR System

In AR system, the precise position of invisible structures (e.g. tumor or vessels) is required for rendering. Our AR system consists of 4 parts: segmentation, reconstruction, registration and AR visualization, which are illustrated in Fig. 1(a).

In the segmentation part, a series of CT slices was acquired beforehand. The 3D Slicer toolkit [15] was used to obtain segmented surfaces of kidney and tumor, where tumor is the small object inside the kidney. In the reconstruction part, we use 5 captured endoscopic images and adopt the algorithm in [16] to get a reconstructed depth map and point cloud. The reconstructed point cloud of partial kidney is in the camera coordinate which is shown as the blue point cloud in Fig. 1(b). In the registration part, we use the iterative closest point (ICP) algorithm [17] to register CT segmented kidney model to the reconstructed one. After registration, the pose of the CT model with respect to the camera coordinate is identified, which is illustrated in Fig. 1(b). The image for AR visualization is rendered after the acquisition of this explicit spatial relationship.

Fig. 1.
figure 1

Overview of our AR system: (a) the components of our AR system; (b) an illustration of the spatial relationship for AR rendering

Full size image

2.2 Technical Approach of Each Visualization Mode

In this part, we will present the technical implementation of each visualization mode. Assume the original endoscopic image is denoted by O(x, y). The virtual object rendered from the point of view of the camera is V(x, y). The AR view is denoted as R(x, y).

Transparent Overlay. In this mode, the AR view is obtained through Eq. 1. The adjustable transparency parameter \(\alpha \) is a user-specified constant.

$$\begin{aligned} R=O\cdot \alpha +V\cdot (1-\alpha ) \end{aligned}$$
(1)

Virtual Window. In this mode, we need the reconstructed point cloud of the viewing object on which a window is created. This provides a feeling of getting a view of the inside of the object. A user should define a rectangular framework of the virtual window at first. In order to generate the sense of reality, the window wall is also created after the definition of the window normal. In our method, the window normal is defined by the average of the point normals in the specified rectangular area. A normal for a vertex \(x_1\) is calculated as the eigenvector of S with the smallest eigenvalue. S is the covariance matrix of all the neighboring vertices \(x_i(i=1,\ldots ,n)\) around \(x_1\) and is computed in Eq. 2. The wall and the background are decorated with some textures in order to further generate the sense of reality.

$$\begin{aligned} S= & {} YY^T\nonumber \\ Y= & {} (y_1,y_2,\ldots ,y_n)^T\nonumber \\ y_i= & {} x_i-m\nonumber \\ m= & {} \frac{1}{n}\sum _{i=1}^n x_i \end{aligned}$$
(2)

Random-Dot Mask. In this mode, a rectangular region is also required at first. Similar to the virtual window method, we create many small holes on the reconstructed point cloud. The size and the density of these holes can be adjusted according to the user’s preference.

The Ghosting Method. In this mode, a ghosting map is firstly generated. The ghosting map indicates which part of the endoscopic image is important. Then the value of transparency \(\alpha \) of each pixel is decided by the corresponding pixel in the ghosting map. The high value in the ghosting map assigns high value of \(\alpha \), which means that the corresponding part of the endoscopic image is important and should be less disturbed by the virtual structure. Then, the AR view is obtained through Eq. 3.

$$\begin{aligned} R=O\cdot \alpha (x,y)+V\cdot (1-\alpha (x,y)) \end{aligned}$$
(3)

The ghosting map in our system is obtained by analyzing the endoscopic image. Inspired by [14], the analysis includes edge detection, color differences and local contrast. Edges are seen as important parts of the original image. Color difference is computed in Eq. 4 from the global point of view, where \(\overline{O}\) is the mean color of the entire image. Local contrast is computed in Eq. 5. We define a region around a pixel (x, y). The pixels in the region are \(O_i(i=1,\ldots ,m)\), where m is the number of these pixels and \(\overline{O}_m\) is the mean color in this region. All the above color computation is calculated in the CIELAB color space, since it conforms with human color perception. The pixel is considered to be important in the camera image and assigned to 1 in the ghosting map if it satisfies one of the following conditions: (1) It belongs to the detected edges; (2) Its color difference is larger than a pre-defined threshold; (3) Its local contrast is larger than another threshold. The final ghosting map is a binary map. Figure 2 shows the process to generate the ghosting map.

$$\begin{aligned} Color(x,y)=\Vert O(x,y)-\overline{O}\Vert \end{aligned}$$
(4)
$$\begin{aligned} Contrast(x,y)=\sqrt{\frac{\sum _{i=1}^m (O_i-\overline{O}_m)^2}{m}} \end{aligned}$$
(5)
Fig. 2.
figure 2

The process of generating the ghosting map

Full size image

The Depth-Aware Ghosting Method. This mode is our improvement based on the original ghosting method [14]. In the ghosting method, the virtual objects can be correctly perceived to be behind the visible surface rather than floating over it. This is because the method considers the saliency of the endoscopic image. However, depth information between the virtual structure and the real surface is not presented in it. This information is important when the virtual structure has complex spatial geometry. Therefore, we integrate the distance between the virtual structure and the visible surface in the final AR view [18] after adopting the ghosting method. This process is illustrated in Fig. 3. First, a ray connecting a viewer and a vertex \(v_i\) of the virtual object is created. Then, a vertex \(o_i\) of the visible surface which is closest to the ray is picked. The distance is computed between \(v_i\) and \(o_i\) and it is used to modulate the transparency obtained in the ghosting method. In the above process, surface models of the virtual structure and the visible surface need to be acquired which is not difficult in our study. Both CT segmented surface and the reconstructed surface can be used.

Fig. 3.
figure 3

An illustration of computing the distance between the virtual structure and the visible surface

Full size image

In our study, the virtual objects can be divided into simple and complex categories. AR visualization of simple virtual structure like tumor can be achieved in transparent overlay, virtual window, random-dot mask and the ghosting method. Because some complex structures like vessels are usually wide spread, completely observing them through a small window is not possible. In these cases, transparent overlay and the ghosting method are more suitable. We also adopt the depth-aware ghosting method to them. In the next section, we will present our AR visualization results of both simulated and in vivo experiments. Further, we will discuss the pros and cons of each visualization mode.

Fig. 4.
figure 4

The operation scene for in vivo experiment: (a) the external scene of the operation. The guide device is used to control the orientation of the endoscope through magnet and the inflator pump is used to pump air into the pig’s abdomen to provide enough space for the movement of the endoscope; (b) the internal scene of the operation

Full size image

3 Experiments

In this section, two kinds of experimental environment were carried out in AR visualization. The first experiment used a simulated phantom where a 3D printed kidney model with some artificial textures was used. This model was used to explain our visualization modes described in the previous section. The second was a in vivo experiment with a real pig’s kidney. The real operation scene is shown in Fig. 4. The augmented tumor or vessels are illustrative and do not reflect the real situation. Our paper focuses on the effect of AR visualization rather than the entire AR system. In both experiments, only AR visualization is evaluated and the accuracy of segmentation, reconstruction and registration is not discussed.

AR visualization results of simulated experiment are shown in Fig. 5. The in vivo results are presented in Fig. 6. These results indicate that AR visualization can provide useful navigation and vital structures targeting during a surgery.

Fig. 5.
figure 5

AR visualization results of simulated experiment: (a)–(d) visualization of the tumor by transparent overlay, virtual window, random-dot mask and the ghosting method; (e)–(g) visualization of the vessels by transparent overlay, the ghosting method and the depth-aware ghosting method

Full size image
Fig. 6.
figure 6

AR visualization results of in vivo experiment: (a)–(d) Visualization of the tumor by transparent overlay, virtual window, random-dot mask and the ghosting method; (e)–(g) Visualization of the vessels by transparent overlay, the ghosting method and the depth-aware ghosting method

Full size image

Different visualization modes are analyzed from these presented results. We asked students from our lab for their opinions on the AR visualization modes. Transparent overlay does not consider occlusion cues and can easily result in false perception of spatial relationship [11]. As for visualization by virtual window and random-dot mask, correct occlusion relationship is provided. Virtual wall and small holes can be seen as visual cues to cause the sensation that the virtual structure is behind the real surface. However, random-dot mask causes a feeling of clutter and has a bad user experience. The ghosting method uses the important region of the endoscopic image to create the sense of the correct spatial relationship. However, this method has poor performance when the endoscopic image does not have sufficient features [13]. The depth-aware ghosting method has improved performance when the virtual structure has complex spatial geometry. Comparing its results with the ones in the ghosting method, the depth information of the furcations of the vessels are clearer in the depth-aware ghosting method. We can tell which furcations are in the front and which are in the back from the depth-aware ghosting method. We conclude the pros and cons of these AR visualization modes in Table 1. As all the visualization modes have shortcomings as well as advantages, we can combine all those modes into our AR system. Surgeons can choose AR visualization modes according to their personal preference.

Table 1. Comparison of different AR visualization modes
Full size table

4 Conclusions

In this article, we implement a whole AR system and different AR visualization modes. The AR visualization modes in our system are: Transparent overlay, virtual window, random-dot mask and the ghosting method. We also introduce the depth-aware ghosting method for improved perception of the depth information of the complex virtual structures.

These visualization modes are tested on both simulated and in vivo experiments. The outcome indicates the feasibility of different visualization modes. We also evaluated and compared these visualization modes. The pros and cons of each visualization mode are summarized in a table.

In our current AR visualization, tumor and vessels are presented separately. In the future, we plan to demonstrate the visualization technique to display tumor and vessels at the same time and consider strategies to reveal the relative relationship between them. The visualization evaluation in this paper is mainly based on qualitative results. A more quantitative assessment from experts will be given in the future. Furthermore, the AR system’s accuracy and stability will be evaluated thoroughly. Our ultimate goal is to apply our AR system and AR visualization techniques to real surgery operations.