1 Introduction

Laparoscopic surgery is one of the modern minimally invasive surgery (MIS) techniques. Compared with conventional open surgery, patients with laparoscopic surgery usually have less pain and shorter recovery time [1]. However, it is difficult to master the skill for an interne. To provide a safe, repetitive and low-cost solution, virtual laparoscopic surgical training systems have been developed. At present, there are already a number of commercial VR-based laparoscopic surgery simulators, such as Lap Mentor [2] and LapSim [3] in the market. In clinic, cholecystectomy is one of the major surgeries, which can be performed by laparoscopic surgeons. The involved organs and tissues in cholecystectomy are shown in Fig. 1. The main procedures of cholecystectomy basically can be divided into four major steps: (1) dissect the fat tissue that wraps the cystic artery and duct; (2) clamp the cystic artery and duct to avoid massive bleeding; (3) cut the cystic duct and artery; and (4) dissect the connective tissue between the gallbladder and liver. The above four steps (Fig. 2) are the main training tasks for cholecystectomy simulation.

Fig. 1
figure 1

Organs and tissues involved in the cholecystectomy

Full size image
Fig. 2
figure 2

Typical cholecystectomy procedures [4]. a Dissect the fat tissue that wrap the cystic artery and duct; b clamp the cystic duct and artery; c cut the cystic duct and artery; d dissect the connective tissue

Full size image

In this paper, we focus on the first step (dissect the fat tissue which wraps the cystic artery and duct). In this procedure, doctor will insert long pencil-like surgical instruments into patient’s abdominal cavity, and instruments will collide with soft tissue. The first technical challenge is the deformation of soft tissue. Mass spring method is widely used in simulation tasks due to its efficiency and easy implementation [5,6,7]. But it is not an ideal choice for surgical simulations, which need high physical accuracy. Another popular approach is finite element method (FEM), which subdivides the deformable body into multi simpler elements, and the shape functions defined on these elements are assembled to approximate the constitutive deformation. FEM has been widely used in surgical simulation [8]. Although FEM can get accurate simulation result, it can be very challenge when modelling electrocautery in some complex environments due to the remesh of the model. Position-based dynamics (PBD) is another deformation method, which can provide a stable performance and high level of interaction controllability [9]. In recent years, it has been widely used in surgical simulation [10,11,12]. However, PBD cannot guarantee the physical accuracy, and this limits its application in surgical simulation which need high level of accuracy. All the above methods belong to mesh-based methods, and we need to maintain topology information during simulation. For the applications in which the mesh topology changes frequently, they are not the appropriate solutions.

Alternatively, we use meshfree method which is a physics-based deformation approach. Here, the continuum object is discretized into point samples and meshfree models do not need to maintain the topological information among data points. The problem associated with the complicated topology like geometry distortion or ill shape is avoided. These advantages make it suitable for the surgical simulation with frequently topology change. Electrosurgery is a surgical technique which dissects tissues by heat generated from high-frequency electrical current. The part of burnt tissue absorbs large quantities of heat and transfer into gas instantly. It is frequently used to dissect or remove the soft tissue, such as membrane, fat and fascia, surrounding the abdominal organs. So physics-based electrosurgical cautery simulation becomes one of the research hot spots in VR medical simulator. To ensure the dissection with physical meaning and couple it with meshfree deformation model, a meshfree dissection method based on heat transfer is built. Compared with mesh-based dissection methods, meshfree model avoids the instability problem or ill-shaped elements caused by remeshing and models the discontinuity [8].

In this paper, we propose two physics-centric meshfree models to simulate the deformation and dissection of soft tissue. The dissection procedure involves the thermal transmission and phase change from solid into gas. Finally, marching cube method is used to capture the modification of surface mesh. Each step is parallel implemented on GPU. These techniques are also applied in our developed VR laparoscopic surgery simulator for cholecystectomy training. A questionnaire is designed to evaluate the practical using experience of our simulator.

2 Related works

Our research focuses on meshfree deformation and physics-based electrosurgery dissection simulation. Here, we only list the most relevant works.

Pan et al. [12] present a metaball–meshfree hybrid model to handle the deformation and dissection of soft tissue. When scalpel moving across the spheres in the metaballs model, the dissected spheres will be removed and new spheres are generated from particles in each side of cutting surface. Müller et al. [13] proposed point-based animation for elastic and plastic deformation. Joldes et al. [14] introduce the meshless total Lagrangian adaptive dynamic relaxation method (MTLADR). In surgical simulation, Zou et al. [15] presented a particle-based local high-resolution model for deformation. In [16], a radial-based function point interpolation method (RPIM) was proposed to treat the biological aspects of soft tissue.

Dodde et al. [17] analysed temperature distribution in biological tissues subject to a bipolar electrosurgical procedure under FEM framework. A physics-based electrosurgery dissection approach was proposed by Lu et al. [18], which handle the topology change with low computational cost. Kim et al. [19] developed a carving mesh method which used in burning fat tissue, they employed the volume-based potential fields to capture shape modification, and hexahedral finite element method was employed to handle deformation of tissues. They further used this method in their cholecystectomy simulator [20]. Jin et al. presented a cutting algorithm to handle soft tissue dissection [21], and it was extended to 3D model in [22]. Cheng et al. [23] used the meshless method to model virtual cutting in surgery simulation by considering the deformation behaviour at different cutting stages, which made their work closer to reality. Berndt et al. [24] built the particle model driven by PBD, when the temperature reaches the threshold, and the particle starts to loss mass until it disappears.

3 Overview

We aim to offer a solution with physics-based interactive environment in surgical training. The deformation and dissection models are both built on particle clouds. The geometric model of soft tissue is first voxelized, and particles are assigned at each voxel centre. We use two types of particle clouds: one is for deformation (phyxels), and the other is for heat transfer (hexels). The phyxels sampling should be sparser than hexels sampling, since deformation process is more complex than heat transfer in our simulation. And a denser hexel cloud benefit detailed surface reconstruction. All the attributes, such as mass, density, location, temperature and heat, are carried by the physically simulated particles. Figure 3 presents an overview of the two particle clouds.

Fig. 3
figure 3

Overview of phyxels and hexels

Full size image

In each simulation loop, we first compute the strain and stress and derive the elastic force for each phyxel, which drive the deformation. For each hexel and vertex, the movement is computed by the interpolation of their neighbouring phyxels displacements (Fig. 4). Next, we check whether the electro-hook is close enough to those hexels which are responsible for transferring heat. If so, those hexels will receive heat from the hook in each time step, and their phase is governed by a latent heat model. Finally, due to the fact that hexels will be removed from the model when they absorb enough heat, this change should be reflected by the geometrical model and we dynamically construct the surface using marching cube (MC) method.

Fig. 4
figure 4

Movements of hexel and vertex are driven by the interpolation of neighbouring phyxels’ displacements

Full size image

Deformation, heat transfer and surface reconstruction are performed sequentially in our system. In each step, the computation is implemented parallel on GPU to guarantee the efficiency. In order to output stable haptic feedback, the feedback frequency must be at a consistent 1000 Hz rate or higher. Generally, haptic force computation is executed in a separate, high-priority thread. We use OpenHaptics Library to interact with Geomagic Touch device. Figure 5 shows an overview of the whole process.

Fig. 5
figure 5

Overview of the physical model and simulation of electrocautery procedure

Full size image

There are three major contributions in our work:

  • A GPU-based meshfree method is designed to handle the deformation of fat, which covers the gallbladder and needs to be removed by electrocautery;

  • A meshfree dissection model is proposed to simulate the electrocautery procedure on GPU. To improve the realism, it involves the phase change of fat tissue which may gasify from solid into gas, when temperature and heat reach the threshold;

  • A GPU-based marching cube method, which supports the volume texture synthesis, is presented for 3d surface reconstruction.

Before describing the detail of proposed approaches, we first briefly introduce our developed VR laparoscopic surgery simulator, which applies these techniques. The simulator hardware (Fig. 6a) consists of a keyboard, a monitor, a workstation with two haptic devices inside the simulator, a foot pedal and two laparoscopic instrument handles. Geomagic Touch is employed for the haptic feedback device, which could offer 6-DOF input data and 3-DOF force feedback. It can function as a cautery hook, a grasper or any other necessary surgical instruments in laparoscopic surgery. Figure 6b illustrates the snapshot of software interface for our simulator. It illustrates the anatomy of liver, gallbladder and other surrounding tissues. These organs are obtained from a standard 3d human anatomy database [25]. Referenced from the real cholecystectomy videos, animators manually modified these models to ensure it can be used in the surgery simulation.

Fig. 6
figure 6

Hardware and software interface of our developed VR simulator. a The hardware interface of VR simulator; b the software interface of VR simulator

Full size image

4 Meshfree method for the deformation of fat tissue

Due to the advantages of point-based system on simulating fat tissue removal, we design a meshfree method to build its physical model and handle the fat tissue deformation when it interacts with instruments. The simulation loop is illustrated as shown in Fig. 7.

Fig. 7
figure 7

Deformation loop

Full size image

This meshfree method is based on continuum mechanics, and we use the elastic equations proposed in [13]. In initialization step, we set the values of material parameters, such as Young’s modulus, Poisson’s ratio and material density. For each phyxel, we compute the average distance \( \bar{r}_{i} \) to its K-nearest neighbours. From experimental experience, K is usually chosen as 10 and neighbouring information can be stored in a Hash table. The support radius \( h_{i} \) is chosen to be a multiple of \( \bar{r}_{i} \). The mass is initialized as:

$$ \begin{array}{*{20}c} { m_{i} = s\bar{r}_{i}^{3} \rho ,} \\ \end{array} $$
(1)

where s is the scaling factor and ρ is the material density. The density at phyxel \( i \) can be computed as:

$$ \begin{array}{*{20}c} { \rho_{i} = \mathop \sum \limits_{j} m_{j} w_{ij} ,} \\ \end{array} $$
(2)

where \( w_{ij} \) is the weight between phyxel \( i \) and its neighbour \( j \) is computed using the following polynomial kernel:

$$ \begin{array}{*{20}c} {W\left( {r,h} \right) = \left\{ {\begin{array}{*{20}l} {\frac{315}{{64\pi h^{9} }}\left( {h^{2} - r^{2} } \right)^{3} ,} \hfill &\quad { r < h} \hfill \\ {0,} \hfill &\quad { r \ge h} \hfill \\ \end{array} } \right.,} \\ \end{array} $$
(3)

where \( r \) is the distance to the phyxel and \( h \) is the phyxel’s support radius. The volume is given by \( v_{i} = m_{i} /\rho_{i} \). Figure 8 illustrates our constructed meshfree model for the fat tissue.

Fig. 8
figure 8

Construction model for the gallbladder and fat tissue. a The exterior fat tissue model with texture; b exterior and interior mesh (two layers) of gallbladder and fat tissue; c particle clouds in the area of fat tissue, which contained both phyxels (brown points) and hexels (blue points)

Full size image

After initialization, the simulation loop begins. In each time step, we first check whether any external force is added to the deformation model. If instruments touch the soft tissue surface, the phyxels near touch point receive external touch force. Then, we compute displacement gradient, which can be approximated using the following moving least squares (MLS) method:

$$ \begin{array}{*{20}c} {\nabla U = \left[ {\begin{array}{*{20}c} {\begin{array}{*{20}c} {u_{x} } & {v_{x} } & {w_{x} } \\ \end{array} } \\ {\begin{array}{*{20}c} {u_{y} } & {v_{y} } & {w_{y} } \\ \end{array} } \\ {\begin{array}{*{20}c} {u_{z} } & {v_{z} } & {w_{z} } \\ \end{array} } \\ \end{array} } \right] = {\mathbf{A}}^{ - 1} \left( {\mathop \sum \limits_{{j}} \left( {{\mathbf{u}}_{{\mathbf{j}}} - {\mathbf{u}}_{{\mathbf{i}}} } \right)\left( {{\mathbf{x}}_{{{\mathbf{ij}}}} w_{ij} } \right)^{\text{T}} } \right), } \\ \end{array} $$
(4)

where \( \varvec{u}_{\varvec{i}} \) and \( \varvec{u}_{\varvec{j}} \) are displacement vector of phyxel \( i \) and its neighbour j. \( {\mathbf{x}}_{{{\mathbf{ij}}}} = {\mathbf{x}}_{{\mathbf{j}}} - {\mathbf{x}}_{{\mathbf{i}}} \), \( A = \mathop \sum \nolimits_{j} x_{ij} x_{ij}^{\text{T}} w_{ij} \). \( \nabla {\mathbf{U}} \) represents the spatial derivatives matrix. The Jacobian matrix can be computed as:

$$ \begin{array}{*{20}c} { J = I + \nabla {\mathbf{U}}^{\varvec{T}} = \left[ {\begin{array}{*{20}c} {\varvec{J}_{\varvec{u}}^{\varvec{T}} } \\ {\varvec{J}_{\varvec{v}}^{\varvec{T}} } \\ {\varvec{J}_{\varvec{w}}^{\varvec{T}} } \\ \end{array} } \right].} \\ \end{array} $$
(5)

To calculate strain, we chose the quadratic Green–Saint–Venant strain tensor: \( \varepsilon = {\mathbf{J}}^{{\mathbf{T}}} {\mathbf{J}} - {\mathbf{I}} \). And the stress \( \sigma \) can be computed by: \( \sigma = {\mathbf{D}}\varepsilon \cdot \sigma\) depends on the strain ε and material constant matrix \( {\mathbf{D}} \). We assume our model is an isotropic material that \( {\mathbf{D}} \) only depends on Poisson’s ratio and Young’s modulus. The force acting on phyxel \( i \) and its neighbours \( j \) can be computed by following equations:

$$ \begin{array}{*{20}c} {{\mathbf{f}}_{i} = - 2v_{i} {\mathbf{J}}_{{\mathbf{i}}} \sigma_{i} {\mathbf{d}}_{{\mathbf{i}}} = {\mathbf{F}}_{{\mathbf{e}}} {\mathbf{d}}_{{\mathbf{i}}} ,} \\ \end{array} $$
(6)
$$ \begin{array}{*{20}c} {{\mathbf{f}}_{j} = - 2v_{i} {\mathbf{J}}_{{\mathbf{i}}} \sigma_{i} {\mathbf{d}}_{{\mathbf{j}}} = {\mathbf{F}}_{{\mathbf{e}}} {\mathbf{d}}_{{\mathbf{j}}} ,} \\ \end{array} $$
(7)

where \( {\mathbf{d}}_{{\mathbf{i}}} = {\mathbf{A}}^{ - 1} \left( { - \mathop \sum \nolimits_{{\mathbf{j}}} {\mathbf{x}}_{{{\mathbf{ij}}}} w_{ij} } \right) \) and \( {\mathbf{d}}_{{\mathbf{j}}} = {\mathbf{A}}^{ - 1} \left( {{\mathbf{x}}_{{{\mathbf{ij}}}} {\text{w}}_{{ij}} } \right) \). We also consider the volume conserving force:

$$ \begin{array}{*{20}c} {{\mathbf{f}}_{{\mathbf{i}}} = - v_{i} k_{v} \left( {\left| {\mathbf{J}} \right| - 1} \right)\left[ {\begin{array}{*{20}c} {\left( {{\mathbf{J}}_{{\mathbf{v}}} \times {\mathbf{J}}_{{\mathbf{w}}} } \right)^{\text{T}} } \\ {\left( {{\mathbf{J}}_{{\mathbf{w}}} \times {\mathbf{J}}_{{\mathbf{u}}} } \right)^{\text{T}} } \\ {\left( {{\mathbf{J}}_{{\mathbf{u}}} \times {\mathbf{J}}_{{\mathbf{v}}} } \right)^{\text{T}} } \\ \end{array} } \right]{\mathbf{d}}_{{\mathbf{i}}} = {\mathbf{F}}_{{\mathbf{v}}} {\mathbf{d}}_{{\mathbf{i}}} ,} \\ \end{array} $$
(8)
$$ \begin{array}{*{20}c} {{\mathbf{f}}_{{\mathbf{j}}} = {\mathbf{F}}_{{\mathbf{v}}} {\mathbf{d}}_{{\mathbf{j}}} .} \\ \end{array} $$
(9)

The final force can be represented as

$$ \begin{array}{*{20}c} { {\mathbf{f}}_{{\mathbf{i}}} = \left( {{\mathbf{F}}_{{\mathbf{e}}} + {\mathbf{F}}_{{\mathbf{v}}} } \right){\mathbf{A}}^{ - 1} \left( { - \mathop \sum \limits_{{\mathbf{j}}} {\mathbf{x}}_{{{\mathbf{ij}}}} w_{ij} } \right),} \\ \end{array} $$
(10)
$$ \begin{array}{*{20}c} {{\mathbf{f}}_{{\mathbf{j}}} = \left( {{\mathbf{F}}_{{\mathbf{e}}} + {\mathbf{F}}_{{\mathbf{v}}} } \right){\mathbf{A}}^{ - 1} \left( {{\mathbf{x}}_{{{\mathbf{ij}}}} w_{ij} } \right).} \\ \end{array} $$
(11)

Finally, a leap–frog integration scheme is employed to update the position and velocity:

$$ \begin{array}{*{20}c} {{\mathbf{u}}_{{\mathbf{i}}} = 2{\mathbf{u}}_{{\mathbf{i}}} - {\mathbf{u}}_{{{\mathbf{i}}_{{{\mathbf{old}}}} }} + \left( {\frac{{{\mathbf{F}}_{{{\mathbf{total}}}} }}{{m_{i} }}} \right)\Delta t^{2} ,} \\ \end{array} $$
(12)
$$ \begin{array}{*{20}c} { {\mathbf{v}}_{{\mathbf{i}}} = \frac{{{\mathbf{u}}_{{\mathbf{i}}} - {\mathbf{u}}_{{{\mathbf{i}}_{{{\mathbf{old}}}} }} }}{{\Delta \varvec{t}}}. } \\ \end{array} $$
(13)

Though the deformation has a realistic physical performance, it suffers the high computation cost. To achieve the real-time visual and force feedback in surgery simulation, the simulation loop is implemented on GPU (Sect. 7).

5 Meshfree dissection model for electrocautery simulation

Generally, the electrocautery chooses adjustable current to heat the soft tissue directly. It can be used to disconnect and dissect the soft tissue in laparoscopic surgery. In our work, we introduce a particle-based model to simulate the thermal transmission of soft tissue contacted with electro-hook. To simulate the smoke generation, the gasifying process is also considered. Figure 9 illustrates the pipeline of this thermal transmission model.

Fig. 9
figure 9

Pipeline of the thermal transmission

Full size image

Similar to deformation, the geometry model is voxelized at first and sample particles at each voxel’s centre. Several thermal attributes (such as temperature, heat and density) are assigned to each hexel particle. For initialization, we follow the steps in meshfree deformation in the previous section. Firstly, for each hexel \( {\text{i}} \), search its \( {\text{K}} \)-nearest neighbours and calculate the average distance \( r_{i} \) to them. Then, compute the support radius and estimate the mass and density by using Eqs. (1) and (2).

The thermal transmission model references the research from Iwasaki et al. [26]. As shown in Fig. 10, in each time step, the temperatures of hexels are computed by three heat transfer processes: (1) heat transfer among neighbouring hexels, (2) heat transfer from the surrounding tissue and (3) heat from the electro-hook. The heat from neighbouring hexels of i is calculated by:

$$ \begin{array}{*{20}c} {\Delta Q_{\mathrm{ne}} = c_{\mathrm{d}} \mathop \sum \limits_{j} m_{j} \frac{{Q_{j} - Q_{i} }}{{\rho_{j} }}\nabla^{2} W\left( {r_{ij} ,h_{i} } \right),} \\ \end{array} $$
(14)

where \( Q_{i } \) is the heat of hexel \( i \), t is the time, \( {{c}}_{\mathrm{d}} \) is the thermal diffusion constant, \( r_{ij} \) is the distance between hexel \( i \) and its neighbour \( j \), \( h_{i} \) is the support radius and \( W \) is a smoothing kernel as proposed in [27].

Fig. 10
figure 10

Particle-based latent heat model. a Three heat transfer process, b temperature diffusion

Full size image
$$ \begin{array}{*{20}c} {W\left( {r,h} \right) = \frac{15}{{2\pi h^{3} }}\left\{ {\begin{array}{*{20}l} { - \frac{{r^{3} }}{{2h^{3} }} + \frac{{r^{2} }}{{h^{2} }} + \frac{h}{2r} - 1, } \hfill & {0 \le r \le h} \hfill \\ {0, } \hfill & { {\text{otherwise}}} \hfill \\ \end{array} } \right..} \\ \end{array} $$
(15)

For the heat from surrounding tissue, we approximate the temperature of surrounding tissue as a constant \( T_{\mathrm{s}} \). The heat transfer is calculated as \( \Delta Q_{\mathrm{s}} = p\left( {T_{\mathrm{s}} - T_{i} } \right)\delta A \), where \( p \) is the thermal conductivity and \( \delta A \) represents the area of contacted region with surrounding tissues, and it can be approximated as \( \delta A = \frac{{6 - n_{i} }}{6}A \), where \( A \) represents the sum of area for six faces of the voxel, which the hexel generated from \( n_{i} \) indicates the number of neighbouring hexels.

The heat generated by electro-hook is calculated as \( \Delta Q_{\mathrm{tool}} = J \cdot E \) [10], where \( J \) is the surface current density and \( E \) is the electric field intensity. Both can be derived from Laplace equation:

$$ {\nabla \cdot \left( {\omega \nabla V} \right) = 0,} $$
(16)

where \( V \) is the electric potential and \( \omega \) is the electrical conductivity. The electrical conductivity is considered to be the constant in our simulation. The Laplacian equation can be solved efficiently and independently.

Finally, the increase in the temperature \( \Delta T \) is calculated from \( \Delta T = \frac{{\Delta Q_{\mathrm{ne}} +\Delta Q_{s} +\Delta Q_{\mathrm{tool}}}}{C \cdot m} \), where \( C \) is the specific heat capacity and \( m \) is the mass of hexel. After the calculation of heat transfer, the phase change of each hexel is determined. We borrow the idea of latent heat from Gao et al. [28]. If the temperature of a hexel exceeds the gasify threshold, it will enter a stage which still absorbs heat but temperature stop rising until the supplied heat exceeds the latent heat threshold, and then, the hexel changes its status into a gas particle and will be deleted from the solid model. Once all the neighbouring hexels of a phyxel are gasified and removed, this phyxel will be set as an invalid particle. And this phyxel will not be used for the strain and stress computation. The motion of hexels is driven by the phyxels nearby. In initialization stage, each hexel searches phyxels in its support radius. When the computation of phyxel deformation finished, the displacements of hexels are calculated by the interpolation of the displacements from its phyxel neighbours, as shown in Fig. 4. Our algorithm executes all these processes on GPU (Sect. 7).

6 Reconstruction of surface mesh

During the electrocautery simulation, hexels may be burnt away frequently. This modification reflects on the topology change of surface mesh. Instead of processing a static model, we construct the fat surface based on the particles dynamically. Hexels are not only responsible for heat transfer but also help with the reconstruction. When hexels near the electro-hook are burnt away, the changes should be captured on the surface in that region. We use a marching cube (MC) method to implement this process. First, we build a volume field for MC grid points; for each grid point located at \( p_{i} \), we find all hexels whose distance to \( p_{i} \) is smaller than the smooth radius given by user and compute the weight via following polynomial kernel \( W_{\mathrm{mc}} \).

$$ \begin{array}{*{20}c} {W_{\mathrm{mc}} \left( {r,R} \right) = \left( {\frac{{1 - r^{2} }}{{R^{2} }}} \right)^{3} ,} \\ \end{array} $$
(17)

where r is the distance between the hexel and grid point and R is the smooth radius. Then, the modified position \( \hat{p}_{i} \) of grid point is calculated as

$$ \begin{array}{*{20}c} { \hat{p}_{i} = \frac{1}{{W_{\mathrm{sum}} }}\mathop \sum \limits_{j} x_{j} w_{ij} ,} \\ \end{array} $$
(18)

where \( x_{j} \) is the location of the adjacent hexel \( j \), \( w_{ij} = W_{\mathrm{mc}} \left( {\left| {x_{j} - p_{i} } \right|, R} \right) \) and \( W_{\mathrm{sum}} \) is the sum of weights. The volume of the grid point is computed as

$$ v_{i} = \left| {p_{i} - \hat{p}_{i} } \right| - \frac{1}{2}L_{\mathrm{aver}}, $$
(19)

where \( L_{\mathrm{aver}} \) is the average spacing among hexels. The mesh is represented by iso-surfaces constructed from the volume field. The more hexels include in the neighbour search, the more details will be preserved in constructed surface. When a hexel meets the condition of gasified and has been removed from the solid model, the surface needs to be reconstructed. In order to satisfy the frame rate of interaction, only the local region near the deleted hexels will be considered. And for further acceleration, the MC process is also carried out on GPU (Sect. 7). Another challenging issue is the texture setting for the newly generated vertices. Instead of using the traditional texture mapping, we employ a tri-planar texture synthesis method in [29]. The reconstructed surface is shown in Fig. 11. The blue point in Fig. 11a represents hexels. The colours indicate its temperature, and the lighter colour reflects the higher temperature. The light blue points are those points near the burnt region. Figure 11b shows the reconstructed surface mesh with texture of this region.

Fig. 11
figure 11

Reconstructed fat tissue surface after electrocautery. a Particle clouds contained both phyxels (white points) and hexels (blue points); b surface mesh with texture

Full size image

Similar to hexels, the motion of surface vertex is also driven by neighbouring phyxels. Because all vertices are located in MC grids, we search the phyxel neighbours for each MC grid point and store them in a Hash table in initialization. During the deformation, we first compute the grid index and hashing the neighbouring phyxel for each vertex. The displacements of vertices are calculated from the interpolation of the displacements for all its neighbouring phyxels, as shown in Fig. 4.

7 GPU implementation

To guarantee a real-time simulation, our proposed methods, including meshfree based deformation, particle-based thermal transmission and surface reconstruction, are fully implemented on GPU with CUDA. For deformation, in initialization step, we search adjacent phyxels for each phyxel, hexel and MC grid point and store them in hash tables. Then, we calculate physical values which are fixed through the whole processes, such as mass, density and the intermediate matrix for computation of body force. All information is stored in a structure of arrays (SOA) and copied into global memories on GPU. In each deformation loop, the calculation of strain, stress and elastic force is performed in parallel by launching each CUDA thread for each phyxel (Algorithm 1). The haptic force is introduced as external force for selected phyxels. Finally, we perform leap-frog integration to update phyxels’ positions and velocities. The hexels and vertices displacements are updated by interpolating the displacements of near phyxels. When deformation completes, we launch each CUDA thread for each hexel to calculate the heat transfer from neighbour hexels and other tissues. Then, if the electro-hook contacts the surface, the hexels located inside the diffusion radius need to calculate the heat from hook. We launch a thread for each of them (Algorithm 2).

figure f
figure g

All MC grids will initialize the volume field at initialization stage followed by the extraction of iso-surface. The mesh only updates when the hexels near hook are burnt away. If there are hexels meet the condition of deleting, we first search the corresponding MC grids and 27 neighbouring grids. The sub-grids need to update the volume field, and new surface will be reconstructed from these grids. When hexels are deleted, the adjacent grids will update its hexels hash table. This operation will cost much time. To handle this problem, we use a uniform grid to accelerate the search process [30]. The three main stages are performed fully on the GPU, and the programme can achieve a satisfying result in computation efficiency.

8 Results, discussion and evaluation

The proposed model has been implemented in a prototyped VR-based laparoscopic simulator. The system is written with C ++ and OpenGL, CUDA is used for the GPU acceleration, and an OpenHaptics lib package is employed to provide haptic device API. Our system is running on a desktop with NVIDIA GeForce GTX 1050Ti, Intel(R) Core(TM)i7-7700 HQ CPU (2.8 GHz, 8 cores) and 8 G RAM. The device update rate is 1600 Hz, and the graphics refreshing rate is about 55 Hz in average.

The simulation results of our method are illustrated in Fig. 12. In the snapshots, the fat tissue can be successively removed and the topology change of the surface mesh is handled naturally. We also implement smoking effect to make the scene more realistic.

Fig. 12
figure 12

Screenshots of the electrocautery of gallbladder fat tissue in cholecystectomy using our method

Full size image

To evaluate the efficiency and visual performance of our simulation, we design four experiments. Kim et al. proposed a volumetric model-based approach to dissect fat for cholecystectomy [20]. The first experiment is to compare the visual performance of our method with them. The result is shown in Fig. 13. It can be found that our method presents more naturally burnt region, which preserves more details.

Fig. 13
figure 13

Visual performance comparison, the top is the volumetric model-based approach [20], the bottom is our method

Full size image

We also compare our method with the current commercial simulator product: LapSim [3] (surgical science, http://www.lapsim.com/), which simply modifies the geometric model of fat and does not consider the physical process of heat transfer. The method proposed in this paper demonstrates the physical meaning to simulation of three heat sources and the heat transfer between particles. We believe that our method is more accurate in physical process (Fig. 14).

Fig. 14
figure 14

Visual performance comparison with commercial surgical simulator, the top is the result from LapSim (surgical science, http://www.lapsim.com/), the bottom is our result

Full size image

It is an important issue to achieve an appropriate result in both visual realism and interactive rate. How to make a trade-off between them is a tricky problem. The particle density needs to be chosen carefully. In general, the application with graphics refresh rate higher than 30 Hz and haptic feedback frequency higher than 1000 Hz can be regarded as a real-time display. We compared the deformation time consumption under different phyxel densities in Table 1. Paper [15] also uses meshfree method to handle the soft tissue deformation, and we compare with them to demonstrate the acceleration with the help of GPU. From the experimental results, we believe that 3041 phyxel is a good selection for fat tissue modelling.

Table 1 Comparison of deformation time cost between our method and the approach in [15]
Full size table

The fourth experiment test is the reconstruction result in different hexel densities after burning the same amount of hexels away. More details of the surface will be captured when the density increases, but the graphics refresh rate will drop down at the same time. To guarantee the real-time performance, we choose the density in Fig. 15c (147,233 hexels) in our simulation.

Fig. 15
figure 15

Surface reconstruction result after burning 100 hexels away under different hexel densities. a 3041 hexels; b 20,291 hexels; c 147,233 hexels; d 1,178,091 hexels

Full size image

We also test the time consumption in each simulation step under different hexel densities. The total physical update rate is about 40 Hz. Haptic computation is governed by OpenHaptics in a separate thread, and it is about 1000 Hz. Table 2 presents the results.

Table 2 Computation time of our method with different hexel densities
Full size table

To evaluate the accuracy of biomechanical force in simulation, quantitative verification is necessary. However, it is difficult to find a well-established method to handle such verification in the context of fat tissue dissection. Here we design a straightforward but efficient way to implement it. With the surgeons help, we make statistics of the force used in fat tissue electrocautery in real cholecystectomy. It is approximately from 0 N to 8 N. This result fits the haptic force measured from the simulation system (Fig. 15). In Fig. 15, ascending curve represents the force when user presses the fat tissue with hook in electrocautery. Recession curve represents the force when user releases the hook (Fig. 16).

Fig. 16
figure 16

Haptic force measured when the hook contacts the fat tissue during electrocautery simulation

Full size image

Finally, we also invited six subjects with laparoscopic surgery experiences to evaluate our simulator. The subjects were asked to conduct cholecystectomy with electrocautery procedure tasks. Then, they will answer a questionnaire consisting of 12 questions: (1) realism of the anatomy, (2) realism in identification of the Calot’s triangle, (3) realism of the appearance of the simulator interface, (4) realism of the instrument handling, (5) realism of the electric dissection, (6) realism of the fat tissue removal, (7) realism compared to the “training box”, (8) the quality of the force feedback, (9) usefulness of the force feedback in training, (10) usefulness in learning the hand–eye coordination, (11) usefulness in learning ambidexterity skills, (12) usefulness in learning the fundamental skills. Each question can be answered on a 5-point Likert scale with 1 for very poor and 5 for very good satisfactory. The results from the questionnaire study are given in Table 3.

Table 3 Descriptive statistics obtained from the questionnaire study
Full size table

The result from the pilot study shows the average scores for all the questions (Table 3). It also indicates which aspects need to be improved. In Table 3, the lowest score (2.9) is “the quality of the force feedback”. Currently, we use the cheapest simplest hard device (Geomagic Touch) and employ the same computation model (Hook’s law) to output the force for the organs and surrounding tissues (e.g. membranes). In future, we will use different physical models to calculate the force for the fat and membrane tissues. And we can also set more accurate physical parameters (e.g. elastic coefficient) through vivo measurement in the animal tests. Also the SD about “realism of the electric dissection” and “realism of the fat tissue removal” is higher than 1, and we need to do more effort to improve in the future.

9 Conclusion and future work

We have developed a laparoscopic cholecystectomy simulator, the techniques used in this system include (1) meshfree deformation, (2) particle-based heat transfer and phase change, (3) surface reconstruction with Marching Cube. These approaches can simulate deformation of soft tissue and electro-hook burning operation with high realism. The generated mesh is smooth in burnt area. The efficiency of the whole process meets the qualification of user interaction. The proposed method is successfully applied to fat tissue removal to identify the cystic duct and artery procedure in laparoscopic cholecystectomy. We believe that the proposed technique can also be applied to other type of surgeries, which involves electro-hook operation.

Despite the successful implementation of simulator, it also has a few limitations. The first is the force feedback computation in haptic rendering. At present, we use a same simple computation model to output the force for the membrane, fat, organs and other issues with different material properties. In future, we can use different methods to compute the haptic force. And verification in computational biomechanics can also be done for a more physical accurate performance by tests [31, 32]. The visual performance can also be improved, like adding bleeding effect and flushing operation.