Introduction

In contrast to traditional surgical operations, laparoscopy does not require the opening of the abdominal cavity. The laparoscopic instrument is inserted into the abdominal cavity through a 10 mm incision in the abdominal wall. Surgeons handle the instruments based on visual feedback from a monitor that shows the inside of the cavity [1]. The advantages of low trauma, shorter convalescence, and less pain to the patients have made laparoscopy a widely used surgical method. However, the restricted vision, limit of DOF, difficulty in hand–eye coordination can easily cause injuries to patient [2]. Therefore, it is necessary to train surgeons and improve their skill. At present, laparoscopic training is performed on animal tissue, cadaveric, and boxes; however, animal tissue is different from humans, cadaveric is expensive, and boxes can only train surgeons for simple skills [3].

Laparoscopic simulator based on virtual reality (VR) can replicate conditions that are accurate in terms of Human anatomy. Organs and tissues in the virtual environment exhibit physical and physiologic characteristics of real human organs and tissues through special algorithm. Trainees can practice different levels of operation repeatedly in the virtual system. This can lower the cost and improve the efficiency of training. So far, there have been many studies that prove the suitability of virtual-reality simulator for training [4–7].

At present, there are many studies that focus on the creation of a laparoscopic simulator. Some studies targeted the simulation of a special operation process. Basdogan et al. [8] made simulated training of laparoscopic surgery on the common bile duct. Heng et al. [2] focused on the development of a training system for knee arthroscopic surgery. Pan et al. [3] studied a virtual laparoscopic rectum surgery. Some studies focused on the basic skills involved in laparoscopic surgery [1, 9]. These systems provide different practice simulations for improvement of basic skills, such as hand-inverse eye coordination and grasp. Some studies focused on the key technology of a laparoscopic simulation, which is the simulation of the deformation of soft tissue [10, 11]. A key technology that improves collision detection in virtual surgery has also been introduced [12, 13].

There are many commercial virtual surgery simulators available in the market. dV-Trainer, developed by mimic, is a simulator that aims to train the use of the da Vinci Robot [14]. It provides validated orientation modules, skill modules, and a complete evaluation system [15]. Robotic Surgery Simulator is also a simulator that focuses on the da Vinci Robot [16]. It mainly provides modules for prostatectomy. The skill level of a trainee can be analyzed through the data collected when training, and the training process can be managed over the internet [17]. SimSurgey SEP was established in 1999 [18]. A series of virtual laparoscopic products by SEP realized the simulation of cholecystectomy, laparoscopy of ectopic pregnancy, oophorocystectomy, etc.

The Visible Chinese Human (VCH) database is developed based on the frozen section images of the human body with high resolution(0.1mmX0.1mmX0.1 mm) [19, 20]. After three-dimensional visual processing step, the complete information of the actual human body was retained. Meanwhile, the information of morpholigy, physical characteristics and physiological function have been constructed. Usually, it is used as the standard calculation model for radiative simulation. As opposed to the three-dimensional model that was manually drawn using medical image reconstruction and anatomical patterns, VCH model can provide more accurate anatomical structure with physiological and physical information which can be used for later physic simulation.

This paper describes the use of VCH to construct the virtual model. The whole technical roadmap is based on universal software development methodology for rapid development. Based on the modular tool that was made with the functions and features of instruments and organs, XML configuration was utilized to quickly create different scenes. We developed a laparoscopic basic skill training system and simulated the process of Cholecystectomy and laparoscopic removal hepatic cysts.

Materials and Methods

The technology roadmap of constructing a virtual surgery in this paper is shown in Fig. 1. In the process of virtually simulating a surgery, the first objective is to build 3D visualization models of instruments and organs. Organs have biological characteristics; this mainly refers to the characteristics of soft tissue deformation. Instruments have the functions such as clamping and switch. The corresponding functions and features are incorporated in the models using computer program. There are models that simulate the collisions that occur during organ–instrument, organ–organ, and instrument–instrument interactions. These models with corresponding functions constitute the data sources. The combinations of different models constitute different scenes. A user can observe a surgery scene through the TV monitor and interact with the scene by using interactive devices.

Fig. 1
figure 1

Technology roadmap of constructing a virtual surgery

Full size image

Geometric Model

Thesimulation of laparoscopic surgery requires the models of different organs within the abdominal cavity, such as the liver, stomach, and intestine. In addition, it requires the models of different surgical instruments necessary to perform an operation. The Mesh model is selected to represent the geometry of organ and instrument. Mesh models from VCH need to be preprocessed to meet the requirements of visualization and calculation. Preprocessing includes the processing of the surface geometry and surface texture. The processing of geometry includes the adjustment in the position of the points and the adjustment of the topology of the connection between these points. Data from VCH is in high resolution, wherein each of the models has more than 10,000 points and triangles. The amount of the mesh model points should be lower than 2000 to improve the efficiency of the system and improve real-time surgery simulation. Processing of surface mainly includes texture mapping, AO mapping, and normal mapping. Texture mapping can represent the color of the surface and the change in light and shade. Prior to attaching texture mapping on the surface, it is required to establish the relationship between surface and texture to obtain UV mapping. UV mapping carries information of the relationship between points and texture mapping. AO mapping can show the minute details of the low-resolution model. Normal mapping can add to the detailing without using more polygons [21].

Figure 2 shows the process of model preprocessing. Mesh models from VCH are not sealed; there are gaps and unclear connections between organs. Therefore, we used 3DS Max and ZBrush to seal the mesh by automatic topology and manually processing the gaps and connections. The resolution was reduced to obtain a low-resolution model. Various mappings were made using mature software for image and graphics processing (3DS Max, ZBrush, xNormal, Ps, Cinema4d). Figure 3 shows the part of the results of model preprocessing.

Fig. 2
figure 2

Flowchart for model preprocessing. The first step was to obtain low-resolution model, and the second step was the obtention of various mappings based on the low-resolution model

Full size image
Fig. 3
figure 3

Part of the results of model preprocessing. A lung. B spleen. C pancreas. D stomach. E gall. F liver

Full size image

The mesh models of instruments were manually made in accordance with real laparoscopic instruments. The results of making mesh models of instruments are shown in Fig. 4.

Fig. 4
figure 4

Mesh models of laparoscopic instruments. A Laparoscopic cameras. The top one is 30° and the below is 0°. B clip applicator. C Trocar. D 4 types of Laparoscopic dissecting forceps. E 3 types of Laparoscopic grasping forceps

Full size image

Data Sources and Modular Tool

The meshes after preprocessing need to be imported into the virtual scene. The format of the Unity3D resources is “.unity3d”. The meshes, mappings, coordinate information, etc., can be stored in .unity3d. Each model was processed separately. Subsequently, the meshes, mappings, information of coordinates, hierarchical data, animation data, and soft tissue deformation data were packaged.

The models used in the virtual operation scene comprise various organ and instrument models. These models should have the characteristic properties and functions of the organs or instruments after being loaded.

The models are basically rendered to display in the scene. The perspective of the user is captured by an inserted celiac laparoscopic camera; the user does not need to see the camera. Therefore, the camera does not need to be rendered and saves additional computational resources. The interaction between models is simulated through collision detection. Intra-abdominal organs have the characteristics of deformation. Instruments that are inserted into the abdominal cavity have 4-DOFs (up and down swing, left and right swing, spin, stretch out, and draw back). Some instrument handles can open and close while some instruments have switch devices (release the titanium clip, electric coagulation electricity). These properties and functions of the models were packaged in the modules to make the development of simulation easier, improve code reuse, and make the system easy-to-maintain. The models were classified based on the analysis of property and function (Table 1).

Table 1 Analysis of property and function
Full size table

It can be seen that different models are actually combinations of different function modules. Such as, the organ model has showable function (i.e., it can be rendered), collision detection, and deformable function. The clip applicator is a combination of showable, collision detection, 4-DOFs, forceps, and switch. Therefore, there are different functional modules that can be combined together to form the abstract model type (Table 2). BaseModel is the most basic model. It can control rendering to control Showable. CollisonModel is the most basic model that has collision detection. The next section, visual effects representation, describes two different methods to implement soft tissue deformation, mass-spring model, and morph target animation. Different methods of implementation of soft tissue deformation are possible for a different model. The “-Lp-” prefix indicates laparoscopic instruments. The addition and combination of new functions can extend the different types of surgical instruments model.

Table 2 Classification and combination of the model
Full size table

Visual Effects Representation

A soft tissue deformation model was developed to simulate the real physical properties of soft tissue and realize the interaction between the instruments and soft tissue. The process of simulating the soft tissue is mainly classified into two types, mass spring model and finite element analysis (FEA). When compared with the mass spring model, the calculation precision of FEA is higher, but FEA cannot meet the requirements of real-time simulation, given that the amount of calculations is larger. In order to meet the requirements of real-time simulation, calculation precision, and complexity, the mass spring model was chosen to be the main deformation algorithm. It was required to build a deformation physical model through preprocessing to give spring properties and physical parameters between points. The Poisson parameter and Young’s modulus were the preprocessing parameters. Preprocessing can reduce calculation of the initial surgery simulation. The deformation module was constructed based on the anisotropy of soft tissue deformation. Soft tissue deformation algorithm is shown in Fig. 5 and the result of deformation is shown in Fig. 6. Morph target animation was also used to simulate the simple and repeated deformation, such as the deformation of normal movement of gall [22].

Fig. 5
figure 5

Soft tissue deformation algorithm. Loading of the physical model during scene initialization. Deformation was simulated through real time updating of the mesh model by the dynamic analysis of the physical model

Full size image
Fig. 6
figure 6

Result of deformation. Soft tissue before (a) and after (b) interaction with instrument

Full size image

Smoke is generated when electric coagulation forceps are used. The smoke was simulated by the particle system of Unity3D. The position, velocity, size, color mapping, etc., of the particle were set to simulate smoke as shown in Fig. 7.

Fig. 7
figure 7

The cystic artery before (a) and after (b) was closed by electric coagulation forceps. (b) smoke simulation when coagulation forceps are used

Full size image

Interactive Device

The use of input devices can enable interaction with the virtual operation environment. A device with five degrees of freedom (5-DOFs) according to the characteristics of the laparoscopic surgery and the limits of the degrees of freedom is shown in Fig. 8a. Instruments can exhibit up and down swing, left and right swing, spin, stretch out, and draw back around the cannula inserted into the abdomen and close or open clamp in the process of surgery. We designed a single manipulator by separating each degree of freedom, as shown in Fig. 8b. Each degree of freedom simulates different operation of the instruments. Separation of each DOF can make it easy to add electrical equipment in the corresponding joint parts to get haptic feedback. Box has certain buttons that allows for the interaction with the software system of the upper computer. A foot pedal based on USB interface was used to simulate the pedals in the surgery.

Fig. 8
figure 8

Device with 5-DOFs (a) and Single manipulator (b)

Full size image

Experiments and Results

Implementation

A prototype system in accordance with the above design was implemented using Unity3D [23, 24]. A 3D visualization of the model was realized using the built-in rendering engine. The collision detection calculations were performed by the integrated PhysX. Smoke and blood simulation was created using the particle system, as described in the above section.

XML was used to realize scene initialization and configuration. The name, model type, and Showable characteristic of each model are described in XML. The properties and functions of each model were endowed after loading the descriptions in XML. The functions are endowed through different model types as shown in Table 2. For example, XML configuration file was created before establishing the cholecystectomy scene. Configuration file of cholecystectomy is shown below:

figure a

Loading the visual model depends on the name, and the assignment of corresponding function depends on the type of the model when initializing a given scene. The deformation of Gall bladder is realized by the mass-spring model and the morph target animation in accordance with the above XML description. Spleen can only be shown in this surgery, while the Stomach can only get the function of deformation by mass-spring model. Finally, a cholecystectomy scene was created based on the XML description. We quickly created different scenarios and designed a virtual laparoscopic training system.

As the prior study of the virtual surgery framework, the function of our system is encapsulated as separate modules. Through writing the structured XML document, the module would be organized exclusively. The feature modify and increase are more flexible and fast than which all code wroted in runtime. Simultaneously the user can benefit from the configuration file in building their own personalized training scenario. The virtual laparoscopic training system consists of two parts, basic skill training and surgery training mode.

Basic Skill Training System

In laparoscopic surgery, two of the most important basic skills are the ability to perceive and move and visual spatial orientation ability. We developed a basic skill training system where the user can practice some special skills. The user can practice the use of laparoscopic camera and handle instrument with wrist movements (Fig. 9a, b). The user is required to pick-up the needle using the grasping forceps and maneuver through the hole of an object and a circle around in the preliminary simulation of thread needle (Fig. 9c). The user should clamp the three balls form a plate and place them in another box (Fig. 9d). In addition, suturing and electric coagulation can also be practiced (Fig. 9e, f).

Fig. 9
figure 9

Basic skill training system. a, b Roaming system with laparoscopic camera. c Practicing how to thread the needle. d Practicing grasp. e Practicing how to use titanium clamp. F Practicing how to use electric coagulation forceps

Full size image

Simulation of Cholecystectomy

Guided and non-guided surgery practice is provided in the simulation of cholecystectomy. The user can practice the entire surgery process based on tips and instructions in the guided surgery practice manual. The user is free to practice non-guided surgery, wherein the user may operate or rectify errors that lead to surgical failures. Surgery flow chart is shown in Fig. 10.

Fig. 10
figure 10

Surgery flow chart of cholecystectomy

Full size image

The resources and configuration file should be loaded to initialize the scene in the beginning. The user selects different instruments in the instrument panel to complete the surgery with different steps.

First, the laparoscopic camera should be moved to operative site (Fig. 11a) in the beginning of guided surgery practice. Second, the user should select the laparoscopic forces from the instruments panel (Fig. 11b) and clamp the bottom of the gallbladder to a reasonable position through the prefabricated ball (Fig. 11c). The current step should be smoothly carried out before proceeding with the next step. The users are required to complete the steps of dissecting the triangle to obtain view of the operation site (Fig. 11d) release titanium clip to clip the cystic duct and the cystic artery (Fig. 11e), use electrocoagulation to cut the cystic duct and cystic artery (Fig. 11f, g), absorb bile and remove the gall bladder based on the instructions (Fig. 11h, i, j). Users are not required to complete a special operation before proceeding to the next step and in non-guided surgery practice.

Fig. 11
figure 11

The simulation process of guided cholecystectomy. a Operative site. b Moving a laparoscopic force to operative site. c Clamping bottom of gallbladder. d View of Operative site. e Releasing titanium clip. f Electrocoagulation cutting of the cystic artery. g Electrocoagulation cutting of the cystic duct. h, i, j absorbing bile and removal of the gall bladder based on the instructions

Full size image

Discussion

This paper describes a method that utilizes VCH to create a visualization model for the virtual surgery. The functions and features of the organs and instruments are analyzed, and a modular tool is designed based on the results of the analysis. By using the modular tool to modify the configuration file, it is feasible to quickly build different virtual surgery scenes such as the creation of virtual laparoscopic hepatic cysts surgery simulation. The Liver model is in the sick state for hepatic cysts. Combined with the operation video, the liver model is modified to make it pathological. The collapse of the liver is a simple process when draining fluid and can represented by a simple animation process. The liver can also be cut and collision deformation can be simulated. Therefore at this point, the model type of liver is MorphIssueModel. Meanwhile, it is required to include laparoscopic attractor in the scene. We can modify and add the models in the scene by modifying the cholecystectomy XML configuration files. The results of building hepatic cysts scene are shown in Fig. 12. Therefore, we can achieve the quick creation of different scenes and improve the efficiency of development (Fig. 13).

Fig. 12
figure 12

Virtual laparoscopic training system

Full size image
Fig. 13
figure 13

Drain fluid before and after in laparoscopic hepatic cysts

Full size image

The modular tool enables the system to be extended easily. By the addition and combination of new functions, it is feasible to extend the different types of surgical instruments used in the model. For example, some surgical instruments can lock clamp to ensure the forceps do not open or close. We can easily add the lock function and incorporate it into the previous forceps model to form a new model type.

By using the encapsulated functionality of Unity3D, such as 3D rendering, collision detection, and the script engine, it is feasible to simplify many of the key steps in the development of virtual surgery scene. This allows developers to pay more attention on the data model of virtual surgery and construction of soft tissue deformation. Most previous researches do not directly use the virtual reality development engine. Some use OpenGL [2, 3], while some use OpenInventor [8]. The complexity in development leads to unitary results of the developed functions. SINERGIA was developed with virtual reality engine WTK [9]. Its functions are more abundant, leading to a more perfect system. The combination of SINERGIA in this paper showed that using the game engine Unity3D to develop a virtual surgery system is feasible, scalable, and efficient.

Although, the quantitative description for verifying the effectiveness of whole system are still lacking. In the interative development process of the virtual laproscopic system, it has been placed in the laparoscopic clinical departments of two hospitals. There are approximately 3 experts (who had performed 100 or more human laparoscopic procedure independently) and 3 novices (who hadn’t performed any laparoscopic procedure independently) have used the system for practice and some descriptive but not quantitative advice has been gave. Here are two summary bellow.

First, the abdominal environment and cavity organs of the simulation system have very high authenticity, but lack of adipose tissue details. It is pointed out the different body types are significant differences in fat which would infuluence the procedure of the surgery. Second, the haptic force feedback gap between the actual operation has been narrowing in terms of accuracy, though it remains substantial.

Conclusion

We established a set of methods to quickly develop a virtual laparoscopic surgery scene. Mature image and graphics processing software were used to process the VCH data to obtain mesh models and mappings. The whole technical roadmap is based on universal software development methodology for rapid development. A set of modular tools were designed to develop different virtual laparoscopic scenes. Soft issue deformation was simulated by mass-spring model and morph target animation. Meanwhile, a device with 5-DOFs was made to interact with the system. In conclusion, we proposed a method that constructs modular virtual reality development tool based on game engine Unity3D and utilizes the configuration to quickly build virtual laparoscopic surgery scene for basic skill training system.