Over the last 10 years, personal computers (PCs) witnessed a very important expansion. Thanks to this evolution, Virtual Reality concepts essentially used for games can today be applied on low-cost computers. In parallel to these technological evolutions, since the end of the twentieth century, the medical world has seen a revolution based on concepts that led to several Nobel Prizes in Physics or Medicine—three-dimensional (3D) medical imaging. By translating the information contained in medical images into a set of 3D models, it is possible to obtain a preoperative 3D model of the patient, a kind of digital clone of the real patient. By combining technologies, a new form of medical education has become possible through the development of computerized patient-specific surgical simulators derived from 3D models. However, several limitations restrain the application of such simulators. In this article we present the next steps in the educative evolution that will lead to patient-specific surgical simulation integrating a web dimension.

From medical image to virtual patient

Modern medical imaging provides essential preoperative knowledge of patient anatomy and pathologies. However, the patient is represented on a set of two-dimensional (2D) images (magnetic resonance imaging [MRI] or computed tomography [CT] scan), and their radiologic interpretation often remains a difficult task. Virtual Reality can be used to facilitate the interpretation of such studies. Applied to medicine, it allows an easier and more extensive visualization and exploitation of medical images. Such an improvement is essentially based on a 3D visualization of patient anatomy from the 2D medical images. The standard 3D visualization method occurs through direct volumetric rendering [1]. This technique, available on all current MRI and CT imaging systems, can be sufficient for a good 3D visualization of anatomical and pathological structures (Fig. 1). It consists in replacing the standard slice view by the visualization of all slices together in 3D. To see internal structures, the initial gray level is replaced by an associated color and transparency. This transparency gives the observer the impression of seeing delineated organs which are not delineated in reality. Indeed, with this technique organ and pathology volume are not available, and it is not possible to perform a virtual resection of an organ without cutting the entire surrounding structure. To overcome these limitations, each anatomical and pathological structure in the medical image has to be delineated. From the delineated structure a 3D mesh can be created and exported for viewing in conventional 3D formats such as X3D (Extensible 3D) or VRML (Virtual Reality Markup Language). With standard software, such delineation is a long and difficult manual task for radiologists manipulating the data. Therefore, several research teams have developed 3D patient modeling software programs that automatically or interactively provide the main visible organs from the CT scan or MRI [2–5]. We have developed several 3D patient modeling software programs that automatically or interactively provide the main organs of the digestive system that can be visualized from CT or MRI scans [6, 7]. Our latest software, 3D Virtual Patient Modeling (3D VPM © IRCAD 2006), also makes it possible to perform the 3D modeling of the thoracic or pelvic area, thanks to improved management of medical knowledge combined with an interactive process [8] (Fig. 2). The main interest of such modelling is to obtain a surface rendering working on all current low-cost computers using an open GL graphics card.

Fig. 1
figure 1

The Haptic Operative Realistic Ultrasonographic Simulator (HORUS) allows a preoperative patient-specific simulation of thermal ablation, the burning effect being also simulated in the virtual ultrasonographic image (right)

Full size image
Fig. 2
figure 2

From patient computed tomography (CT) scans, samples of volume rendering (left) and 3D modeling of thorax (center) and abdomen (right) performed with three-dimensional Virtual Patient Modeling (3D VPM, © IRCAD 2006)

Full size image

Several companies have proposed such 3D modeling, essentially for dental pathologies but more recently for the digestive system, as a remote service online by delivering a DICOM image (MeVis Distant Services AG, PolyDimensions GmbH). These services usually include Web-based order forms with integrated data transfer. In the same way, since 2002 we have set up several free cooperative agreements with distant university hospitals (Geneva, Montreal, and Strasbourg) in order to offer a similar experimental 3D modeling service named MEDIC@. A total of 558 clinical cases have thus been modeled in 3D for the thoracoabdominal area (Fig. 2), 60% being reconstructed for liver pathologies. As the MeVis Distant Services, this distant 3D modeling service showed its efficacy in providing an accurate 3D model of the patient anatomy and pathology with a short delay (between 1 and 3 days). Such Web-based services could represent the first step of any future patient-specific Web-based surgical simulation.

From virtual patient to patient-specific surgical simulator

Existing 3D delineation software platforms are usually linked to surgical planning tools developed by the same teams [9–12]. Surgical planning makes it possible to define the best operative strategy, but it does not allow evaluation of a surgeon’s ability to perform that strategy. This is the objective of realistic surgical simulators. To be efficient, simulators have to reproduce sensations linked to operative gestures: realistic visual perception and realistic haptic rendering through force feedback. Force feedback is an important issue for any realistic surgical simulator. There are today a large range of haptic systems enabling reproduction of free-hand movement of instruments such as a scalpel or needle (essentially Phantoms from SensAble), or constrained movement instruments such as laparoscopic tools (Laparoscopic Impulse Engine from Immersion, Instrument Haptic Port from Xitact, Force Feedback from Karl Storz [13]). There are large numbers of commercial surgical simulators for abdominal surgery, urology, gynecology, and arthroscopy procedures (available through Surgical Science, SimbionixTM, SimSurgery, Haptica, or Karl Storz). Various scenarios are available in separate modules. The increasing realism of visual rendering, owing to the use of textures obtained from real images, and the progress in force feedback mechanisms has enabled those products to acquire some maturity. Nevertheless, although they are attractive, these simulators are limited to simulation of restricted and determined virtual models within a set database.

The main idea of a patient-specific simulator is to offer surgeons the opportunity to carry out preclinical training on a virtual copy of the patient. Only one company (SimbionixTM in partnership with PolyDimensions GmbH) proposes such a preoperative simulation limited to endoscopic arterial pathologies. Few works [14–19] also propose a preoperative patient-specific simulation, essentially of endoscopic procedures [14, 15] or orthopedic surgery [16–18], and sometimes on other, more complex procedures [19]. Our objective is to realize and validate a highly realistic simulator of liver procedures, including realistic physical and visual modeling of organs, real-time force feedback, and the opportunity to realize large resections on any region of the reconstructed patient.

The first patient-specific simulator was developed for laparoscopic liver surgery [20, 21]. It was limited to replicating volumetric resection gestures executed by applying an electric bistoury surgical cutting device on a 3D liver and its internal vascular systems modeled from a patient CT scan. This simulator remains the only system that allows a modification of the topology of a patient-specific simulated organ after a volumetric resection. However, use of such a simulation remains limited because surgeons need a more realistic environment in order to practice surgery preoperatively. In such a case, a complete abdominal cavity would have to be modeled and integrated into the virtual simulation, further indicating that it is necessary to compute interactions between organs and associated deformations.

We have therefore proposed a second patient-specific simulator, the Haptic Operative Realistic Ultrasonographic Simulator (HORUS). The HORUS is a new generation of medical simulator for the training and preparation of operations such as biopsy, puncture, or thermal ablation. It can thus be used to learn ultrasound (US)-guided percutaneous interventions within the framework of a training course for specialists, or preoperatively from CT or MRI images of a patient before carrying out US-guided interventions. Both the 3D operating room and the US images are represented (Fig. 1). To enhance the immersive sensation, two force-feedback Phantom Omnis from SensAble Technologies (Woburn, MA, USA) are used, one for the US probe and one for the needle. Once the biopsy or thermal ablation is completed, an automatic tool can evaluate the performance of the gesture on the basis of pertinent and objective numbers. To simulate a realistic ultrasonographic image, we have developed an automatic process that converts a 3D volumetric medical image (CT scan or MRI) into a realistic US image [22]. This conversion is performed in real-time according to the position of the force-feedback device’s stylus and allows real-time navigation in all planes and directions. Thus it becomes possible for practitioners to practice their intervention virtually before they carry out the real intervention. Obstetric and digestive interventions can be simulated, including amniocentesis, hepatic biopsy, and thermal ablation (Fig. 1) [23–25]. The system has successfully been tested with a 1-mm CT scan of a fetus at 36 weeks gestation, a 2-mm MRI image of a fetus at 29 weeks gestation, 4 patients for hepatic biopsy, and 2 patients for hepatic thermal ablation.

From the results of these previous efforts, working in partnership with the Karl Storz Company, we ultimately developed a new laparoscopic simulator called ULIS (Unlimited Laparoscopic Immersive Simulator). The objective of its first version was to be the first patient-specific laparoscopic basic skill-training simulator based on the Karl Storz Force Feedback System (FFB). The FFB system is composed of two force-feedback entry ports, allowing introduction of Karl Storz instruments that are real surgical tools. This simulator uses 3D modeled patients extracted from CT scan reconstructions that can be deformed locally with force-feedback rendering when a virtual instrument is in contact with simulated tissues. A photo-realistic texturing of tissues was performed manually, which required more than a week of work. The ULIS system includes camera and surgical instrument manipulation and simulation of suction and coagulation procedures. Finally, this simulator presents a new vision of basic skills that is much closer to real laparoscopic surgery than existing simulators, thanks to the use of real surgical instrumentation, patient-specific 3D models, photo-realistic artistic rendering, and realistic physical rendering, including gravity (Fig. 3). The next step in this work will involve developing a preoperative version, which means automatic texturing of 3D-modeled organs and integration of the complete simulator engine, SOFA (Simulation Open Framework Architecture) [26]. The SOFA feature allows multiple interactions and includes the resection developed in our first patient-specific simulator for laparoscopic liver surgery.

Fig. 3
figure 3

Several screenshots of the Unlimited Laparoscopic Immersive Simulator (ULIS) simulator, illustrating the laparoscope manipulation (left), a virtual blood injury in the bowel area (center), and a coagulation including smoke rendering (right)

Full size image

Perspective and conclusions

We have shown that patient-specific surgical simulation has become possible thanks to the development of 3D patient-modeling software platforms, online services, and new simulation techniques. These simulators will be used by both young and expert surgeons to test performance of an entire procedure on a 3D model of a patient reconstructed or downloaded through a specific distant 3D-modeling before the surgeon carries out the actual operation. We anticipate that patient-specific surgical simulators will soon become a major training tool for any surgeon. It is possible to imagine that with such an innovation, simulation will become an integral part of all surgical procedures. Eventually, with such a tool, all procedures would necessarily include a preoperative simulation based on the digital copy of the patient, with the aim of reducing risks and postoperative complications thanks to an improved knowledge of the surgical gestures. Medicine will then approach the quality assurance of aeronautics.