Introduction

More than a decade ago the Institute of Medicine report, “To Err is Human,” increased the public and institutional awareness to the high prevalence of medical errors in modern healthcare and their subsequent impact on patient safety in the United States [1]. Although multifactorial in origin, important subsets of medical errors were the result of invasive diagnostic and therapeutic interventions. This sparked different efforts within health care to address and improve patient safety by minimizing procedural complications [2, 3]. For one, educational bodies began considering methods of training that could improve physician education without putting the patient at risk. In the Institute of Medicine report, medical simulation was proposed as an efficient tool to meet some of these new demands, by allowing skills acquisition and training of invasive procedures in an environment where patients cannot be harmed. During the past decade, ongoing advances in computer graphics, software, and hardware design have refined medical simulators to offer life-like replications of medical and surgical procedures in a variety of medical specialties. Recent technological developments, especially in the field of image processing, now allow the incorporation of patient-specific imagery, such as standard CT or MRI data into these simulations [4]. This new technology has been referred to as “patient-specific” rehearsal or in analogy with the military, “mission,” or “procedure” rehearsal. It allows preoperative rehearsal of actual and upcoming patient cases on the simulator device. It complements the established role that virtual-reality (VR) simulators have been validated for in various minimally invasive procedures, namely as a generic training and assessment tool using preset cases on the simulator. Patient-specific rehearsal marks a shift in how VR simulators can be used as this concept of simulated rehearsal allows practice of a specific event (i.e., an operation) opposed to merely acting as a generic training tool to practice a specific skill. Furthermore, patient-specific VR rehearsal not only allows procedure planning (cognitive rehearsal) but by way of a haptic interface device, also permits a “hands-on” rehearsal of the actual procedure (psychomotor rehearsal). These characteristics may increase patient safety by optimizing patient selection, increasing physician preparation, and preventing complications or suboptimal surgery.

The present review illustrates the current place of VR simulation in medicine, discusses recent developments in patient-specific VR simulation, and draws parallels to other industries. Evidence in this new field of simulation science is reviewed and potential applications are discussed.

Methods

A review of the literature was performed to generate a qualitative summary of the current state of patient-specific virtual reality simulation in medicine. PubMed and Internet search engines were used to retrieve data relevant to patient-specific virtual reality simulation in modern healthcare. All studies, reports, and reviews in the medical field pertaining to patient-specific VR rehearsal with the use of a haptic interface device were included. Reports on simulators only allowing physician training with generic preset cases were excluded. A combination of search items included: patient-specific, virtual reality, rehearsal, mission, procedure, and simulation. Additional articles were retrieved from the reference lists of identified papers as well as discussion with experts in the field of medical simulation.

Results

Use of VR simulation for training and assessment

In modern healthcare, medical simulators are progressively being incorporated into educational services offered to medical students, fellows, practicing physicians, nurses, and paramedical personnel. A major impetus for the incorporation of virtual reality simulation in medicine has been the introduction of minimally invasive procedures, issues of reduced time for training, and patient safety concerns. There are numerous reports on the use of generic VR simulations to acquire, improve, and sustain complex skills and achieve basic competence in the domains of laparoscopy, endovascular surgery, endoscopy, gynecology, and urology [5–7]. Simulation research has shown that acquired skills can transfer to real operations in a variety of specialties, which may improve patient safety [8, 9]. The ultimate success of VR simulation will rely on more widespread incorporation into existing training curricula, something that is still lacking at present. VR simulators are not exclusively used for training purposes, because they can be easily customized to provide both medical programs and certification boards with an objective tool for assessing physician skill and knowledge. Using simulation as a tool for certification is already the case in some specialties, such as vascular surgery [10], although VR simulation is rarely the sole method for assessment or credentialing.

The aviation industry is a prime example of an industry where simulation is deeply rooted in pilot certification and training [11], performed in highly sophisticated ($20 million) full-flight simulators that often qualify as zero flight-time simulators, which allow full conversion training of pilots with no real-time training in the actual aircraft. These high-fidelity simulators also are used for the engineering and design of actual aircrafts as they permit accurate evaluation and testing of prototype aircrafts and procedures in simulation (procedural prototyping) [11].

VR simulation has been an important part of both training and assessment in other high-stake industries, such as the oil, nuclear, and military industries as well [12–15]. Individual and team training in hazardous procedures in environments, such as oil rigs or nuclear power plant control rooms, can be replicated in simulation to allow practice in both standardized and crisis scenarios. Each branch of the military (Army, Navy, Air Force) is inundated with VR applications. The military uses VR for combat training and for vehicle training, including tanks, airplanes, submarines, and naval bridges, but also for medical training through institutions, such as the Telemedicine and Advanced Technology Research Center (TATRC). Many of the simulators can be linked to each other to allow multiuser complex military rehearsals.

In the medical domain, much like in the aforementioned industries, generic (VR) simulation is not exclusively used as a tool to aid the inexperienced trainee. Much like experienced pilots use simulation to learn to fly new aircrafts, simulation also has a role in training established physicians in new techniques and procedures that arise during their career [16]. This is especially true for minimally invasive techniques, such as endovascular and laparoscopic surgery. The additional potential of patient- or procedure-specific VR rehearsal in medicine can now allow physicians to tailor their training to a specific patient at hand.

Use of VR simulation for procedure or patient-specific rehearsal

Procedure rehearsal in nonmedical industries

One area within the military where simulation has been of considerable success is mission rehearsal for fighter pilots. A detailed visual database of the combat field, based on satellite data, is generated within the flight simulation. This enables flight crews to practice all aspects of the mission in rehearsal, such as navigation, targeting, and flight paths, and additionally allows evaluation of the feasibility of certain strategic plans [11, 17, 18]. High-speed connections between multiple simulators also allow evaluation and execution of multiship missions or Joint Taskforce Combat scenarios to test tactical maneuvers against each other in combat [19].

NASA, much like the military, is renowned for its extensive use of simulation to train astronauts and ground control personnel in various procedures. The aerospace industry also relies on the concept of procedure or mission rehearsal to prepare for space missions [20, 21]. Examples include extensive underwater simulations to rehearse all facets of complex missions, such as the repair of the Hubble telescope (Fig. 1), and “combined” simulation rehearsals by interfacing the Shuttle Mission Simulator (a high-fidelity mock-up of the flight deck of the Space Shuttle), a mock Space Station, and the Mission Control Center. This allows for full space mission rehearsal [22].

Fig. 1
figure 1

Pre-space mission rehearsal of Hubble telescope repair in NASA buoyancy pool

Full size image

Sports that rely on technology have been quick to incorporate VR simulation in an attempt to increase performance. Formula 1 teams, such as Ferrari and McLaren, extensively use simulation to prepare their drivers for circuits that they are unfamiliar with before race day. McLaren are reported to have built a £20 million simulator facility consisting of a full-size Formula 1 car [23, 24]. The chassis is suspended on a hydraulic system, which responds to specific the track conditions and to steering wheel and pedal movements. Its high-fidelity software allows incorporation of race specific data, such as minute three-dimensional (3D) track details, current weather conditions, tire wear, and engine response, and allows rehearsal of race specific scenarios to test, rehearse, and perfect all conceivable engineering options, such as gear ratios, aero-level comparisons, and tire settings, ahead of a planned race. Much like in the commercial aviation industry, the level of fidelity of the simulator even allows testing and fine tuning of new engineering developments before implementation in the actual racing cars [23].

Patient-specific rehearsal in medicine

Patient-specific VR simulation has become possible by the concurrent evolution in the domains of VR simulation and that of medical image processing. On the one hand, high-fidelity practice of numerous (generic) interventional and surgical procedures is now possible due to advancements in simulator software and hardware. On the other hand, medical image processing software has evolved likewise and permits automated and accurate 3D volume rendering and reconstruction of cross-sectional two-dimensional (2D) DICOM data, such as standard CT or MRI patient imagery. Furthermore, 3D delineation software platforms allow 3D mesh creation, which gives these structures (virtual) volume and allows for manipulation in a VR environment. An example of such automated software can be found in endovascular simulators (Angiomentor, Simbionix USA Corp., Cleveland, OH) [4] and in the field of VR hepatic surgery [25, 26].

The representation of an organ system or a whole human body as computerized data forms the input of a variety of technology applications. First of all, these 3D volume-rendered images can be used to facilitate the comprehensive review of 2D data, which can otherwise be difficult to interpret. Some of these segmentation programs allow additional manipulation of imagery that may aid the physician in planning their interventions in greater detail. For example, 3D reconstruction software for liver imaging allows assessment of potential resection planes in complex tumor surgery [26]. The 3D imagery also can be used to provide “augmented reality.” This refers to the superposition of 3D-rendered imagery on the real intraoperative view (for example, via the camera view during laparoscopy) to provide “a transparent view” and facilitate navigation [27]. The 3D information also can be incorporated into a VR simulator for the purpose of simulated rehearsal. The present review focuses on VR simulator systems that allow incorporation and interaction with patient-specific models through a haptic interface device of which a summary can be found in Table 1. All simulators rely on standard CT or MRI DICOM data as input. This data is then reconstructed using existing or proprietary reconstruction programs to create a 3D mesh, which forms the basis for the actual simulation.

Table 1 Overview of current research into patient-specific virtual reality medical simulation
Full size table

The first patient-specific VR training device was described by Wildermuth et al. in 2001. His group assessed the feasibility of creating a patient-specific colonoscopy simulator [28]. The simulator system allows interaction by way of both non-force-feedback and haptic devices, simulating different instruments, such as an endoscopic biopsy forceps and a snare for polyp extraction (Fig. 2). Clinicians validated the colon models, the visual display, and force feedback as similar to real procedures but indicated that more refinement of forces and visual appearance is desired before wider deployment and use as a preparatory tool for patient-specific interventions.

Fig. 2
figure 2

Patient-specific colonoscopy simulator (printed with permission)

Full size image

Soler et al. reported on their Haptic Operative Realistic Ultrasonographic Simulator (HORUS), designed for training and rehearsal of interventions, such as ultrasound (US)-guided percutaneous obstetric and digestive interventions, including amniocentesis, hepatic biopsy, and thermal ablation (Fig. 3) [29]. The simulator also provides objective performance metrics, such as distance to the target pathology, safety of the path, and amount of tumor ablated [29]. The system was successfully tested using a 1-mm CT scan of a fetus at 36 weeks gestation, a 2-mm MRI image of a fetus at 29 weeks gestation, 4 patients for hepatic biopsy, and 2 patients for hepatic thermal ablation [29]. At present, no validation reports have been published. Research from the same group also resulted in a new laparoscopic simulator called ULIS (Unlimited Laparoscopic Immersive Simulator) [25]. Patient-specific rehearsal was accomplished by using real surgical instrumentation on patient-specific 3D models that had undergone photo- and physical rendering. In the prototype, actual patient-specific liver resections were not yet possible and further developments are awaited. In the same surgical field, simulation models for patient-specific laparoscopic colectomy also have been described [30]. The simulation, with realistic tissue-deforming effects of mesentery and ileocolic vessels, was assessed by experts and deemed realistic; however, further validation is necessary before the large-scale application on patients in a real-life clinical setting.

Fig. 3
figure 3

Haptic Operative Realistic Ultrasonographic Simulator (HORUS), designed for ultrasound (US)-guided percutaneous interventions (printed with permission)

Full size image

In the field of neurosurgery, Sierra et al. have reported the development of a patient-specific virtual model for navigation for ventriculoscopic interventions [31]. Patient-specific DICOM data are used to create 3D imagery of the brain and ventricles as well as a 3D hollow plastic model of the ventricle, placed in a styrofoam head model. With a NDI Aurora tracking system (Northern Digital Inc, Ontario, Canada), the position of the tip of the endoscope can be followed in the plastic model and virtual model. Preliminary research showed that tracking in both models was near identical and that further development of a fully virtual model with a tailored haptic interface device mimicking the endoscope is feasible. This would make the evaluation of different ventriculoscopic entry points, instrument workspace traversals, and planning/procedure prototyping possible.

In 2009, before the resection of a benign brain tumor on a 48-year-old woman, the exact same surgical procedure was performed on the Neurotouch simulator, developed by the National Research Council Canada [32, 33]. The prototype simulator allows for realistic interactions with 3D rendered MRI patient-specific brain imagery with a proprietary haptic feedback system. Instruments resembling a scalpel, electrocautery, and suction have six degrees of freedom and the simulator recreates the force-feedback of the real tools in relation to the varying resistance of tissue in brain regions with differing toughness (Fig. 4). Photo-realistic on-screen imagery shows the simulated surgery, including bleeding and pulsing gray matter. Several prototypes are under investigation at neurosurgical centers throughout Canada and validation reports are awaited.

Fig. 4
figure 4

Neurotouch simulator developed by the National Research Council Canada (printed with permission)

Full size image

In the field of orthopedics, patient-specific simulation efforts are scarce, but Pettersson et al. have described the first steps in the realization of a patient-specific hip fracture surgery simulator [34]. Their initial research has concentrated on the process of 3D rendering and using this information for visual and haptic purposes, such as providing realistic haptic feedback during drilling with different bone densities. At present, the realism is limited due to the use of an affector pen instead of an actual (simulated) handheld drill. In the field of craniofacial surgery, Meehan et al. describe a visiohaptic simulator system for both general-purpose training and preoperative, patient-specific rehearsal, helping the surgeon to decide on the optimal osteotomy location for craniofacial bone distraction surgery (for congenital or traumatic facial deformities) [35]. One or two SensAble Phantom haptic devices control several tools (drill/saw that can interact with the bone data). Furthermore, a virtual nerve monitor is incorporated that signifies nerve activity in the vicinity of the osteotomy. However, these nerve and blood vessel localizations are not yet related to patient data. The simulator is therefore not a full task simulator and soft tissue modeling is not replicated. A computer-based simulator to train cleft lip repair does replicate facial soft tissue, by adding an additional mesh between skin and bone meshes to simulate the volume of enclosed tissue. The soft tissues can be incised and moved to perform any of the major steps involved in cleft lip repair (Fig. 5). Although the authors emphasize the training potential, they also acknowledge its potential as a preprocedural planning and rehearsal tool [36].

Fig. 5
figure 5

A simulator to train for patient-specific cleft lip repair (printed with permission)

Full size image

Parikh et al. report their efforts on the creation of a patient-specific endoscopic sinus surgery simulator [37]. Because the difficulty in this type of surgery is largely related to anatomic variability, the emphasis in this simulator is on the visualization of the relevant anatomy and navigation of the endoscope through the sinus regions. The so-called Stanford Virtual Surgical Environment (VSE) for rhinologic procedures allows a surgeon to drive a virtual endoscope through the sinus regions (Fig. 6). A present limitation of the simulator is that therapeutic actions and tissue deformation are not replicated. Nonetheless, face validity was established by experts during virtual and real interventions on two patients. Even subtle anatomic variations were noted on the virtual imagery. Importantly, in this simulator set-up, the automated volume rendering of CT images and the creation of a subsequent simulation was reported as being extremely fast, only taking a matter of seconds. The authors speculated that a commercial variant of the VSE simulator would cost $10,000, which is considerably less than simulators used in other medical domains. This could expedite implementation of such devices in daily practice and increase the number of physicians with access to this kind of patient-specific simulation technology.

Fig. 6
figure 6

Stanford Virtual Surgical Environment (VSE) for rhinologic procedures (printed with permission)

Full size image

In 2002, the preliminary results of a simulator were described incorporating patient-specific 3D CT and MRI images of the mastoid region, allowing for a simulated rehearsal of a mastoidectomy procedure using a PHANToM haptics device [38]. Although the authors mentioned that face validity was rated adequately by otolaryngology surgeons, no further reports were found on the further development or implementation of the simulator in daily practice in a patient-specific setting.

At present, patient-specific VR rehearsal is most prominent in the domain of endovascular surgery. The first procedure rehearsal for a complex endovascular intervention was conducted at EuroPCR in 2006, using a Procedicus Vascular Interventional System Trainer (Mentice AB, Gothenburg, Sweden) to rehearse preoperatively a carotid artery stenting procedure on a 64-year-old male. An excellent similarity was noted between the simulated and actual patient case, with the operator using the same endovascular material during both cases [39]. A good correlation between angiography images also was noted. This first case was however a one-off, as the necessary preprocessing of patient imagery was time-consuming and expensive and had to be outsourced to the simulator company. This made the setup of rehearsals unpractical in a clinical setting.

Since then, another medical simulation company (Simbionix USA Corp., Cleveland, OH) has developed the PROcedure Rehearsal module for its Angiomentor group of endovascular simulators for carotid artery stenting procedures. This software and hardware is commercially available and via a user-friendly interface an automated and manual segmentation of MRI and CT DICOM data of the arch and carotid vessels is readily performed (Fig. 7). This allows the creation of VR rehearsals by the interventionalist or surgeons themselves in less than an hour. The resulting simulation contains a model of the patient’s anatomy and replicates the visual, auditory, and tactile aspects of the carotid stenting procedure for that particular patient. The simulator also records various objective quantitative metrics, tool selections, and fluoroscopy use and enables the user to record certain procedure steps to create a preoperative strategic plan.

Fig. 7
figure 7

Process of patient-specific rehearsal for carotid artery stenting using the Angiomentor Simulator (left). Example of its use as a rehearsal tool in the Simulated Operating Suite (SOS) before the actual case in the Angiosuite (right) (Imperial College London)

Full size image

Scientific research performed by the EVEResT research group has established that setting up these patient-specific virtual simulations is feasible in the clinical setting, using standard patient CT or MRI imagery to create the simulations [4]. Face validity was established by experts, and VR rehearsal has been shown to influence both experienced and inexperienced interventionalists in their tool choice for complex procedures [40]. When using this technology in vivo on actual patients, several research groups noted a strong similarity between simulated and real procedures with regards to endovascular tool choice, fluoroscopy preferences, and access strategy [4, 39–44]. More importantly, there is data to suggest that that a full-task CAS procedure rehearsal can increase the performance of novice interventionalists performing patient-specific cases in a simulated environment compared with no preparation at all [44]. Further research is focusing on the effect that procedure rehearsal may have on the nontechnical performance of the interventionalist, on the team performance, and on patient-safety. Efforts are underway to expand the procedure rehearsal capabilities of the simulator to interventions for abdominal aneurysms, coronary angioplasty, and below-the-knee angioplasty.

Discussion

Patient-specific VR rehearsal, possible due to the integration of 3D image data into the simulations, is a promising technological advance within medicine. It enables VR simulation to be used for more than generic training and assessment, and provides the opportunity for patient-tailored rehearsal and planning of interventions. The concept of VR procedure rehearsal has already been adopted with success in other high stake industries, such as the military, aviation, and sports.

Although the generic modules within simulation can undoubtedly effectively replace a part of the training on patients and shorten the learning curve for procedures, trainees still have to fine tune their skills on actual patients under the supervision of a mentor. Patient-specific rehearsal seems an ideal instrument to facilitate this crucial step and potentially increase patient safety during resident training. From a training perspective, patient-specific simulations also can be compiled as a growing library of cases that can be practiced and analyzed, providing other trainees insight into difficult or challenging realistic procedure variants. The benefits of patient-specific rehearsal, however, are not limited to trainees as experienced practitioners may benefit as well. This is in part due to the fact that structured tools to aid the immediate preoperative preparation, other than a thorough review of medical imagery, do not exist. Although a detailed review of medical images will aid cognitive preparation, patient-specific rehearsal can additionally offer psychomotor preparation for the lead physician and entire (surgical) team. Through repetitive rehearsal, the interventionalist has the opportunity to discover problems and form a strategic operative plan. Evaluating different surgical tools, approaches, and strategies and addressing potential complications could result in a more optimal use of equipment, shorter procedure times, and ultimately a reduction in complications with an increase in patient safety.

The potential of patient-specific rehearsal also is broader than improving technical proficiency alone. In medicine, much like other high-risk industries, procedural success is strongly related to nontechnical aspects, such as optimal teamwork, leadership, situation awareness, decision making, and communication. Numerous adverse events within high-risk environments, such as the operating room and emergency department, are caused by human error and could be prevented by enhanced teamwork [45]. To prevent such errors in the aviation industry, the Crew Resource Management program is firmly implemented in aviation training and focuses on team training and optimization of nontechnical skills. In medicine, patient-specific rehearsal seems to be an excellent instrument to enhance the safety in the operating room by preparing the whole team on both a technical and nontechnical level.

In this respect, patient-specific rehearsal also can be used as a debriefing tool when complications or adverse events occur. By reenacting the actual operative circumstances in simulation, it can be used to analyze errors and provide objective feedback enabling continued training for the surgeon and team, and also as part of a physician credentialing process.

Because patient-specific rehearsal allows a detailed case review, it seems an ideal tool to assess patient suitability for specific operations and guide treatment by identifying high-risks patients. Examples include technically demanding procedures, such as carotid artery stenting, or complex tumor resections in the domains of liver and neurosurgery. The information gained from these kinds of rehearsals also can be used to educate and prepare patients before their intervention, providing them insight into the different procedure steps, specific risks, and expected results.

If VR simulators evolve and achieve higher levels of fidelity, they could eventually be used for procedure prototyping: acting as “guinea pigs” to test new operative techniques, material, and approaches and as such guide the process of device engineering and product refinement. This would undoubtedly prove to be safer, shorten product development, and prove to be cost-effective. However, much like in the aviation industry, this will only become feasible when computer algorithms supporting the simulation software allow the simulator to abide all laws of physics and replicate human anatomy, physiology, and biomechanical properties to the highest degree. If this level of fidelity is reached, it seems reasonable to suggest that the technology of VR rehearsal can merge with the potential surgical robotics have in performing intricate procedures. Thereby surgeons cannot only practice a patient procedure beforehand until perfection is reached, but these actions and data can be recorded, verified, and translated to the surgical robotic. This robot can then perform the identical procedure on the actual patient, but faster, ergonomically, technically superior, and with better patient outcomes. Accurate tele- or remote surgery will be possible whereby a surgeon, or preprogrammed robotic, performs interventions without being physically present in the operating room [46]. All of these aforementioned techniques can potentially be merged into one robotic console or workstation that allows the surgeon to perform both open and minimally invasive surgery, remote telesurgery, preoperative rehearsal, intraoperative image-guided surgery, and surgical training [47].

At present, however, major developments are still necessary, because the majority of simulators providing patient-specific rehearsal are still prototypes. Cooperation between academic research groups, medical professionals, and simulator companies will be vital to commercialize patient-specific VR simulation and make it accessible to a wide group of practitioners in different medical specialties. Just as generic simulation had been scientifically validated for the purpose of physician training, patient-specific rehearsal will have to undergo the same scrutiny and be validated for its role as a rehearsal, planning, and team training tool. In the future, it can be expected that diverse technologies, such as VR rehearsal, augmented reality, robotic surgery, and telesurgery, will merge to improve patient care and safety through automation of patient selection, planning, and intervention. Medicine will then approach the quality assurance of other high-stake industries, such as aeronautics, where these technologies have already been successfully implemented.