The widespread adoption of laparoscopic surgery has led to the development of many novel approaches for surgical training and assessment. One of the foremost is the development of laparoscopic simulators [1]. The need for these simulators is largely because the skill sets required for laparoscopic surgery cannot be acquired exclusively via the old “apprenticeship model” of observership and assisting [2–6]. In addition, examples reported in the literature describe serious complications related to technical errors. As a result, there is increasing need to ensure that trainees have demonstrated sufficient technical competencies before performing laparoscopic procedures on patients [7]. Recently, considerable interest has led to the development of various virtual reality simulators for both generic and procedure-specific laparoscopic skills.

Several simulator systems have been developed, and studies have mainly demonstrated that these systems can differentiate between experienced surgeons and novices (construct validity) [8–12]. However, little evidence exists to show whether assessment of psychomotor skills using these trainers correlates with actual performance in the operating room (predictive validity).

The current study aimed to validate the role of virtual reality computer simulation as a method for assessing psychomotor skills in laparoscopic surgery. We aimed to investigate whether performance in the operating room, assessed using a modified Objective Structured Assessment of Technical Skill (OSATS), correlated with the performance parameters registered by LapSim (Surgical Science Ltd., Gothenburg, Sweden) during simulated laparoscopic tasks.

Methods

Demographics

The study enrolled 10 surgical residents (3 females) with a median of 5.5 years (range, 2–6 years) since graduation who had similar limited experience in laparoscopic surgery (median, 5; range, 1–16 laparoscopic cholecystectomies). All the study participants were required to perform three repetitions of all seven basic skills tasks on the LapSim laparoscopic trainer and one laparoscopic cholecystectomy in the operating room.

Virtual reality procedure

LapSim, version 3.0 is a PC-based virtual reality system. It consists of a 19-in. monitor and a laparoscopic interface module with two instruments and a footswitch (Fig. 1). The software is run on a dual-processor Pentium III computer with 256 MB of RAM using Windows 2000 (Microsoft, USA). The software consists of two modules equipped, respectively, for basic skills and procedure-specific skills (laparoscopic cholecystectomy and laparoscopic gynecologic procedures).

Fig. 1
figure 1

Correlations between operating room performance and performance demonstrated on a virtual reality simulator. (A) Time. (B) Error scores. (C) Economy scores

Full size image

The basic skills package offers eight tasks with increasing complexity. Task 1, camera navigation, provides basic navigational skills, with balls appearing that must be centered in the camera view. Task 2, instrument navigation, trains the three-dimensional navigation of two instruments (both hands) in a two-dimensional world by pointing out randomly appearing balls. In task 3, coordination, one hand controls the camera and the other an instrument pointing out randomly appearing balls. Task 4, grasping, is the first task involving manipulation of objects by grasping, stretching, and removing blood vessels using grasper instruments. In task 5, lifting and grasping, an object with tissue-like properties must be lifted to grasp and remove a needle and put it in a box. Task 6, cutting, simulates a real surgical procedure by the cutting of a vessel using ultrasonic scissor (controlled by a foot pedal). Task 7, clip applying, is the most complex task, requiring that a vessel be cut between two clips, with bleeding occurring if the vessel is overstretched, which calls for a prompt response (suction and a new clip). Finally, task 8, suturing, provides skills for laparoscopic suturing in a realistic intraabdominal environment.

The computer system registers performance parameters for each task: time, error parameters (tissue damage, measured as millimeters of tissue deformation and as a single touch), badly placed clips, blood loss, rip failure, instrument misses, and maximum stretch damage), and economy of motion parameters (path length and angular path).

Time, path length, and angular path are measured for all the tasks. The remaining parameters also are measured depending on the nature of each task.

Laparoscopic cholecystectomy in the operating room

After performing these tasks on the LapSim, all the study participants performed a laparoscopic cholecystectomy in the operating room with conventional laparoscopic equipment. The attending surgeon supervising the procedure was instructed to intervene only in the case of safety concerns. The attending takeover events were noted and excluded from the final assessment.

The operating room procedure was video recorded and blindly assessed by two expert independent observers using a modified OSATS rating scale (Table 1). The assessment in the operating room was based on three parameters: time used, error score, and economy of motion score.

Table 1 Modified Objective Structured Assessment of Technical Skill (OSATS)
Full size table

Statistical analysis

Data were analyzed using the Statistical Package for the Social Sciences, version 11.0 (SPSS, Chicago, IL, USA). During the tasks on the LapSim, time, errors (tissue damage, millimeters of tissue damage [tasks 2–6], and error score [incomplete target areas, badly placed clips, and dropped clips [task 7]), and economy of movement parameters (path length and angular path) were registered.

The correlation between time, economy, and error parameters during the simulated tasks and the operating room procedure was statistically assessed using Spearman’s test. A p value of 0.05 or less was considered statistically significant. Values are presented as median (range) unless stated otherwise.

Results

Significant correlations were demonstrated between the time used to complete the operating room procedure and the time used for task 7 (r s = 0.74; p = 0.015). There were significant correlations between the error score during laparoscopic cholecystectomy and tissue damage in task 4 (r s = 0.81; p = 0.04), task 5 (r s = 0.80; p = 0.05), and task 6 (r s = 0.77; p = 0.01); the millimeters of tissue damage in task 3 (r s = 0.89; p = 0.001) and task 6 (r s = 0.69; p = 0.028); and the error score in task 7 (r s = 0.67; p = 0.034). Furthermore, statistically significant correlations were observed between the economy of motion score for the operative procedure and LapSim’s economy parameters including path length in task 1 (r s = 0.69; p = 0.028), task 2 (r s = 0.79; p = 0.006), task 3 (r s = 0.82; p = 0.004), task 4 (r s = 0.88; p = 0.001), task 5 (r s = 0.93; p < 0.001), and task 7 (r s = 0.81; p = 0.005); and the angular path in task 2 (r s = 0.82; p = 0.004), task 3 (r s = 0.85; p = 0.002), task 4 (r s = 0.96; p < 0.001), task 5 (r s = 0.93; p < 0.001), task 6 (r s = 0.98; p < 0.001), and task 7 (r s = 0.90; p < 0.001).

Scatter plots of the values for the laparoscopic cholecystectomy and task 7 for time, error score, and economy of movement score are given in Fig. 1.

Discussion

As greater emphasis is placed on patient safety in surgical education, the need for standardized objective means to evaluate competency will be required. The expectation is that surgical trainees will have demonstrated basic competencies in simulated environments before performing procedures in the operating room.

Since the introduction of virtual reality simulation as a training and assessment tool in minimally invasive surgery, significant research efforts have been invested in demonstrating the validity and reliability of these systems. Most of the available evidence is for construct validity [15]. However, the relevance of these findings for clinical practice in the operating theater has been challenged. Our study demonstrated that surgeons who perform well in the simulated environment are competent in the operating room and vice versa, thus proving the clinical value of the assessment provided by the simulation systems.

The current study found significant correlations for all broad categories of performance on the LapSim and laparoscopic cholecystectomy performed for human subjects. The strongest correlation was found between the time required to complete the cholecystectomy and the time used for task 7. This is particularly meaningful given that task 7 is the most complex task on the LapSim, requiring the integration of several laparoscopic skills. Similarly, error scores and economy of motion scores were strongly correlated between the simulator and performance in the operating room. These results show that the LapSim computer simulator can measure psychomotor skills relevant for laparoscopic surgery in the operating room.

The advent of laparoscopic surgery has led to the development of numerous simulators for both general and specific laparoscopic skills. However, with the implementation of these systems in the training process, there has been a need to demonstrate that they are realistic, provide a reliable assessment of skills, and ensure that skills acquired in the virtual environment can be transferred to the operating room before their incorporation into the surgical curriculum.

The modified OSATS rating scale was used in our study to assess performance in the operating room. This tool, developed and thoroughly validated by Reznick et al. [13] in Toronto, currently is widely accepted as a tool for assessment of surgical technical skills. The original OSATS, developed for use in live observations, was modified for assessment of video-recorded procedures by Grantcharov et al. [14]. The scale showed excellent construct validity and interrater reliability and was proved to be valid as a tool for video assessment.

The current study may have some potential sources of bias. Scoring of the performances on the human laparoscopic cholecystectomy was subjective. This subjectivity was, however, minimized by the use of a structured and validated rating scale as well as two independent blinded assessors. The sample size in this study was small. Thus, larger studies are needed to confirm our findings. Furthermore, the learning curve for LapSim has not yet been fully delineated. All the participants were therefore exposed to identical familiarization procedures (i.e., completion of all seven tasks twice).

The current study demonstrates that performance scores registered by a virtual reality simulator correlate with performance in the operating room. Thus, the computer model represents a valid tool for evaluating surgical trainees, and can provide them with an objective quantitative assessment of their laparoscopic skills. Future studies are needed to determine the exact role of virtual reality simulators in a comprehensive surgical training and assessment curriculum.