Introduction

The use of modern moderate- to high-fidelity simulators seen today gained momentum in the early 1990s due to medical education reform [1]. Recent advances have allowed sophisticated technologies to become more affordable [1] and, coupled with reports of stress to ill-prepared, skill-deficient young doctors, has meant an increase in the need for simulator-based training in the undergraduate years [1]. Similarly, changes to post-graduate training has necessitated the need for sound educational approaches to training [1, 3]. Reductions in junior doctors’ working hours [3] and a more streamlined, shorter specialist training model [3] have raised the need for simulator training in order to expose the trainee to sufficient cases, and thus increasing the material covered [1].

Acceptance of virtual reality (VR) training has been slow due to the reluctance of the medical community to use such a model for validated training. Reasons for this include the lack of evidence validating such training and the lack of any data showing a benefit to patients.

Simulation is widely used in the aviation and nuclear industries [2]. Commercial airline pilots are trained in 5,000 h and astronauts in 12,000 h, yet there is growing concern for post-graduate medical education with an estimated reduction in surgical training from 30,000 to 8,000 h [3].

Attitudes have started to change, with studies showing a definite benefit from simulated training to the surgical trainee and the patient [4]. One study has demonstrated that improved performance on a simulator leads to less operative complications in the setting of laparoscopic surgery [4]. Other studies in several specialties report positively on simulation: flexible sigmoidoscopy [5], endoscopic sinus surgery [6] and intra-corporeal knot tying [7]. There are also several studies investigating simulation in the setting of novel procedures such as carotid artery stenting [8, 9]; with subsequent FDA approval for carotid stenting being published in 2004 [10], the American College of Surgeons also support simulator training, which they believe will improve patient safety [11]. However, a meta-analysis was more cautious [12] and several authors advocate the need for further validation [13].

While there is literature supporting and validating the use of virtual reality simulators in surgical training in endovascular techniques [14–16], there has, to our knowledge, been no equivalent evaluation of training simulators for radiological trainees, nor an assessment of how simulators might be integrated into radiology training.

The aim of this study was to characterise the progress of trainees using a proprietary interventional simulator trainer by monitoring the time taken to complete tasks, with increasing repetitions.

Materials and methods

This prospective cross-sectional study was designed to record the time to completion for renal artery angioplasty over five repetitions. The study design was approved by the local research and ethics committee, and informed consent was obtained from all the participants.

All 12 of the new annual intake of radiology registrars agreed to participate in the study and data was collected prospectively by the authors between October 2006 and July 2007.

The simulator used in this study (Procedicus VIST, Mentice, Göteborg, Sweden) is a multimedia device that simulates endovascular techniques. The system is comprised of a desktop computer simulating a three-dimensional (3D) model of the arterial system (Fig. 1). It is linked to a haptic force feedback system that provides tactile feedback. A dual screen system allows selection of equipment and a simulated fluoroscopic image. Separate devices allow pneumatic injection of contrast, balloon insufflation and stent deployment. Foot pedals and a control panel allow control of the fluoroscopy suite.

Fig. 1
figure 1

A 3D view of the arterial system provided by the simulator

Full size image

A simulated procedure based on left renal artery angioplasty was devised with multiple steps to incorporate a number of tasks, particularly regarding equipment selection, guide-wire and catheter manipulation. The study and the modified left renal artery angioplasty procedure was discussed and subsequently demonstrated to the trainees in an afternoon session.

Trainees then performed five of these procedures over the course of their first 6 months of training.

The data collected on a proforma was used to account for the various steps taken by the trainees. The primary end-point measure was total procedure time. Secondary end-point measures included split-times, fluoroscopy times, metric information from the machine (such as accuracy of placement of balloon relative to the stenosis) and the number of mistakes made.

Split-times for specific sections of the procedure were also recorded as follows:

  1. Stage 1:

    Time to diagnostic aortogram (a basic component of the procedure).

  2. Stage 2:

    Time to cross the stenosis (requiring more guide-wire and catheter manipulation).

  3. Stage 3:

    Time to inflation of the angioplasty balloon (requiring more catheter exchanges).

The type and threshold for mistakes were agreed by the two observers. Mistakes included:

  1. 1.

    Incorrect selection of equipment—size, catheter, dose of contrast.

  2. 2.

    Incorrect use of equipment.

  3. 3.

    Incorrect selection of the next appropriate stage of the task.

The trainee was immediately informed if a mistake was made, in order to allow for correction and to ensure the procedure continued without losing significant time. Full feedback was given to each trainee at the end of a completed task, in order for the trainee to learn from errors and potentially avoid repeating them in subsequent tasks.

Field-notes were also made recording any other observations. These too were fed back to the trainees at the end of each procedure. These were purely observations made with regard to technique of using the simulator and not specifically task-related errors.

Times were analysed using a two-tailed paired Student’s t-test, comparing the time taken from one subsequent run to the next.

Total time (a marker of overall performance) and the split-times (markers for basic and more complicated tasks) as well as the fluoroscopy time were analysed.

The results were interpreted in light of the field-note observations and profile of mistakes.

Results

Fifty-two runs were completed by 12 trainees. All trainees undertook at least three runs, with the majority completing four or five.

The mean procedure time decreased from 16.6 min to 9.8 min over the five runs (Table 1). It tended to decrease in a step-wise manner except on the fourth attempt, where this time increased compared with the third (Fig. 2). All times were statistically significant compared with the first attempt, but there was no significant improvement comparing sequential performance times (Table 2). Analysis of the split-times for the three stages showed a similar trend of improvement.

Fig. 2
figure 2

The mean procedure times for five repeated simulator runs

Full size image
Table 1 Comparison of times taken for five repeated simulator runs
Full size table
Table 2 Comparison of mean total procedure time between serial attempts (p values)
Full size table

Although there was a general improvement in time, the number of mistakes fluctuated. The number of mistakes ranged from 0 to 10 and the mean number of mistakes made varied from 0.7 to 2.6 per procedure without any particular trend.

Common mistakes made initially were with equipment selection, lack of familiarity with the equipment (for example, needing to repeat the aortogram because of failing to co-ordinate the timing or using the incorrect pedal) or omitting steps in the procedure (such as failing to use a flush of contrast to confirm an intra-luminal position). Mistakes that continued to be made included poor movements of wires and catheters (for example, movement of a catheter without a supporting wire, too rapid wire movements and malposition of the angioplasty balloon).

Fluoroscopy time also decreased. Sequential comparison of the fluoroscopy time compared with the first run only reached significance at third run (p = 0.04). Although there was further decrease in mean time in the later runs, there was no significant improvement due to wide variation of use, with some operators using the foot pedal indiscriminately, despite feedback at the end of each procedure (Table 3)

Table 3 Flouroscopy times
Full size table

.

Discussion

Our study demonstrates that simulator training results in improved performance. The quantitative data demonstrates that overall time decreased with repeated attempts. The time taken to complete the whole procedure decreased and showed a profile of gradual improvement, mostly occurring early and tending to plateau with time.

Both simple and more complex stages showed improvement. Stage 1 involved obtaining a diagnostic aortogram and this improved rapidly and the times reached plateau early. Stages 2 and 3 involved more complex stages, such as guide-wire manipulation into the ostium or across the stenosis and catheter exchanges. These findings unsurprisingly suggest that simple tasks are easily learned and more complex tasks take longer to master.

The use of fluoroscopy also decreased, but this decrease was not statistically significant due to wider variation between the trainees and some used the foot pedal indiscriminately despite feedback. It could be postulated that, as this was a complex task, the operator may have concentrated his efforts on selecting equipment and hand-to-eye co-ordination, whilst neglecting optimal control of the fluoroscopy pedal.

Some operators demonstrated an excellent learning curve, consistently making fewer errors. Analysis of the group as a whole, however, demonstrated that mistakes did not systematically reduce. This may have been due to the large time intervals between subsequent attempts, and all participants were exposed to live interventional cases in their training curricula. Some of these may have included renal interventional procedures.

The qualitative observations and profile of mistakes demonstrated an ongoing learning process. This appeared to be supported by both improved familiarity with the equipment, and increased confidence amongst trainees. The latter, however, caused some trainees using the simulator to actively experiment with catheter and guide-wire manipulations, leading to an increased error profile.

Given the above observations, it can be concluded that although one or two trainees were performing the task optimally without making mistakes by the end of the fifth run, several participants were still making errors and had room to improve their performance.

Our methods of both observation and metric measurement have been described as reliable [17]. However, the machine-generated measures of performance are relatively limited—procedure time, fluoroscopy time, measurement of contrast and balloon or stent position—and are crude measures of performance. Also, there is no simulation of complications such as dissection. Further limitations of our study include the small sample size and our measurable endpoints. The participants of this study do not necessarily represent a general trend of radiology trainees but the results do allow a basis for power calculations for further research. The measurable endpoints are only surrogate markers of performance. In clinical practice the success of an interventional procedure is considered by the quality of the diagnostic images, the clinical outcome of the therapeutic intervention and the experience of the patient. These are all complex measures, which can not yet be modelled on radiological virtual reality systems. Until these measures are available measures of time to completion and screening time may be helpful in identifying those trainees who fail to progress in developing interventional vascular hand-eye co-ordination.

Our study did take into account an assessment of technique through the mistake profile. Dawson et al. [15] demonstrated a marked improvement in time from 42 to 19 min after 8 h training on a 2-day course, with fluoroscopic use decreasing from 14.5 to 2.2 min. The study trained vascular surgery residents to perform arteriography and interventions in aortoiliac, renal and carotid artery disease. This dramatic improvement leads to a concern of “negative training”—for example, if the operator is repeating a standardised task with the single aim of improving time, dramatic improvements in time could be made due to the stereotypical nature of the simulated stenosis—though parameters can be changed (for example a “high take-off” renal artery), in an experimental setting, the same scenario is used—the same artery, the same origin, the same stenosis. Our participants were judged to have made a mistake if their wire or catheter manipulations were deemed to be cavalier or race-like.

Neequaye et al. [16] examined 20 surgical trainees over eight sessions. Though the authors conclude that there was a plateau at the fourth attempt in iliac angioplasty, this was to do with a single step: placement of the balloon across the stenosis. The overall time, though improved, did not plateau, in line with the results from our study. Indeed, another strength of our study was that it was conducted in a pragmatic setting, with radiology trainees, who were exposed to interventional procedures throughout the year-long period, whereas the time setting of the eight sessions is unclear and the surgical trainees (namely SHOs and registrars, who would not have been exposed to endovascular procedures).

Despite the presence of several studies showing improved performance at tasks, there are several unanswered questions concerning skill transfer to real clinical situations and how to best integrate the use of simulation into radiological training.

There are several reports discussing different theories of gaining the knowledge, skills and attitudes required for achieving competency in practical surgical or interventional tasks [18–24]. Our method of using the simulator in this experiment was modelled on Kolb’s experiential learning [23]—acquisition of knowledge followed by activity, with resultant reflection or mentor-guided feedback lead to improved performance in the next cycle. If the acquisition of competency follows specific stages, then simulator training can allow knowledge acquisition and opportunity to repeat and practice specific tasks. Simulator training can, therefore, again offer an opportunity to perform parts of tasks and to take decisions about what to do next, while being in a remote environment with no risk to the patient. Advantages of simulator training include it being trainee-focused, occurring in a safe learning environment, where mistakes are entirely positive learning events with no risk to patients. Simulation does not rely on case-flow as material is constantly and consistently available. Disadvantages include that the training occurs in a sterile environment. As a question of “real world” skills, simulation may not be realistic enough, and is not validated. There is concern regarding negative training, where situations may arise where the simulator does not properly replicate the task—leading to the development or acquisition of inappropriate actions.

Conclusions

Our study demonstrates that training on the simulator does improve performance in line with other studies. Total procedure time did reduce significantly by repetition over a period of months. The improvement in error rates is more variable between trainees. The strengths of our study include it taking place in a pragmatic setting with access to real interventional cases across the study, and that assessment of technique and therefore safety was incorporated into our evaluation. Although some operators did perform the task without error, the majority of trainees continued to make errors after five attempts and would, therefore, potentially benefit from further simulator training. It remains unclear how to integrate this form of training in current programs and the optimal role for the simulator in interventional training is yet to be determined.