Introduction

The adoption of robotic surgery across multiple specialties and it is now common occurrence at academic centers have led institutions to develop and incorporate their own formal robotic curriculums. These curriculums however are often institution- and specialty-specific [1]. Expert consensus has provided recommendations that robotic surgery curriculum development ought to include a virtual reality (VR) portion as part of simulation training [2]. Institutions that have developed their own curriculum have demonstrated improvement in robotic skills and increased trainee operative console time by performing virtual reality training in conjunction with synthetic exercises [3]. The Da Vinci Skills Simulator (DVSS) has demonstrated the highest face and content validity among robotic surgery VR simulators [4]. Other studies have demonstrated that VR exercises on the DVSS require a score > 90% to reach a standardized expert-level score [5]. A general surgery robotic curriculum that adopted the > 90% standard for VR exercises as part of their program demonstrated improvement in robotic skills assessed through the GEARS score [6].

Barriers to novice robotic surgeons have been identified as lack of familiarity with the robotic platform/technology and lack of psychomotor skills [1]. Robotic simulation training can assist in improving these skills and increase surgeon efficiency [1, 7]. VR training has demonstrated translatability to robotic operative skills. A meta-analysis looking at correlations between VR simulation training and skill acquisition identified 5 out of 8 studies that demonstrated translation of skills from a VR platform to the operating room [7]. Six out of eight of those studies were performed on the DVSS. VR exercises were variable (19 various exercises were identified) as was skills assessment which occurred on both human and animal models and involved assessing metrics, such as OR time, estimated blood loss, RO-SCORE, GEARS assessment, or patient outcomes. Additionally, only one of the studies included trainees from multiple specialties while other looked at specialty-specific trainees, medical students, or expert surgeons [8,9,10,11]. Additional studies have identified needs for objective and automated measures that help demonstrate improvement in robotic skills [12]. The variability in curricula development, cohorts, and assessment tools identifies a need for a standardized approach to VR simulation training with broad-based applicability, simple progress tracking, and automated assessment [13].

The Intuitive learning platform offers a new interface for all trainees and administrators to assign and track DVSS VR exercise progress [14]. Furthermore, it collects automated performance metrics (APM) on each exercise performed. Analysis of APM provides information with regard to both trainee performance and individual exercise performance. This includes exercise performance scores, time to exercise completion, attempts per exercise, economy of motion, penalty scores, and progression tracking. We hypothesized that trainees would show improvement in post-test scores after completion of the VR curriculum by demonstrating skill development and that performance would not differ significantly based on specialty but would differ based on PGY level. Additionally, we sought to utilize the automated performance metrics provided by the platform to assess areas for improvement in the curriculum and where to focus further skill progression.

Methods

Needs assessment

Our institution’s multidisciplinary robotics committee met regularly in 2019 through 2020 to discuss ways to enhance trainees’ preparation to participate in robotic surgery cases as a bedside assist and console surgeon. The committee performed an extensive literature review of other robotic training programs and performed a needs assessment of specialty-specific training concerns. Results from this process concluded that across all residency and fellowship training programs, attendance noticed a high variability of basic robotic skills among residents of all levels and new fellows. It was also noted that studies had been conducted which demonstrated benefits of VR-based training as part of formal robotic curriculums and showed that VR-based training can lead to skill acquisition [3, 4, 6, 7, 15].

Curriculum development

The exercises for the curriculum were chosen from the list of exercises on the DVSS which are divided up into the following categories: camera control and clutching, endowrist manipulation, fourth arm integration, energy and dissection, and needle driving [16]. The curriculum used this model to break up the exercises into seven distinct sessions and included 24 exercises (Fig. 1). The exercises included were similar to those used in other curriculums as well as exercises included in fundamentals of robotic surgery (FRS), thus eliminating the variability seen in exercise selection from other studies which had a narrower focus in skill evaluation [6, 8,9,10,11, 15]. A score of > 90% on each exercise was set as the standard passing score as this was demonstrated by prior studies to be a score consistent with expert skill level [5].

Fig. 1
figure 1

Virtual reality-based robotic curriculum on the Da Vinci Skills Simulator

Full size image

Learner assessment and tracking trainee progress

A pre-test and post-test were included in the curriculum to assess trainee improvement. Starting in February of 2020, all residents and fellows were enrolled in the VR curriculum. This included trainees from the following specialties: GYN, general surgery, urology, thoracic, hepatobiliary, colorectal, and ENT. Trainee progress was monitored using the Intuitive Learning site where they could track their APMs and observe their progress on demand. Monthly progress reports were generated from the APMs and distributed to all robotic committee members and program directors. (Sample exercise performance report demonstrated in Fig. 2).

Fig. 2
figure 2

Copyright Intuitive Learning

Individual exercise performance tracker. This demonstrates a trainee’s progress on the Three-Arm Relay 2 exercise. Each attempt can be accessed to look at specific performance metrics

Full size image

Data collection and curriculum evaluation

The Intuitive Learning platform was monitored routinely to track trainee progress and generate a monthly report to the trainees program directors. Exercise completion and APMs were recorded on a spreadsheet. Pre- and post-test scores were tracked. The DVSS was queried for all exercise data from the 12-month period of Feb 2020–Feb 2021. This provided data on every exercise performed by every trainee on the simulator in that time period. A T test was performed comparing all pre-test and post-test scores. Simulator data were analyzed to determine individual exercise performance. An analysis was performed to assess for differences based on specialty and PGY level by evaluating the following metrics: completion scores, attempts per exercise, time on completed exercises, and penalty scores. Trainees that completed the curriculum or completed > 50% of the curriculum were asked to fill out a curriculum evaluation survey.

Results

From Feb 2020–Feb 2021, 73 trainees were enrolled in the curriculum with 53 active users at time of data analysis. 21 Trainees completed the entire curriculum (6 General Surgery, 6 Urology, 5 GYN, 2 HPB, 1 Colorectal, 1 Thoracic). 23 Trainees had partially completed the curriculum (> 30% of exercises completed). Curriculum exercises were attempted 7450 times. The majority of trainees who completed the curriculum required on average 8 weeks for completion. Trainees spent a total of 416 h and 31 min on the curriculum. The average time for trainees to complete the curriculum was 7 h and 19 min. Analysis of pre-test and post-test scores showed significant improvement across all three exercises: Ring Rollercoaster II + 180.1%, Three-Arm Relay III + 174.45%, Anterior Needle drive ATW + 91.78% (Table 1).

Table 1 Pre-test and post-test comparison
Full size table

23 Trainees responded to our curriculum evaluation survey. 99% of trainees agree that the VR curriculum improved their robotic skills. 71.5% of trainees believe that completing the VR curriculum has granted them more console time in the operating room. The majority of trainees found the ability to track their progress using the automated platform helpful. Three-Arm Relay and suturing exercises were deemed most useful by trainees. Ring Rollercoaster exercises were deemed less useful or repetitive by trainees.

Analysis of variance (ANOVA) comparing specialties: General Surgery, GYN, and Urology showed no statistical difference in terms of completion scores, completion time (s), and penalty score. Attempts per exercise were noted to be significantly higher in general surgery than other specialties. ANOVA comparison based on level of training demonstrated that junior residents (PGY1-3) had significantly lower number of attempts per exercise, fewer penalties, and higher completion scores when compared to senior residents and fellows (Table 2).

Table 2 Exercise metrics categorized based on skill session
Full size table

Individual exercise data analysis provided the following metrics: average time spent per attempt, average number of attempts needed to achieve a passing score (> 90%), average amount of time investment needed to achieve a passing score, and average penalty score (Table 3). Exercises, such as Three-Arm Relay II and Ring Rollercoaster II, required the largest time investment to achieve a passing score (53 min 21 s, 50 min 36 s) whereas Clutch and Energy Pedals II required the shortest time investment (1 min 10 s, 4 min 42 s). Comparing exercise categories revealed that the Ring Rollercoaster exercises require significantly more attempts, more average attempts to achieve a passing score, and more penalties than the Energy Pedal Selection exercises (Table 4). When broken down based on exercise category, the majority of time on the curriculum was occupied by endowrist exercises (38.9%) followed by needle driving (25.8%), retraction arm (24.4%), camera control (4.2%), energy pedals/ dissection (4.1%), and finally, knot tying (2.7%) (Fig. 3).

Table 3 ANOVA comparison based on training level
Full size table
Table 4 Comparison of Ring Rollercoaster and Energy Pedal exercises based on recorded metrics
Full size table
Fig. 3
figure 3

Distribution of time spent on each individual exercise

Full size image

Discussion

Implementation of a robust VR-based robotic curriculum is feasible across multiple surgical specialties and can develop basic robotic skills. The Intuitive Learning platform makes curriculum design, assigning exercises, and tracking trainees’ progress incredibly easy. The automated performance metrics recorded by the DVSS provide an objective assessment tool and on-demand progression tracking for both trainees and instructors. Automated objective assessment data can greatly reduce the burden required by instructors to evaluate skill progression in trainees and streamline the learning process. This was the first study performed that used these data to not only compare the performance of specific trainee groups but also compare the metrics of individual exercises.

The curriculum was designed based on the principle of deliberate practice with repetition overtime to build skill performance and prevent skill degradation. Pre-test and post-test analysis demonstrates statistically significant improvement in VR performance which coincides with trainees overall perception of improved robotic skills. Trainees also reported that their console time in live OR settings increased after curriculum completion. This suggests that VR simulation serves a valuable role in developing basic robotic skills and upon completion, trainees can expect to get a better quality experience in the live OR settings. While the curriculum completion included members from 6 out the 7 specialties engaged in robotics at our institution, only 21 current trainees out of 53 completed the curriculum in its entirety. This lower attrition rate may be due to several factors including not making the curriculum obligatory, lack of interest among certain trainees, and significant time investment required for completion. Earlier survey data gathered revealed that the majority of trainees thought the time investment was the largest barrier to completion.

The differences seen in the ANOVA data were not expected. The higher number of attempts on exercises taken by General Surgery compared to GYN and Urology may have reflected that robotic general surgery was a newer service line with general surgery residents having less clinical experience relative to the other divisions. Even more interesting was the comparison based on training level. Junior residents (PGY1–3) took fewer attempts, incurred fewer penalties, and had higher completion scores compared to senior residents (PGY 4–5) and fellows. The small sample size of the analyzed groups may have contributed to this finding. However, it can also be theorized that more senior-level trainees already have some prior operative robotic experience and were attempting to translate what skills they have acquired to the simulator. Naive robotic trainees may not require an adjustment period and thus be able to complete the exercises in a more efficient fashion.

Analyzing individual exercise metrics revealed that trainees spent a large amount of time performing endowrist exercises (Ring Rollercoaster, Sea Spikes, etc.) and a minimal amount of time on exercises involving the energy pedals and knot tying. The time investment placed in the Ring Rollercoaster exercises correlates with survey data that trainees found these exercises most repetitive. Unlike traditional open surgical instruments or straight stick laparoscopic instruments, learning to manipulate wrist action is novel to robotic surgical instruments with no analogy to apply from prior clinical training. The working field of view for robotics can be a very tight space if the economy of motion provided by wristed instruments is underutilized. Without understanding the dynamic application of wristed motion these exercises may have required more repetitions and longer times to achieve a 90% score. This observation led us to create video demonstrations of “tips and tricks” for each exercise to shorten the trainees learning curve. The data generated will help us evolve our curriculum to better serve the needs of surgical trainees. Conversely, we can look to add more exercises that involve knot tying or energy pedals to improve the distribution of these skills in the curriculum.

Conclusion

A standardized multidisciplinary VR robotic curriculum is feasible and helps develop basic robotic skills. Intuitive Learning offers a powerful tool for tracking trainee progress, assessing individual exercise metrics, and allowing surgical educators to make adjustments to robotic training curriculums based on data.

Limitations

This study has several limitations. First, this curriculum is not validated to demonstrate improved robotic skills in the operating room or correlated with patient outcomes. While other studies have shown skill improvement with their robotic curriculums, none have done so with a purely VR-based curriculum. Our improved post-test scores and survey data provide a perception of improved skills. Next, our comparative analysis based on specialties and training level had small sample sizes. The differences seen among the groups may be due to outliers and more subjects will be required to confirm these differences. Finally, this study was performed at a single academic center and thus may not be generalizable; however, the improved scores and perceptions of improvement correlate with similar studies.