Introduction

Ureteropelvic junction obstruction (UPJO) is an impairment of urinary transport out of the renal pelvis, which arises due to intrinsic or extrinsic narrowing of the proximal ureter [1]. Intrinsic causes include a congenital aperistaltic segment of ureter or acquired strictures from stone disease or inflammatory conditions [1, 2]. The most common extrinsic cause of ureteral obstruction is from a lower pole accessory crossing vessel [1, 2]. This condition is overall quite rare, with a congenital incidence of 1 in 500 live births [3]. Although not all patients with UPJO are symptomatic, those that are may develop pain, urinary tract infections, kidney stones, renal deterioration and/or renal loss [1, 4].

The preferred treatment for UPJO is dismembered pyeloplasty [1]. The procedure involves transecting the ureter at the level of the narrowed segment and re-anastomosing the ureter to the renal pelvis. If there is a crossing vessel, the anastomosis is transposed anteriorly to the crossing vessel to correct the underlying pathology [1]. With advances in minimally invasive technology and in an effort to minimize morbidity, laparoscopic and robotic approaches have become the mainstay of treatment [1, 5, 6]. In particular, robotic surgery has facilitated intracorporeal suturing of the anastomosis resulting in excellent success rates with minimal morbidity [5, 6].

As UPJO is a relatively rare condition, hands-on exposure may be limited by the low surgical volume of this procedure, which can be a challenge for the training of urology residents. Many residency programs in the US use simulation-based training to supplement trainee operative experience. The benefit of robotic simulation for residents and experienced surgeons has been well established [7]. The majority of the literature regarding surgical education as it applies to robotic training however, is limited to virtual reality models [7]. Using virtual reality models, variables such as operative time, economy of motion, instrument collision, and instrument force can be evaluated, however, these given models are two dimensional, certain objective measurements are more difficult to assess (i.e. patency, caliber, leak rate, surgical margins) without a physical model.

With the advent of 3D printing, tangible models are now being used in surgical training and its educational value is being assessed. Several pilot studies using highly realistic 3D models have demonstrated improved comprehension of surgical anatomy by residents and have been rated favorably by surgeons and trainees [8,9,10]. Few studies, however, have evaluated objective surgical outcome measures [11].

Crowdsourced review of surgical performance has emerged as an alternative method to expert surgeon review given that the latter is relatively time consuming. Studies have demonstrated that crowdsourced review has a good ability to differentiate the performance of surgeons with varying levels of experience, and has a strong correlation between aggregated crowdsourced scoring and expert review [12, 13]. In addition, crowdsourced methods are faster and more cost effective [12, 13].

We developed a robotic-assisted dismembered pyeloplasty simulation using a 3D-printed silicone-based model of UPJO for surgical training and sought to perform face validation of the model and objectively assess surgical performance and learning outcomes using a crowdsourced platform.

Methods

This prospective study was reviewed and approved by our Institutional Review Board. Eleven participants (eight urology residents, PGY-3 to PGY-5 and three faculty, fellowship-trained in robotic surgery and with previous pyeloplasty experience) were recruited after obtaining informed consent.

Lazarus 3D, LLC (Houston, TX, USA) created the models using 3D modeling software based on computerized tomography (CT) scan imaging and printed using a silicone-based material. The model renal unit consisted of a kidney, a dilated renal pelvis to simulate UPJO, and ureter. The models were secured in a laparoscopic box trainer, and the da Vinci® Si robotic system (Intuitive Surgical Inc., Sunnyvale, CA, USA) was docked (Figs. 1, 2).

Fig. 1
figure 1

Laparoscopic box trainer with da Vinci® Si robotic system docked and pyeloplasty model secured inside

Full size image
Fig. 2
figure 2

View from inside the laparoscopic box trainer, with 3D-printed model secured and da Vinci® Si robotic system docked

Full size image

Prior to the simulation, all participants watched a 5-min edited video of an experienced surgeon performing a robotic-assisted dismembered pyeloplasty on a 3D-printed silicone model. The steps of the procedure were as follows, (1) incising the renal pelvis at the level of the stricture, (2) spatulating the ureter laterally, (3) continuously suturing the posterior aspect of the ureter to renal pelvis, (4) continuously suturing the anterior aspect of the ureter to the renal pelvis and (5) tying the ends of the suture together. These steps included all of the major skills necessary to perform a typical dismembered pyeloplasty [1, 14]. 3-0 prolene on an RB-1 needle was chosen for this simulation, as it is a synthetic monofilament suture similar to the typically used PDS with several favorable properties, including ease of passage through the silicone model, visual contrast against the renal pelvis background, and cost effectiveness. Ureteral stent placement was not included as this simulation focused on completion of the anastomosis. The participants then performed a right-sided robotic-assisted dismembered pyeloplasty following the above steps on one model on two separate occasions (Fig. 3). The time to complete the anastomosis was measured. For the second trial on the same model, participants had to excise the previous suture line, and timing and scoring was started after this point to maintain uniformity. Following the procedure, the model was assessed for completeness of the operation, patency of the anastomosis, and whether or not a leak was present. Finally, the participants were surveyed for face validity of the model using a 5-point Likert scale and for pre- and post-procedure confidence in their ability to perform a pyeloplasty unassisted.

Fig. 3
figure 3

Anterior anastomosis being performed on a 3D-printed silicone model of UPJO

Full size image

De-identified video recordings of each simulation trial were collected. A standardized 10-min segment, starting with the placement of the first suture of the anastomosis, was sent to C-SATS (Crowd-Sourced Assessment of Technical Skills Inc., Seattle, WA, USA). Scoring was done using GEARS criteria, a validated assessment of robotic surgical performance [15]. These criteria assign scores ranging between 0 and 5 in five fundamental categories of robotic surgical performance: depth perception, bimanual dexterity, efficiency, force sensitivity, and robotic control. These scores are summed for a maximum score of 25, with higher scores indicating better performance [15]. Each simulation trial was reviewed by 30–34 crowdsourced workers recruited and trained by C-SATS. An aggregated and averaged GEARS score was generated for each simulation trial using a linear mixed-effect model, which weights scores based on the reviewer experience. These final scores were reported in our study.

Results

All 11 participants completed the study. 10/11 participants had previously completed da Vinci® Surgery online training modules or logged hours on the da Vinci® VR Simulator. Level of experience with robotic-assisted dismembered pyeloplasty ranged from observing 1–4 procedures being performed (PGY-3s) to having performed the majority of a case five or more times (attending surgeons). No participants had prior experience with 3D-printed models for surgical simulation.

All participants performed a complete simulated robotic-assisted dismembered pyeloplasty on two separate occasions with patent anastomoses. In 14/22 of the trials, no leakage was noted from the anastomosis. The average time to complete the first trial was 44.4 min, and the average time to complete the second trial was 43.2 min (p = 0.44). Eight participants were able to improve their time on the second attempt (Fig. 4). Junior residents (PGY-3) took an average of 58.1 min, senior residents (PGY-4/5) 45.6 min, and attendings 22.4 min. Attendings completed the task significantly faster than junior and senior residents (p = 0.005 and p = 0.013 respectively). Senior residents were not significantly faster than junior residents (p = 0.082). The correlation between the level of training and time to complete the first and second trials was − 0.91 and − 0.79, respectively.

Fig. 4
figure 4

Total time to complete simulation for each participant ordered by level of training

Full size image

Prior to the simulation, participants including both residents and attendings reported an average confidence in performing the procedure of 2.18 on a 5-point Likert scale. The average confidence after one and two simulations was 2.55 and 2.82, respectively (p = 0.09 for the difference in initial and final confidence by t test). Among only residents, a significant increase in confidence was seen between the initial and final surveys: 1.63 vs 2.38 (p = 0.03).

C-SATS scoring based on video review of the simulation trials showed an average GEARS score of 17.1 for the first trial and 17.6 for the second trial (p = 0.13) (Fig. 5). On average, junior residents scored 16.7, senior residents 17.0, and attendings 18.5. There was not a significant difference between junior residents and attendings (p = 0.066). The correlation between the level of training and initial and final GEARS scores was 0.37 and 0.60, respectively. Participants had higher average scores on the second trial in all five categories, with a significant increase seen in depth perception (p = 0.006) (Fig. 6). The correlation between the level of training and improvement in GEARS score was 0.45. Senior residents showed the greatest improvement in score: 16.5 vs 17.6 (p = 0.09).

Fig. 5
figure 5

GEARS score for each participant for each simulation ordered by level of training

Full size image
Fig. 6
figure 6

Average GEARS scores (with SD) by category. There was a significant increase in score for depth perception between the two trials (p = 0.006)

Full size image

On a 5-point Likert scale, the model received average scores of 4.36 ± 0.50 (mean ± SD) for aesthetics, 4.18 ± 0.40 for overall Feel, 3.55 ± 0.69 for realism, 4.72 ± 0.47 for usability, and 4.72 ± 0.47 for suturability.

Discussion

We developed a 3D-printed model for robotic surgical training and used an established crowdsourced scoring system to assess simulation performance and learning outcomes. To our knowledge, this is the first study that has used 3D-printed surgical simulation and crowdsourced scoring assessments to objectively assess trainee robotic surgical education.

In our cohort of 11 participants, we found that all participants were able to perform a complete robotic-assisted dismembered pyeloplasty on two separate occasions and there was a decrease in overall time to perform the procedure between the first and second trials. Participants rated the models as highly usable and realistic. Overall, participants had increased confidence in their ability to perform the procedure by the end of the simulation, with residents demonstrating a statistically significant increase in confidence from baseline. Using crowdsourced scoring to track performance, there was a trend towards an increase in overall GEARS score, however, this was not statistically significant. When the GEARS score was subdivided by category, there was a mean improvement in each category and depth perception was significantly improved. This improvement across all GEARS categories and time to perform the procedure demonstrates concurrent validity. There was a significant difference in time to complete the task between attendings and junior residents, and there was a strong correlation between time to complete the task and level of training. There was a moderate correlation between performance by GEARS criteria and level of training. This relationship between performance and level of training demonstrates construct validity of the simulation as it was able to reliably differentiate novices from experts. In addition, we noted that senior residents had the highest improvement in performance overall, suggesting that there may be some value in having prior robotic experience.

Several prior studies have shown the usefulness of virtual reality simulation for robotic surgical training. A systematic review by Moglia et al. [7] of current simulators, including the da Vinci® Skills Simulator, dV-Trainer, and RoSS simulator, rated well for face and content validity. Despite being able to develop skills moving inanimate objects with robotic instruments (peg transfers, ring walks, etc.), the main limitation of this type of virtual training environment is that they have no resemblance to surgical procedures. Cheung et al. [8] performed a similar study to our own, using silicone-based 3D printing to create a model for simulating pediatric laparoscopic pyeloplasty. The model was secured in a laparoscopic box-trainer and 27 participants performed a right laparoscopic pyeloplasty. The model received good scores for face validity, including handling, usefulness, and aesthetics. No data were collected on the performance of participants.

Maddox et al. [11] constructed 3D-printed spongy agarose gel-based models of kidneys from CT imaging using a method similar to our own. These were modeled after renal units of seven patients with suspected malignancies. Partial nephrectomy and renorrhaphy were performed on the models for surgical rehearsal prior to live robotic partial nephrectomy. They prospectively compared the outcomes of these seven patients to those for whom no model was constructed and found those cases with models had significantly lower estimated blood loss, and a trend towards larger tumor size, higher nephrometry score, longer warm ischemia time, fewer positive surgical margins, shorter hospitalization, and fewer postoperative complications. No data were collected on the face validity of the model or the learning outcomes of participants.

Our study expands upon the current literature by demonstrating the face validity of a silicone-based 3D-printed model, which was found to improve participant speed and significantly improved resident confidence with the procedure. In addition, we implemented crowdsourced assessment methods to quantitatively show that this model improved resident robotic surgical performance, with significant improvement in depth perception.

This study has several limitations. First, this feasibility study is limited by the small number of participants and a small number of trials per participant, due to the cost of the model and infrequent robot availability. Ideally, we would have recruited a large number of residents with varying levels of training and followed them longitudinally, however, given financial constraints was not this feasible. Although the small sample size limits the generalizability and strength of our conclusions, we found in our pilot study that 3D-printed models can be used as an adjunct to VR simulations in a training program. As with other surgical procedures, robotic pyeloplasty has a learning curve and this study has likely captured performance early in the learning curve of many participants, and may, therefore, underestimate the benefit of this model. In addition, this study also had unavoidable intrinsic limitations of the model. This included cost limitations requiring the model to be used twice per participant. Given that we were only comparing the time to perform the anastomosis, this was unlikely to have much of an effect on outcomes. Finally, our study only looked attechnical performance on a stationary inanimate model and therefore no conclusions can be drawn about how this may ultimately translate to patient-related outcomes.

There are many future areas of study. Immediate goals include improving the next iterations of this simulation based on the data and participant feedback. Another priority is to increase the availability of these models, by lowering cost through cheaper production and by making models multi-use. For example, models can be developed that have a dilated renal pelvis, a crossing vessel, and a parenchymal tumor, to allow it to be reused for multiple simulated procedures (i.e. pyeloplasty and partial nephrectomy). In addition, securing dedicated robotic simulation facilities will allow greater access, experience and performance improvement. Future studies are currently being aimed at evaluating the benefit of surgical rehearsal with patient-specific models by comparing simulation performance to actual surgical performance.

Surgical education has always struggled with the concurrent goals of providing residents with thorough training experience, while also balancing the need for patient safety and excellent patient outcomes. This is particularly true for uncommonly encountered pathology. We hope to integrate simulation using 3D-printed models into a standard residency curriculum as a complement to live and virtual reality surgical training to address educational gaps moving forward and improve the training of our future residents.

Conclusions

Using 3D-printed silicone-based models, participants were able to perform a complete robotic-assisted dismembered pyeloplasty for training and skill acquisition. We demonstrated the face validity of simulation using a silicone-based 3D-printed model, which was found to improve participant speed and significantly improved resident confidence with this procedure. Crowdsourced assessment demonstrated that this model significantly improved depth perception. This is the first study that has used 3D-printed surgical simulation and crowdsourced scoring assessments to objectively assess trainee robotic surgical education. This feasibility study indicates that 3D-printed models and Crowdsourced assessments show promise as methods to address current surgical educational gaps.