Introduction

In the Netherlands, approximately 2500 radical prostatectomies are performed annually of which 90% are performed using the surgical robot, i.e. robot-assisted radical prostatectomy (RARP). The RARP is a complex but highly standardized operation to cure local prostate cancer. However, RARP is hampered by serious side-effects [1,2,3] such as urinary incontinence, which occurs in 4 to 26% of the patients [4,5,6], and erectile dysfunction, which occurs in 14 to 90% of the patients [7, 8].

Previous research has shown that greater surgical experience is associated with better postoperative outcomes [9,10,11]. Therefore, the Dutch Society of Urology (NVU) increased the minimally required number of annual RARP per hospital from 50 to 100 procedures to improve functional results and reduce complications. However, at the moment there is no minimum annual number of procedures per surgeon.

Various authors suggested that systematic evaluation of skills, both technical (surgical) and non-technical (communication and teamwork) may be more effective in improving the surgeons’ skills than a quota alone [12, 13]. Thorough analysis of surgical videos can possibly elucidate which steps or facets of surgery may be related to disadvantageous results such as postoperative complications (i.e. bleeding and leakage of the vesico-urethral anastomosis) and adverse functional outcomes (i.e. erectile dysfunction, incontinence) [12, 14].

To standardize video analysis, a detailed description of all the separate surgical steps is needed. In the past, different assessment instruments containing individual steps of the RARP have been defined [15,16,17], but these methods are mostly intended for providing feedback during training of new robotic surgeons or to evaluate the skills of more experienced robotic surgeons by means of video analysis. So far, no specific method has been developed to investigate how a surgeon’s skills and surgical events as assessed on video are related to adverse postoperative outcomes of RARP.

The present Delphi study is designed to evaluate whether experts in the field of RARP can identify the surgical and non-surgical factors in RARP that are potentially associated with negative aspects of postoperative outcomes.

The following key questions were to be answered: which steps of the RARP and which peri-operative events (i.e. bleeding, usage of coagulation, usage of haemostatic clips and suturing) are most likely associated with postoperative complications (i.e. bleeding and leakage of the vesico-urethral anastomosis) and adverse functional outcomes (i.e. erectile dysfunction, incontinence)? How can these steps of the RARP and these peri-operative events be incorporated in an RARP assessment instrument?

Methods

During a focus group consisting of three Dutch urologists, one urologist in training, and one cognitive task analysis expert a list of statements was created, describing the surgical steps and possible peri-operative events of the RARP procedure as well as their possible association with (1) direct postoperative complications and (2) functional outcomes. These statements were formulated in order to investigate which steps of the surgery and which peri-operative events should be included in an instrument for video analysis. This assessment instrument will form the basis for further research on the possible associations between surgical skills and adverse postoperative outcomes.

Expert panel

The expert panel for the Delphi study was selected based on recommendations of three separate independent urologists who are experts in the field of robotic surgery. Based on these recommendations, 18 experts in the field of robot-assisted radical prostatectomy were selected. In this group, multiple proctors and educators of different fellowships in robotic surgery were included since they have intricate knowledge of the possible origins of complications in surgeons with all levels of experience. The experts were invited by e-mail. If no response was given the experts were contacted by telephone to ask whether they were interested to participate in the Delphi study.

Consensus procedure

To achieve consensus, a two-step procedure was used (Appendix Fig. 1): the first step was an online two-round Delphi Survey involving Dutch urologists experienced in RARP. The second step was a consensus group meeting with the same Dutch urologists to discuss the results of the online survey and to identify the aspects of the surgery and the peri-operative events which might be associated with postoperative adverse outcomes. The steps of the Delphi process are based on protocols for consensus finding [18,19,20,21].

Online two-round Delphi Survey

The results of the initial focus group were used to define seven domains in which the statements could be categorized. The domains were organized as follows: (Tables 1, 2).

Table 1 Domains of statements used in the online Delphi Survey and consensus meeting
Full size table
Table 2 Results of the first and second online Delphi Survey and combined consensus results after Delphi meeting
Full size table
  1. 1.

    The relation of the statement to postoperative complications;

  2. 2.

    The relation of the statement to functional results;

  3. 3.

    Surgical steps associated with complications (i.e. bleeding and leakage of the vesico-urethral anastomosis);

  4. 4.

    Surgical steps associated with postoperative erectile dysfunction;

  5. 5.

    Surgical steps associated with postoperative urinary incontinence;

  6. 6.

    Factors that play a role in the origins of postoperative complications;

  7. 7.

    Elements that should certainly be included in the training of novice surgeons.

The statements were used to design a two-round online Delphi Survey to obtain consensus on identifying the relevant steps of the RARP procedure and their possible causal relation to postoperative complications and adverse functional outcomes.

The panel members were asked to rate the relevance of each statement using a 9-point Likert scale according to the discriminatory power of each surgical step to correspond with the specified postoperative complication. A rating of 1 was defined as “extremely disagree” and a rating of 9 was defined as “extremely agree”. As described in the RAND/UCLA Appropriateness Method [22], for each item, the median agreement score, lower limit interpercentile range (IPR), and upper limit IPR and Disagreement Index (DI) were calculated. A median agreement score of 1.0–3.0 was considered to be “disagree”, 3.1–6.9 as “uncertain”, and 7.0–9.0 as “agree”. A DI value above one (> 1) indicated a lack of consensus among the participants regarding the association between the statement and the postoperative complication.

In addition to the consensus statements, seven general questions (Table 3) were included in the first online survey to assess the experts opinions on the project and their willingness to cooperate in further research. In the first round, the participants were invited to suggest additional items that should be included in the second-round survey. The second survey consisted of the consensus statements. After each round, the scores for each item were anonymized to a mean ranking score for the whole group and reported back to the participants.

Table 3 The participants and their institute, occupation and voting status
Full size table

Consensus group meeting

During a consensus panel meeting, the statements on which consensus had been reached in the two-round online survey were reviewed and statements on which no consensus had been reached were discussed and voted on. The meeting was chaired by a urologist of the Dutch Cancer Institute, Amsterdam (HvdP).

The statements from the online survey were presented to the panel, and participants were asked to motivate their opinions on each of the statements for which no consensus had been reached previously. The list of approved steps and aspects was then categorized to develop an initial RARP assessment instrument for evaluating the surgical procedure on video. This assessment instrument was subsequently judged on face validity by the 12 experts who participated in the Delphi process.

Informed consent

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2013 [23]. Informed consent was obtained from all participants for being included in the study.

Results

Delphi Survey

The results of the preliminary focus group meeting were used to formulate 72 statements on surgical steps and possible peri-operative events of the RARP procedure and their possible association with (1) direct postoperative complications and (2) functional outcomes. These statements were divided over seven domains and incorporated in an online survey (Table 1).

A total of 18 Dutch experts in robot-assisted radical prostatectomy (RARP) were identified and invited to participate in the two-round online Delphi Survey. In the first round, 12 of the 18 (67%) invited experts participated in the survey. Of these 12 participants, 10 experts responded to all statements, and two participants reported difficulties with the survey resulting in a partial response to the statements.

In the first round, participants did not propose any additional statements. Of the 72 statements reviewed in the first round, 18 statements on which a clear consensus had been reached (i.e. a median agreement score of 1 or 9) were excluded from the second round. The remaining 54 statements were incorporated in the second online survey round.

In the second round, nine of the 12 participants of the first round participated in the survey. Of these nine participants, eight completed the survey and one reported difficulties with the survey resulting in a partial response to the questionnaire.

General questions

Results of the general questionnaire (Table 3) show that 83.0% of the experts believe that patient outcome can be improved by analysis of critical surgical factors. According to 75.0% and 83.3% of the experts who participated in this study, video assessment is suitable for predicting complications and functional patient outcomes, respectively. According to 92.0% of the experts, the use of video assessment could reduce the risk of complications. All experts were interested in participating in the analysis of surgical videos. Most experts had the means to record surgical videos (80.0%) and can link these videos to surgical data (90.1%).

Consensus group meeting

Of the 18 invited experts, five participated in the consensus group meeting. In total, this meeting was attended by nine participants, whose occupation and voting status are presented in Table 4.

Table 4 Results of general questions about video analysis, registration of postoperative outcomes, and intention to participate in the analysis of RARP videos
Full size table

Final consensus statements

Table 2 shows the statements on which consensus was reached, organized per domain. The results of the Delphi Survey and the consensus group meeting were used to develop the assessment instrument PROTEST (PRostatectomy video Observation to Evaluate and Score Technical skill) (Appendix Table 5). This instrument contains the seven steps of the RARP surgery and the perioperative measurements that are considered to be most likely to be related to complications and adverse postoperative outcomes.

The relation between the statements and postoperative complications

Consensus of ‘agreement’ was reached on three out of 14 statements (Table 2) regarding the relation of the statement to postoperative complications. Consensus of ‘disagreement’ was reached on six out of 14 statements, one of which received a unanimous ‘disagreement’ score (i.e. median score of 1 and disagreement index = 0). No consensus was reached on the five remaining statements. The panel agreed on the following statements: “Shorter operating times result in fewer complications” and “Shorter duration of coagulation results in fewer complications”. All participants disagreed with the statement “It is better not to use the third arm of the robot when looking at complications”.

The relation between the statements and functional results

The participants of the Delphi Survey reached consensus of ‘agreement’ on two out of 14 statements concerning functional results (Table 2). The panel reached a consensus of ‘disagreement’ on five out of 14 statements. None of the statements received a unanimous score. No consensus was reached on the seven remaining statements. The panel agreed on the following statements: “Zooming in more on the operation field provides better vision as it leads to better functional results.” and “A shorter duration of coagulation results in improved functional results.”

Steps of the surgery associated with complications

During the Delphi Survey, consensus of ‘agreement’ was reached on seven out of 11 statements regarding the steps of the surgery that might be associated with complications. The panel reached consensus of ‘disagreement’ on two out of 11 statements (Table 2). No statements received a unanimous score. No consensus was reached on two remaining statements.

The panel agreed that the following steps of the surgery might be associated with complications: “Bladder neck dissection”, “Ligation of prostate pedicles”, “Nerve preservation”, “Management of prostate apex/urethra”, “Vesico-urethral anastomosis”, “Lymph node dissection”, and “Wound closure and specimen removal”.

Steps of the surgery associated with postoperative erectile dysfunction

The Delphi Survey panel reached a consensus of ‘agreement’ on four out of 11 statements regarding the steps of the surgery that might be associated with postoperative erectile dysfunction (Table 2). The experts unanimously agreed that “Nerve preservation” and “Management of prostate apex/urethra” might be associated with the incidence of postoperative erectile dysfunction. A consensus of ‘disagreement’ was reached on five out of 11 statements, four of which received a unanimous ‘disagreement’ score. No consensus was reached on the two remaining statements.

Steps of the surgery associated with postoperative urinary incontinence

During the Delphi Survey, the panel reached a consensus of ‘agreement’ on two out of 11 statements (Table 2) regarding steps of the surgery that might be related to postoperative urinary incontinence. The panel reached a consensus of ‘disagreement’ on five out of 11 of these statements. No consensus was reached on the four remaining statements. A unanimous consensus of ‘agreement’ was reached on one of these steps, and a unanimous consensus of disagreement was reached on three of these steps.

The panel agreed that the following steps of the surgery might be associated with postoperative urinary incontinence: “Pelvic floor muscle exposure/opening of the endopelvic fascia”, and “Urethrovesical anastomosis”.

Factors that play a role in the origins of postoperative complications

Both the Delphi Survey and the consensus meeting reached a consensus of ‘agreement’ that all (four out of four) the proposed factors (Table 2) could play a role in the origins of postoperative complications. The experts unanimously agreed on the relevance of the following factors: “Teamwork”, “Communication between the surgeon and the surgical team”, and “Surgical skills of the surgeon”.

Elements that are essential for the training of novice surgeons

The participants of both the Delphi Survey and the consensus meeting reached a consensus of ‘agreement’ that all (seven out of seven) proposed elements of training (Table 2) are essential for the training of novice surgeons. There was unanimous agreement on the need to implement the following training assessment methods: “Supervised practice on real patients”, and “Fellowship”.

Discussion

The aim of this study was to develop an assessment instrument for the evaluation of surgical videos to elucidate the association between surgical skills and postoperative outcomes after a robot-assisted radical prostatectomy (RARP). To investigate this association, we invited all Dutch experts in RARP to participate in a standardized Delphi procedure in order to identify surgical and non-surgical factors in RARP that are potentially associated with an adverse postoperative course and to assess whether any of these parameters may be worth evaluating for the prediction of postoperative outcomes.

We found that the majority of Dutch urologists specialized in RARP are interested in an instrument for video analysis of their surgical skills in relation to the postoperative outcomes. These urologists indicated that they were interested to participate in the current study because they considered video analysis to be useful for the improvement of surgical skills and the subsequent reduction of postoperative complications.

Consensus group meeting

During the consensus group meeting the panel members agreed that the duration of the surgical procedure and the duration of coagulation could be causally related to the rate of postoperative complications. They advised to investigate whether such a causal relation exists. The panel also agreed there could be a causal relation between the duration of coagulation and the level of adverse postoperative functional results.

The panel agreed that postoperative complications, postoperative erectile dysfunction and postoperative urinary incontinence could result from events during the following steps of the RARP procedure: “Pelvic floor muscle exposure/opening of the endopelvic fascia”, “Bladder neck dissection”, “Ligation of prostate pedicles”, “Nerve preservation”, “Management of prostate apex/urethra”, “Urethrovesical anastomosis”, “Lymph node dissection”, and “Wound closure and specimen removal”. The panel agreed that these steps should be incorporated in an assessment instrument to investigate whether they are related to adverse postoperative outcomes.

PROTEST assessment instrument

Based on the consensus reached during the Delphi Survey and the consensus meeting, the PROTEST assessment instrument was developed (Appendix 2). This instrument can be used to assess the skills of a surgeon through analysis of a video recording of the surgery.

This assessment instrument was developed with the input of the panel members and was reviewed by all panel members in order to give them the opportunity to refine and clarify the assessment instrument. One item, “Wound closure and specimen removal”, was not included in the PROTEST assessment instrument, because this step is not recorded on surgical videos.

When comparing the results of the current Delphi study to existing assessment instruments [15, 24, 25], the developed PROTEST assessment instrument shows a combination of subjective surgical skills assessment and objective metrics of procedural steps and events. The developed PROTEST assessment instrument is different from the GEARS assessment instrument where the focus lies solely on the subjective scoring of 5 technical domains of surgical skill, with no objective measurements [23]. The GERT assessment instrument comparable to the PROTEST assessment instrument as it focusses on different features of the surgery (i.e. clipping, suturing, use of the retractors and use of suction), but it only gives feedback on possible errors whilst performing this features, there is no room to score subjective surgical skill [24]. The PACE assessment instrument is similar to the GERT assessment instrument as it evaluates specific steps of the RARP procedure, similar to the PROTEST assessment instrument, but only gives feedback based on errors whilst performing these steps and there is no room for subjective surgical skill analysis [15].

Implications of study findings for clinical practice and research

This Delphi procedure resulted in an overview of possible origins of complications after RARP and in a new assessment instrument that can be used to objectively assess a surgeon’s skills.

The PROTEST assessment instrument gives detailed insight into the proficiency of the surgeon on each of the individual surgical steps of the RARP. It combines the answers to two general subjective questions with multiple objective measurements in order to provide detailed feedback to the surgeon.

Future studies should explore whether the factors identified in this Delphi process are indeed causally related to postoperative complications and whether video assessments by means of the PROTEST instrument can help in the training of novice surgeons and improving the skills of RARP surgeons.

Limitations

A limitation of this study is that we consulted all the urologists specialized in RARP who are registered in the Netherlands. Future studies with larger panels and international participants might add other factors that could contribute to complications after RARP. A second limitation is that the answers to the general questions might be subject to participation bias and hence they cannot be generalized to the total group of Dutch urologists. Finally, the responses of panel members could have been influenced by the fact that the consensus meeting was not led by an independent chair.

Conclusion

Dutch experts in the field of RARP have reached consensus on seven surgical steps and four aspects of the RARP procedure that may be related to postoperative complications after RARP. The resulting assessment instrument, PROTEST, can be used to assess surgical skill. The resulting assessment instrument will be tested in future research to determine its validity for assessing the relationship between surgical skills and adverse postoperative outcomes after RARP.