Introduction

Recently, there has been a growth in interest in laparoscopic rectal cancer resection. Whilst some large studies have shown that equivalent outcomes to open surgery are achievable [1, 2], two recent large multicentre randomised controlled trials (RCTs) failed to demonstrate non-inferiority to open surgery in terms of the quality of surgical resection [3, 4].

Laparoscopic rectal surgery is technically demanding, with the narrow and deep pelvic anatomy representing a substantial challenge for the surgeon. Proficiency gain for laparoscopic mesorectal dissection has been demonstrated to be more protracted than that for other task components of laparoscopic colorectal surgery [5]. Given that oncological outcomes following rectal cancer resection are substantially affected by the quality of surgery [6–8], there is a risk that any increase in uptake of laparoscopic rectal resection, if performed by insufficiently trained surgeons, could have a major impact on surgical and oncological outcomes. A detailed assessment of operative task performance underpins the development of training and assessment in this field; however, at present no objective assessment methodology has been described for laparoscopic rectal cancer resection surgery [9]. To undertake such a systematic analysis of a complex surgical procedure, it is necessary to first define an optimal description of the technique against which deviations, or errors in technical performance, can be identified and categorised using a structured methodology [10].

Human reliability analysis is a technique that was developed for evaluating the interface between humans and complex systems, whereby the system is described in terms of its constituent tasks, and each task is analysed to identify potential errors and their consequences in a prospective manner. This technique has been modified for the detection and categorisation of technical errors and near misses enacted within surgical procedures, including laparoscopic cholecystectomy, colectomy, and pyloromyotomy [11–13]. The insight provided by such a detailed evaluation, termed objective clinical human reliability analysis (OCHRA), can help to highlight technical challenges, particularly with technically advanced surgery. The use of OCHRA to assess laparoscopic rectal cancer surgery has never been evaluated.

The aim of this exploratory study was to evaluate the validity and reliability of using the OCHRA technique for evaluation of technical performance of laparoscopic rectal surgery.

Materials and methods

Hierarchical task analysis

To describe laparoscopic TME in an ergonomic format, a hierarchical task analysis was created, which describes the procedure in sequential tasks, which are each completed by the surgeon to perform the procedure. This was based upon a recently published consensus document describing the task areas of laparoscopic TME [14]. For the purposes of analysis, these tasks were categorised into “abdominal” tasks of the procedure (exposure of operating field; vascular pedicle division, colon mobilisation; splenic flexure mobilisation) and “pelvic” tasks (posterior mesorectal dissection; anterior mesorectal dissection; lateral mesorectal dissection; TME completion; rectal division and anastomosis). To allow assessment of laparoscopic APE procedures, a separate task analysis was created to describe the perineal parts of this operation based upon the description used for formative assessment in the English “LOREC” low rectal cancer Nation Development Programme [15] and Holm’s widely cited description [16]. This replaced the last two pelvic tasks from the hierarchical task analysis of laparoscopic TME; otherwise, the tasks of the two procedures were identical.

Error analysis

Potential technical error events in laparoscopic rectal surgery were defined and categorised based upon previous descriptions of OCHRA and the interviews described above [12, 17]. For this study, a consequential error was defined as “any action (or omission) that resulted in a negative consequence (e.g. bleeding, injury to mesorectum or hypogastric nerves), or increased the operating time of the procedure through necessitating corrective action”. A non-consequential error was defined as “any action (or omission) that increased the likelihood of negative consequence and under different circumstances could have had a consequential effect”.

It was apparent from the interviews that some variation exists in the order that surgeons perform the component tasks of laparoscopic rectal resection, and so unlike previous descriptions of OCHRA [11, 18, 19], deviations in the sequence in which tasks performed were not logged as errors. Only execution errors were considered for this study.

To explore the spatial distribution of errors enacted during pelvic dissection, the location that errors identified during these tasks occurred was recorded in one of seven zones. A clock face was initially trialled for this; however, this was found to be inconsistent, and therefore, division of the mesorectum into lateral, anterior and posterior zones on the right and left side of the pelvis was identified as to be a more reliable approach. For posterior mesorectal dissection in the midline, it was difficult to consistently identify a border between left and right; therefore, the midline sulcus formed between the mesorectal buttocks was designated a seventh zone for description of error location.

OCHRA testing

The OCHRA methodology was applied through evaluation of video recordings from a series of 20 consecutive laparoscopic resection operations performed for mid or low rectal cancer at Yeovil District Hospital, Yeovil, England between October 2012 and November 2013. Operations were performed using a standard operative technique, with medial-to-lateral mobilisation of the sigmoid colon, division of the vascular pedicle, selective splenic flexure mobilisation, and then posterior entry into the mesorectal dissection plane. Tumours were treated by laparoscopic TME or extralevator abdominoperineal excision (ELAPE) depending upon the tumour location.

Laparoscopic parts of the procedure were recorded from the laparoscopic theatre stack using a MediCapture 300 HD video recorder (MediCapture Inc. Philadelphia, USA). The perineal dissection of ELAPE was recorded using a sterile laparoscope, but other extracorporeal steps of the procedure were not recorded.

Data extraction

Videos of the operations were viewed and analysed by the research fellow, and all identified errors logged for each procedure. The procedural task, spatial location within the pelvis, and circumstances of all error events were logged. To facilitate interpretation, errors were categorised as either errors of retraction and errors of dissection based upon the perceived principal mechanism for the error occurring. Each error was also assigned a code describing any consequence that resulted.

Operative and post-operative clinical outcome measures were collected for 30 days following surgery by the clinical team. To reduce risk of bias, the research fellow performing the analysis of the videos was blinded to outcome data until after analysis of the operative videos was completed. The duration of the operation was defined as from the time of incision to wound closure. Blood loss was estimated from the weight of surgical swabs and blood within the suction apparatus. The plane of dissection of the resected specimen was independently evaluated by a consultant histopathologist using a three-point ordinal scale according to standard reporting guidelines [20].

Statistical analysis

The frequency of total and consequential errors was calculated for each operation. Error frequencies in different regions were compared using the Wilcoxon signed-rank test and Friedman two-way analysis of variance as appropriate. Reliability was assessed through calculating Pearson’s phi for test–retest correlation of error frequencies for the first ten cases, which were assessed on two separate occasions, 6 months apart. Clinical validity of OCHRA for assessment of technical performance was tested through correlation between frequency of errors with the pathological quality of mesorectum, blood loss, and total operating time by calculating Spearman’s Rho.

As this was an exploratory analysis, no formal power calculation was performed; however, to identify trends in the data, two-tailed tests of statistical significance were used with p < 0.05 considered to signify a statistically significant result.

Results

Twenty operation video recordings were analysed with OCHRA. Fifteen of the patients were male, median age was 70 years (range 24–8 years), and median body mass index was 26 kg/m2 (range 22–40 kg/m2). Thirteen patients underwent laparoscopic TME, and seven underwent laparoscopic ELAPE.

Error analysis

A total of 335 execution errors were observed during the 20 operations, with 299 of these having a directly observed consequence. Median number of errors per case was 15. The most common error mechanisms described were “dissection in wrong tissue plane” (n = 107), “too much blunt force applied to tissue” (n = 48), and “traction applied in wrong direction” (n = 43) (Table 1). Two hundred and ninety-nine errors had directly observed consequences. The most common consequences were “mesorectal injury into fat” (n = 91), “mesorectal fascia injury” (n = 76), and “bleeding” (n = 86) (Table 2). No significant difference was observed between individual surgeons in the number of errors per case (p = 0.24).

Table 1 Categories and frequency of observed execution errors (total n = 335)
Full size table
Table 2 Categories and frequency of consequences of observed consequential errors (total n = 299)
Full size table

Error distribution

Significantly more errors occurred per operation during pelvis tasks (median 12 errors, range 6–26 errors) compared with abdominal tasks (median 5 errors, range 0–9 errors) (p < 0.001). For errors enacted during mesorectal dissection, significantly more errors occurred on the right side of the pelvis (n = 118) compared to the left side (n = 92) (p = 0.03). Specifically, the right posterior mesorectal dissection was the area with most error events observed (n = 52), and there were significantly more errors recorded here than the left posterior mesorectal dissection (n = 20) (p < 0.001) (Figs. 1, 2). However, more errors were recorded in the left anterior sector (n = 32) compared with the right anterior sector (n = 17, p = 0.04) (Figs. 1, 2).

Fig. 1
figure 1

Magnetic resonance imaging scan of mesorectum demonstrating the total number of errors observed in each sector of the pelvis during mesorectal dissection of the 20 cases

Full size image
Fig. 2
figure 2

Box plot showing spatial distribution of error events observed during pelvic mesorectal dissection

Full size image

Reliability and clinical validity for assessment of performance

A strong correlation was observed between error frequencies upon test–retest analysis of the first ten cases (r = 0.97, p < 0.001) (Fig. 3). There was a moderate but significant correlation between plane of dissection reported by the histopathologist and total error frequencies (r s = −0.51, p = 0.02). Frequency of errors identified during pelvis phases of the operation correlated with plane of dissection (r s = −0.54, p = 0.02); however, error frequencies during abdominal phases did not (r s = −0.26, p = 0.27). Total error frequency per case did show a statistically significant correlation with total blood loss (r s = 0.61, p = 0.004). However, error frequencies did not correlate with total operating time (r s = 0.27, p = 0.27).

Fig. 3
figure 3

Scatter graph with line of best fit showing test–retest reliability correlation between observed error frequencies of first ten cases when analysed on two occasions 6 months apart

Full size image

Discussion

This prospective exploratory study is the first to report on application of an objective methodology for identification of technical errors enacted in laparoscopic rectal cancer resection surgery. It has shown that the technique can be successfully used to evaluate technical performance of these complex procedures with a degree of validity and reliability. This can be a valuable academic tool for accreditation of surgeons prior to taking part in randomised controlled trial investigating laparoscopic rectal surgery.

The grade of dissection plane as assessed by an expert histopathologist using standardised objective criteria has been shown to correlate with local recurrence rates following open surgery [6, 8]. The significant correlation between OCHRA error frequencies and the TME quality scored reported here supports the clinical validity of this technique for assessment of technical performance of laparoscopic rectal cancer resection. The current study was, however, an exploratory project with a small sample size; hence, no attempt was made to examine the predictive validity of the enacted errors for post-operative complications, as this would likely require a much larger sample size.

Whilst audit of clinical outcomes and pathological evaluation of resection specimens can provide a retrospective indication of surgical performance, such surrogates cannot inform about the technical factors that underlie the outcomes. Additionally, such a retrospective approach requires a significant number of cases before negative outcomes can be observed. The objective assessment methodology used in this study allows direct identification and quantification of technical errors enacted during an individual laparoscopic rectal cancer operation. OCHRA can hence provide feedback about the mechanisms and circumstances that may have resulted in the poor outcomes or specimen.

Nearly a third of all observed errors in this study were related to dissection in the wrong plane in the pelvis. This highlights the difficulty and importance of achieving the correct plane of dissection in rectal cancer surgery. In comparison, a study using OCHRA for laparoscopic colon surgery identified excessive blunt force and poor visualisation of instruments as the most common error codes [12].

Since a non-selected, consecutive cohort of patients were evaluated here, the errors observed may be considered a true reflection of those difficulties encountered by surgeons when performing such surgery. The surgeons who performed the operations in this study are all experienced laparoscopic colorectal surgeons—this is supported by the low rates of errors (median 5) observed within the abdominal tasks of the procedure which is similar to frequencies amongst an “expert” group (median 4) in a previous description of OCHRA applied to selected cases of laparoscopic colonic surgery [12]. The variation between cases in observed error frequencies demonstrates the potential of OCHRA to detect differences in technical performance at the specialist level. This contrasts with other assessment methodologies that have been developed mainly for evaluation of trainees’ basic skill acquisition and displays a “ceiling effect” when applied to specialists [21].

In this exploratory study, the higher frequencies of errors were observed on the right side of the pelvis merits consideration and further investigation. One possible explanation could be attributed to the ergonomic set-up of performing the operation. The majority of surgeons perform laparoscopic mesorectal dissection using a dissecting instrument held with the right hand whilst stood on the patient’s right side. From this position, the manipulation angle between the surgeon’s operating and retracting instruments is narrower when operating on the right side of the pelvis compared to the left and hence increasing the difficulty of task performance, which is consistent with the findings of previous research investigating the optimum manipulation angle for laparoscopic task performance [22]. Further research is required to investigate this and to identify the optimum ergonomics to overcome any limitation in the laparoscopic dissection of the right side off the pelvis. Technological developments may also help address these limitations; for instance, the improved depth perception offered three-dimensional laparoscopic display units and the greater degrees of freedom available with robotic operating platforms.

There are limitations to the OCHRA methodology described here. Substantial time is needed to analyse the operative videos, and assessors require specialist training and practise in the technique. However, this technique could be used as an academic/research tool to objectively evaluate surgeons’ competence when they enter randomised controlled trials. Whilst we have demonstrated excellent test–retest reliability of OCHRA, implying that the rater applies a consistent definition of what constitutes an error, only the test–retest reliability has been evaluated here. Further work is needed to investigate inter-observer error frequencies. It is also possible that the errors observed in this single-centre study may not reflect wider practice. However, all surgeons who contributed to this study have performed more than 50 laparoscopic rectal resections, and the operative technique that they use does not differ from a recent international Delphi consensus study describing technical steps of laparoscopic TME [14].

The detailed evaluation of technical performance of laparoscopic rectal surgery in this study opens up the potential application of the OCHRA technique for research in the field of minimally invasive rectal cancer resection. With increased uptake in laparoscopic rectal cancer surgery, there has also been a growing interest in researching interventions and technologies that can reduce the technical complexity of such surgery to improve outcomes for patients [23]. The variation in error frequencies observed even amongst expert surgeons suits OCHRA to use in trials assessing the impact of interventions upon the technical complexity of a surgical procedure. However, a larger study is required to further evaluate the correlation of error frequencies measured by OCHRA with post-operative outcomes, and the impact that patient and tumour factors have upon error frequencies.

In conclusion, this study has shown that use of the OCHRA technique for evaluating technical performance of laparoscopic rectal surgery is feasible with a certain degree of reliability and validity. The distribution of errors observed in this study supports that the pelvic tasks of the operation are more complex to perform than the abdominal tasks and that particular attention may need to be focused upon reducing errors on the right side during dissection in the pelvis. Future work should investigate the wider potential applications of OCHRA in laparoscopic rectal cancer surgery.