Introduction

A significant portion of complications in hospitalized patients occurs following a surgical procedure with both recognized and under-recognized events playing a role [1, 2]. As such, there is interest putting in place measures to identify, remediate, and prevent these intraoperative adverse events (IAE) [1]. To achieve this goal, safety systems have evolved over time into clinical practice that include maintenance of a sterile field, instrument counting, as well as “time outs” [3]. Shortcomings of these systems are inherent to an inability to continuously surveil the entirety of an operating room. Outcomes scales, such as those by Clavien et al [2], focus on post-operative patient outcomes and retrospective chart reviews [3]. These analyzes also fail to directly correlate intraoperative adverse events with key clinical outcomes.

As the recognition of intraoperative adverse events become more critical to troubleshoot quality systems, near-miss identification has emerged as critical in preventing an adverse outcome [4]. Near-miss identification relies on self-reporting of events and retrospective chart reviews; both of which have been shown to be inferior to direct observation in recognizing such events [5]. Adverse events have been correlated with intraoperative decision making as well as distractions occurring in the operating theater outside the surgical field. Indeed, workflow interruptions have been demonstrated to have a negative effect both on the clinical outcomes and patient safety [6,7,8]. Within urology, one report found that intraoperative interruptions can comprise up to 50% of the case duration [9]. Interestingly, surgical device-related interruptions can occur in up to 30% of operations [6, 7].

True objective observation of intraoperative adverse events and clinical outcomes have been difficult to correlate given the absence of a true method for cataloging in real-time all operating room occurrences that can impact the patient’s recovery. Specifically, traditional methods (audio and video recording) fail to offer a quantifiable methodology for assessing operative quality. The OR Black Box® (OBB) (Surgical Safety Technologies Inc, Toronto, ON) captures video and audio input from the operating room including the surgical field along with physiologic data through the anesthesia monitors and ultimately use artificial intelligence algorithms to assess intraoperative processes to identify sentinel events, which could lead to an unplanned outcome. This provides the potential to develop a real-time feedback system inside the OR. [10] This paper presents the initial experience and result analysis with the OBB technology in a urologic setting.

Methods

Study design

An investigative prospective study in adult patients (age > 18 years old) undergoing elective laparoscopically assisted urologic surgery at a tertiary academic health center by an experienced single surgeon (LK) was conducted. Case data were collected between February and October 2019. The objective of the study was to determine the frequency and characteristics of intraoperative safety threats, errors, and distractions collected in accordance with an institutional quality initiative. Incomplete data collection, technology failure, or inappropriate capture times resulted in exclusion.

Sample and data collection

Surgical cases were recorded using a multichannel data capture platform that synchronized multiple feeds from cameras, microphones, and digital data sources in the operating room (OR Black Box ®, Surgical Safety Technologies Inc, Toronto, ON). Recorded video feeds included room views from two perspectives, and the intra-corporeal camera view. Prior to proceeding to the operating room, written informed consent was obtained from all patients and care providers. Recording began once the patient had been draped and was terminated upon completion of the surgical site closure. Completed recordings were kept on a secure OBB server for 30 days, after which the recording and log of the case were then deleted. Analysis occurs after the recording is obtained and within the 30-day period prior to destruction of all non-deidentified recording content. Synchronized recordings from each case were subsequently reviewed on a single interface using proprietary OBB software by board-certified surgeons who had received at least 3 months of training in the software and protocol. A combination of surgeon analysis and computer algorithms evaluate issues, such as operating room sterility and efficiency, safety checklist utilization, and surgical team communication. Based on these inputs, Surgical Safety Technologies are able to provide analysis and insights regarding these metrics and provide feedback to OR institutional leadership as a point of quality improvement.

Data analysis

Each case was first divided into procedural phases: surgical access, dissection, resection of lesion, reconstruction, inspection, and closure. In addition, for each procedure between 1 and 3 critical steps were identified by a practicing surgeon preoperatively (partial nephrectomy: dissection of renal hilum, mass excision, and renorrhaphy; nephrectomy: dissection of renal hilum and excision of the kidney; adrenalectomy: adrenal dissection). The OBB system’s data were analyzed to determine when the primary surgeon was transferred between attending surgeon and the trainee. Distractions were analyzed using the Distractions in Surgery Index framework [11]. Additionally, the objective structured assessment of technical skills (OSATS) were used to measure and categorize both trainee and attending surgeons’ technical skills. Raters only provided OSATS scores for a priori critical steps of the procedure using only laparoscopic video feeds for calculation in order to blind the analysts to the identity of the operator [12]. A priori determinations of critical steps of the procedure were done in consultation with a urologist. If there was more than one critical step, then an average score for the procedure was calculated. The SEVERE rating scale was used to characterize intraoperative adverse events [13]. Non-technical skills were assessed using the NOTSS, SPLNTS, and ANTS frameworks for surgeons, scrub-technicians, and anesthesia, respectively [14,15,16]. Operating room traffic was also measured, defined as the number of people entering and exiting a case. Finally, any break in sterile technique during the procedure was noted, as was whether or not it was rectified.

Statistical analysis

Statistics were obtained regarding the procedure type, intraoperative distractions and type of adverse intraoperative events using mean and median for continuous data, and frequency analysis for categorical data. To report intraoperative adverse events, we examined the frequency of adverse events per procedure-hour and summarized the distributions across cases using medians and interquartile ranges. All analysis was conducted using SAS 9.4 (SAS Institute Inc., Cary, NC) and R software (R foundation, Vienna, Austria).

Results

During the study period, data from 112 total cases were captured and analyzed; however, there was incomplete data in 29 cases. As a result, 80 sequential cases were analyzed via the OBB system (Fig. 1). Fifteen different types of cases were completed with the majority of these cases being laparoscopic partial nephrectomy (n = 36; 45%), laparoscopic radical nephrectomy (n = 20; 25%), and laparoscopic adrenalectomy (n = 7; 9%).

Fig. 1
figure 1

Data capture by OBB between February to October 2019. There were 80 successful case captures and 29 incomplete case captures. The majority (86%) of the incomplete case captures came as a result from incorrect capture initiation or conclusion. Most of the cases captured were urologic in nature; however, other case data were also collected and analyzed.

Full size image

OSATS score was the highest with the attending surgeon (32/35) and lower among trainees (28/35). The highest OSATS category noted was knowledge of the specific procedure, whereas the lowest OSATS category was noted to be respect for tissue and economy of time and motion. In general, there were 598 non-technical (i.e., not case/surgery specific technique) observations also noted. Of these, 79% (~ 6 episodes/case) were noted to be positive in nature and were related to high situation awareness, excellent team communication, and leadership (Fig. 2). The remaining 21% of observations, approximately 2 episodes were case, cited instances where lapses in situational awareness, task management, or team-based communication occurred.

Fig. 2
figure 2

Non-technical skills (total 598 observations) observed over 80 cases.

Full size image

Across all cases, there were a total of 138 clinically significant IAEs, 10 of which (7.2%) were of the highest severity (5 on the SEVerity of intraoperative Events and REctification Tool (SEVERE) matrix) [13](Fig. 3). The most common IAE overall was bleeding (n = 63), followed by mechanical injury (n = 39); thermal injury (n = 33); and spillage (n = 3). Of these, 70 (51%) occurred during an a priori defined critical step of the operation (for instance, organ dissection, vascular control, or anastomosis). The median IAE per critical step was 1 event (IQR 1–2) versus 2 (IQR 1–3) for the overall case.

Fig. 3
figure 3

Overall, 138 adverse events were recorded and the majority of these events were some type of bleeding event. A total of 1244 distractions were noted over the 80 collected cases. The majority of the distractions were related to personnel communication or equipment-related notifications. Telecommunication distractions were common as well, with the overwhelming majority relating to the OR phone.

Full size image

Distractions were common across all cases. All-cause distractions numbered 1244 over the 80 cases (Fig. 3), the majority of which were related to equipment notifications and personnel. On average, there was noted to be 83 instances of alarms or alerts on average per case and 1346 external communication episodes overall (76% of which was OR telephone). The median rate of external communication per case was 18 events (IQR 11–22) totaling 10 min in duration; and per critical step was 4 (IQR 2.75–8). In addition, median room traffic per case was 65 entries/exits (IQR 42–76); and per critical step was 17 (IQR 10–65).

Lastly, the OBB was able to note when events in the operating room risked breaking sterile protocol. Among the 80 included cases, there were 115 instances of potential infectious risk. The overwhelming majority (95%) of these potential infectious risks occurred as a result of glove self-contamination (n = 109).

Discussion

Over the past several decades, an increasing focus on prevention of post-operative complications has been evolving [17]. Many of these complications could have their genesis as intraoperative adverse events in the operating theater. Prevention of adverse events has been primarily focused on retrospective analysis (e.g., root-cause analysis and morbidity and mortality conferences [18]) or preoperative tools (e.g., WHO surgical checklist [19]). However, both types of instruments rely on participant buy-in and are subject to human error and recall bias [20]. In addition, adverse events may occur during the course of a procedure that may not be recalled that may have played a larger role in the genesis of the adverse event that previously thought. Tools, such as CSATS and OSATS, have been used as capability assessment tools but are still somewhat subjective and give predominantly qualitative assessment focused only on a surgeons skill [21, 22]. While these instruments may be used to improve the overall ability of the surgeon, they neglect other contributory factors, such as communication between operating room team members or distractions outside the surgical field.

The OBB, inspired by the aviation industry, was designed as an objective means of assessing OR events to ensure safety best practices as well as routine preoperative checks and intraoperative surgical skill assessment [23]. By utilizing audio and video feeds from inputs both in the surgical field and throughout the operating room, adherence to surgical safety principles can be better enforced. In addition to evaluating safety, both surgical technical and non-technical skills can be evaluated and feedback provided not only to the surgeon, but the entire operating room staff. Feedback regarding equipment malfunctions or inadvertent alarms can also be provided to manufacturers. By assessing not only each individual participant but the team as a whole, enhanced communication and teamwork will likely aid in reducing human factor-related safety threats.

The OBB is an information tool that provided detailed feedback around all aspects of urologic perioperative and intraoperative care. Our objective was to collect these data to understand the patterns and effects of surgical adverse events in order to help prevent them. Ultimately, these preliminary data demonstrated that urologic surgical teams encounter a significant amount of cognitive and auditory disturbances. Yet, the relevant clinical consequences of these disturbances remain to be determined and further study to correlate these data is required. In general, there were noted to be 83 alarms or alerts characterized as auditory disturbances during each case. The median rate of external communication was 16 events, 4 of which occurred during technically critical steps. The ACS issued a statement acknowledging the growing evidence that OR distractions have a negative consequence on patient safety. [24] In previously simulated settings, surgeons exposed to distractions have been shown to demonstrate inferior surgical proficiency [25]. Cognitive distractions can be related to externally related communication. Cognitive distractions have been shown to result in a greater negative impact on surgical accuracy and successful completion of surgical tasks in the simulated setting [26].

In addition to these distractions, the software platform enabled the capture of intraoperative errors and adverse events. In general, 138 adverse events were captured across 80 completed cases with bleeding being the most common event. There has been evidence to state that while any single adverse event may not become clinically significant, the accumulation of multiple minor adverse events may be critically important. While at this point, the OBB is unable to correlate whether a specific operative adverse event is directly related to a patient adverse event, this will be an important area of future study.

Adverse events may also be able to be correlated with the experience level of the operator. At academic teaching hospitals, trainees contribute a longer duration as the main surgeon especially in less complex cases [27]. Operator skill level directly correlates to patient clinical outcomes, and therefore, a system that tracks ability may be useful in monitoring progress and improving clinical outcomes [28]. The current system uses the OSATS tool manually but also in parallel is developing an autonomous assessment system.

While our study is the first to use the OBB in the urologic setting, it is limited by the fact it was performed at a single center with a single attending urologic surgeon performing laparoscopic urologic procedures. As such, while these data are upper tract focused and may be less generalizable to other aspects of urology, there has been corroborating evidence seen in the general surgery literature27. As the institution and attending surgeon knowingly had installed the OBB in the operating theater, there is the potential for Hawthorne effect interference, yet we believe there was sufficient washout over the months observed. In addition, as the data currently were collected and analyzed off-site through the firm’s secure facility, there is no ability to extract patient-specific medical record locator data to track post-operative clinical outcomes at this time. As such, a notable limitation of this study is the lack of post-operative clinical outcome data. An important future study will be the correlation of clinical outcome information from the electronic health record and with intraoperative adverse events from the OBB. Analyzing post-operative infectious complications may help elucidate the impact of the high rate of breaks in sterility observed in this study and therefore contribute to the development of intraoperative protocols to enhance sterility.

Conclusion

This prospective study demonstrates the ability of the OBB to capture IAEs during minimally invasive urologic surgery. The data demonstrate that IAEs can occur during all phases of an operation and environmental distractions are common in the operating room. Future work will dissect out the role of each category of IAE’s in post-operative issues as well as assess a causal link between distraction and the frequency and severity of IAEs. These analysis could form the basis for establishing OR protocols to minimize post-operative complications. Similarly, these lessons could even be the basis of educating surgeons in settings beyond large academic centers. Only through collecting intraoperative information in a systematic fashion will a means to analyze and prevent adverse surgical events be developed in order to dramatically improve patient outcomes.