Introduction

Along with the development of healthcare systems in parallel with increasing complexity, human factors are now recognized as important contributors to safety in surgical practice, as in other high-risk industries [1]. Non-technical skills (NTS), such as crew resource management (CRM) in the aviation industry, are recognized by the World Health Organization (WHO), which developed an education curriculum for patient safety focusing on organizational systems and NTS [2]. Several behavior marker systems are available for different areas of healthcare [3]. Surgery is recognized to carry risk and can be demanding. Expertise in surgical technique is essential and is underpinned by NTS, which have been introduced for individual surgical team members, as follows: “anesthetists’ non-technical skills” (ANTS), [4] “non-technical skills for surgeons” (NOTSS), [5] (Table 1) and “scrub practitioners’ list of intra-operative non-technical skills” (SPLINTS) [6]. Since most operations are performed by specialized multi-disciplinary teams, systems such as “observational teamwork assessment for surgery” (OTAS) [7] and “Oxford non-technical skills” (NOTECHS) [8] have been developed to evaluate the whole team in the operating theatre.

Table 1 The non-technical skills for surgeons (NOTSS) skills taxonomy
Full size table

NOTSS is a behavior rating system designed to assess surgeons through structured observations of the non-technical aspects of performance during surgery. This system has been adopted worldwide for guiding the education of trainee and practicing surgeons [9–11]. In Japanese surgical societies, technical skills have been the focus for education, whereas NTS systems have been introduced only recently, [12] although their importance is gradually being acknowledged by the whole medical community. We developed the Japanese version of NOTSS in 2008 (called “jNOTSS”) [13] and have run annual meetings for faculty development over the past 8 years, in collaboration with the NOTSS founders. This pilot study evaluated the jNOTSS rating system for surgical trainers and trainees, and measured the impact of NOTSS education on surgical teams using the OTAS tool.

Methods

Participants

Between 2012 and 2013, four surgical trainers were recruited as NOTSS assessors. All four trainers were attending surgeons at the Department of Gastrointestinal Surgery in Kanagawa Cancer Center, and certified surgical specialists according to the Japanese Surgical Society criteria. One trainer constructed the e-learning system and tutored the other trainers in the master class. A total of six surgical trainees from the same institution were also recruited.

Measurement

  1. 1.

    NOTSS: Each category was scored using the following rating system with an orderly rating score of 1 = poor, 2 = marginal, 3 = acceptable, 4 = good, and N/A = not applicable (skill not required or expected for given clinical situation).

  2. 2.

    OTAS: The whole operating room (OR) team was rated using OTAS at three stages within the pre-, intra-, and post-operation phases, using the rating score of 0 = problematic, 1 = compromised through inadequate behavior, 2 = slight detriment, 3 = neither hindered nor enhanced, 4 = moderately enhanced, 5 = highly enhanced, and 6 = very highly effective [7].

  3. 3.

    Knowledge: Before and after the e-learning education, knowledge and understanding of human factors and NOTSS was assessed by 10 short questions. Space for free text comments regarding NOTSS and the e-learning content was made available.

Design

While participating in the study, the trainers accessed the e-learning system to learn about human factors and the NOTSS system. The e-learning was done by the coauthors during the master class for NOTSS arranged by the Royal College of Surgeons of Edinburgh, RCSEd, at the website of a non-profit organization, Kanagawa Standard Cancer Treatment Support System, KSATSS. Before and after the e-learning education, knowledge and understanding for human factors and NOTSS was checked by 10 short questions and a written test, with free comment on NOTSS and the learning system. After the learning program, the trainers attended a master class for NOTSS rating. During this course, the NOTSS rating on the demo-video was completed by each trainer, who then attended lectures in a master class according to the RCSEd, including discussion on the rating system. After the master class, the trainers commenced rating the trainees according to the NOTSS taxonomy.

The trainees began participating in the study after an induction in the NOTSS system and the study protocol. Four weeks after the commencement of NOTSS rating by the trainers, the trainees accessed and finished their e-learning and completed written tests. The rating of the trainees continued for a further 4 weeks after the education.

Operations selected for observations were non-emergent, frequently performed procedures, of less than 5 h’ duration. The timing of observations was agreed among the participants and the OR team members prior to surgery commencing.

Primary outcome and analysis plan

The endpoints were the NOTSS and OTAS scores. The NOTSS and OTAS scores before and after e-learning were compared by the t test with a significance level of p < 0.05. Reliability of the NOTSS rating among trainers was evaluated using video scenarios of an operation before and after the NOTSS Masterclass training, and Cronbach α > 0.7 was decided as a priori acceptable reliability. Statistical analyses were done using SPSS (version 22). This study was approved by the institutional review board and done in accordance with the Declaration of Helsinki. During the operation, there was no intervention of the patient and general informed consent for observational studies was obtained before surgery. The outline of the study was explained to participants and informed consent given. The study was supported by Grant-in-Aid for Scientific Research (C) 25460633 and a non-profit organization, KSATSS.

Results

Between 2012 and 2013, four trainers and six trainees participated in this study conducted at Kanagawa Cancer Center in Japan. The NOTSS ratings for the trainees were done appropriately in each team, 12 times before the e-learning lecture and 12 times after completing it, repeated twice for each trainee. The OTAS rating was also done 9 times and 5 times, respectively (Fig. 1).

Fig. 1
figure 1

Flow diagram of participants, lectures, and rating. NOTSS non-technical skills for surgeons, OTAS observational teamwork assessment for surgery

Full size image

In two demo-videos, the Cronbach α for NOTSS rating by the trainers was 0.28 and 0.06 before the masterclass, and then 0.81 and 0.82 after it. The test scores for understanding human factors and the NOTSS system were 5.89 ± 1.69 and 8.00 ± 1.32 before and after the e-learning course, respectively (mean ± SD, p = 0.010, n = 9). In the free space for questions and comments in the test paper, the most common comment was a sense of incongruity for the taxonomy of NOTSS, which was unfamiliar to the trainees at first. In contrast, after the e-learning and seminar, they assumed that their understanding of human factors and NTS had improved and that their new visions and skills could be used practically in real clinical situations.

The OTAS scores for the whole team in the second preoperative phase, the first to third intraoperative phases, and the first and third postoperative phases improved significantly after education of the trainees (Table 2). However, there was no difference in NOTSS scores in each element before and after education (Table 3). The NOTSS scores of the trainees were not high, but they were nearly acceptable. Among the mean scores for 12 elements, 5 (42 %) and 7 (63 %) scored 3, being acceptable or better before and after education, respectively. The total scores of teamwork and communication, as well as leadership categories were slightly higher after the learning sessions, although the differences in each category were not significant (Fig. 2).

Table 2 Observational teamwork assessment for surgery (OTAS) scores before and after trainee education
Full size table
Table 3 Non-technical skills for surgeons (NOTSS) scores before and after trainee education
Full size table
Fig. 2
figure 2

Total non-technical skills for surgeons (NOTSS) scores for categories before (pre) and after (post) trainee education. SA situation awareness, DM decision making, CT communication and teamwork, L leadership

Full size image

Discussion

In this study, we validated the Japanese version of the NOTSS system, jNOTSS, which was introduced efficiently in a Japanese Cancer Center. Before the learning events; namely, the e-learning and master course, there was little knowledge of NOTSS taxonomy or consensus in rating among the trainers, but these became standardized after education, enabling us to rate the trainees appropriately. During the master class, rating the demo-video was discussed and then confirmed as mandatory for introducing NOTSS [14].

We observed that the whole team scores, as estimated by OTAS, improved after education, not only for the intra-operative, but also for the pre- and post-operation phases. Introducing NOTSS through learning events may be effective for social skills and team work by improving surgeons’ awareness of the importance of sharing mental models with the whole team. The understanding by both trainers and trainees of human factors and NTS through NOTSS taxonomy, and the consensus for rating among the trainers, will enhance multi-disciplinary team work. Early improvement in patient outcomes through effective implementation of the WHO check list may be associated with those social skills [15]. On the other hand, cognitive and personal skills such as situation awareness and decision making may not be improved by short didactic courses or lectures alone [16], since most are not general skills but need specialized experience. Merely knowing the right thing to do in a given situation is insufficient and does not guarantee that the surgeon will have the NTS to operate competently. For NTS, as exemplified by NOTSS, deliberate practice with feedback from an expert based on a skills taxonomy may be required in addition to knowledge sessions. Simulator training or continuous coaching has also been shown to enhance skill levels in various clinical and non-clinical settings. [17–20]. For example, four NOTSS coaching sessions were found to enhance the non-technical skills of surgical trainees and improve their clinical decision-making and actions in a simulated surgical emergency compared with a control group [21].

The results of the present study are limited since it was a small, non-randomized study that did not evaluate clinical outcome and it was logistically impossible to perform OTAS scoring for every operation [22]. However, it is the first pilot study to evaluate the global NOTSS system in Japan and the aspiration remains to introduce it nationwide. The effectiveness of the NOTSS for achieving observable changes in surgeons’ cognitive skills and clinical outcomes should be evaluated in a large multi-national study that includes coaching or simulation-based training.

Conclusion

Implementation of the NOTSS system for assessing surgical performance is feasible in Japan. Using the NOTSS taxonomy requires appropriate education of both surgical trainers and trainees. The resulting benefit will support the work of the entire surgical team and is likely to improve clinical outcomes.