Introduction

Simulation-based training is increasingly used for surgical technical skills training in all surgical specialties including otorhinolaryngology (ORL): working hours restrictions, focus on patient safety, and a shift towards competency-based training in the surgical curricula are some of the main reasons for this. Furthermore, simulation-based training allows procedures to be practised repeatedly in a safe environment, offering an important alternative for training rare but critical procedures such as emergency cricothyroidotomy [1]. Finally, simulation-based training is supported by large amounts of evidence including positive and transferable effects on knowledge, skills and behaviour [2,3,4].

Reports on simulation-based skills training can be found for a large number of ORL procedures, representing a wide range of modalities including specific task trainers, manikins, animal and tissue models, cadavers, simulated patients, and virtual reality simulators, as summarized in numerous systematic reviews [5,6,7,8,9,10,11,12] and a book [13]. The incorporation of specific programmes of simulation-based training in a training curricula should be preceded by careful consideration of the educational challenge they aim to alleviate: a methodical approach for curriculum development such as the Kern six-step model calls for problem identification and general needs assessment, targeted needs assessment, the definition of goals and objectives, consideration of educational strategies, implementation, and evaluation [14]. Moreover, systematic implementation of simulation-based training into surgical curricula remains a major challenge [15, 16] and should be motivated by a documented need rather than unsystematically or by opportunity. The first step is to identify and prioritize among the countless surgical technical skills that can potentially be trained using simulation-based technology.

In this study, we conducted a national general needs assessment using a structured approach to identify and prioritize technical skills and procedures in ORL for simulation-based training.

Materials and methods

Study design

The study was designed as a national survey of key educational stakeholders in ORL. We used a Delphi process with three rounds planned á priori, so that information from each round could feed into the next round and be re-evaluated by participants. The Delphi method is used to systematically gather information and iteratively obtain consensus from a group of content experts [17], and has successfully been used for curriculum development in other surgical specialties [18, 19]. We conducted Round 1 in October–December 2016, Round 2 in October 2018, and Round 3 in November–December 2018. In each round, non-responding participants were further reminded/urged twice by e-mails 14 days apart. Ethical approval is not required for this type of study.

Participants

Participants were key opinion leaders actively involved in post-graduate otorhinolaryngological education and training, with representation of all ORL training departments in Denmark. These included all clinical professors, residency programme and course directors, heads of ORL departments, heads of clinical education, junior coordinators of clinical education, and board members of the Danish Society for Otorhinolaryngology—Head and Neck Surgery, the Association of Junior Otorhinolaryngologists, and Danish Association for ORL specialists in private practice. In total, this comprised 62 potential participants out of the 478 working members registered by the Danish Society for Otorhinolaryngology—Head and Neck Surgery in 2016 [20]. Participants were contacted by e-mail and the responding participants in the first round constituted the panel.

Delphi Round 1: brainstorming phase

Participants were asked to list in free text all the technical skills and procedures they would expect a resident to competently perform (i.e. safely and independently) at the end of their residency training in otorhinolaryngology. For example, this could be tonsillectomy or ultrasound examination of the neck. Panellists were encouraged to list as many procedures and technical skills that they found relevant and not be limited by current curriculum, educational challenges, practicalities, or training costs. All the suggested technical skills and procedures were reviewed by the authors (SA and JM) and non-technical skills such as communication skills or teaching experience were excluded. Procedures and technical skills were grouped if relevant, for example diagnostic bronchoscopy and bronchoscopy with foreign body removal, and the Dix–Hallpike test and Epley’s manoeuvre.

Delphi Round 2: needs assessment survey

Panellists were asked to assess the procedures identified in Round 1 according to:

  1. (1)

    perceived importance of competency in the procedure by every ORL specialist

  2. (2)

    frequency of the procedure at their department/clinic

  3. (3)

    number of physicians needed to train at their department/clinic, and

  4. (4)

    patient safety/discomfort if performed by a physician without adequate training in the procedure.

All four questions were assigned 1–5 on a Likert scale with lower scores indicating less importance/lower frequency/fewer physicians needed to train/safer procedure.

The feasibility of simulation-based training for each of the identified procedures was assessed by the authors (SA and JM) according to three criteria:

  1. (1)

    availability of simulation-based equipment for training of the procedure,

  2. (2)

    costs of simulation-based training (both establishing and running costs), and

  3. (3)

    overall suitability for dry training in a simulation centre environment.

Each of the criteria were assigned a 1–5 score on a Likert scale (lower scores worse) and the final feasibility score was calculated for each procedure as the grand mean of the scores on items 1–3 by both assessors.

First the importance score was examined: technical skills and procedures without certain agreement of importance (i.e. lower bound of 95% confidence interval < 3) were excluded. The included procedures were next ranked by the Copenhagen Academy for Medical Education and Simulation needs assessment formula (CAMES NAF) [21] (Table 2). The CAMES NAF score was calculated as the mean of Frequency of the procedure, Number of physicians needed to train, Impact on patient safety and discomfort, Feasibility for simulation-based training after linear data normalization to transform the scores to percentages.

Delphi Round 3: elimination and prioritization

The list of included procedures ranked in the descending order of the CAMES NAF score was sent to panellists for final prioritization: panellists were given the opportunity to revise the ranking order and/or to eliminate procedures from the list. Furthermore, due to a discrepancy between the responses in Round 1 and Round 2 as discussed later, panellists were specifically asked to consider if they found intubation a skill needed to be mastered by every ORL specialist after resident training. For the final list of technical skills that should be prioritized for practice using simulation-based training methods, procedures were included if there was consensus on simulation-based training of the procedure by > 75% of the panellists, ranked by the mean ranking of the respondents in Round 3. For the procedures on the final list, the authors performed a literature search to identify possible available simulation-based training models (Table 3).

Statistical methods

Mean scores, 95% confidence intervals, linear data normalization, NAF score, and rankings were calculated using Microsoft Excel version 15.41 for Mac (Microsoft, Seattle, WA, USA). Spearman’s rank-order was used to investigate the relationship of the rankings of the included technical procedures in Round 2 (determined by the CAMES NAF) and in Round 3 (determined by the participants) using SPSS (SPSS Inc., IL, USA) version 23 for MacOS X.

Results

A total of 62 key opinion leaders were identified and 50 responded to the Round 1 (81%) with all categories of key leaders and all regions represented in the panel (Table 1). The median age of the respondents was 47 years (range 28–69) with a median experience in the specialty of 15 years (range 1–36). Forty-three out of the 50 respondents (86%) had completed specialty training, whereas the remaining seven respondents (14%) were residents in training. Between Round 1 and 2, four participants had changed positions and were no longer considered educational stakeholders and were therefore excluded. In Round 2, the response rate was 56.5% of the still eligible panellists (26 of 46), and in Round 3, 20 panellists responded (43.4%). Figure 1 presents an overview of the Delphi process.

Table 1 Number of respondents in each round
Full size table
Fig. 1
figure 1

Overview of the Delphi process

Full size image

In the brainstorming responses from the first round, a total of 493 items were mentioned with many duplicate items. Of these items, 50 unique procedures and technical skills were identified and these were sent out for assessment of importance, frequency, number of physicians needed to train, and patient safety/discomfort in the second round. Thirteen procedures did not reach the pre-defined cut-off score for importance and five procedures were found only borderline important. In addition, one procedure was selected by the authors for specific consideration as the procedure (intubation) is currently not performed by ORL specialists/trainees in Denmark. After the second round, the list therefore consisted of 36 procedures (Table 2) which were ranked according to the CAMES NAF scores and sent out for final prioritization by the panel.

Table 2 List of all procedures identified in Round 1, ranked by importance and scores from Round 2
Full size table

This resulted in broad consensus (> 75%) of importance for simulation-based training of 13 technical skills or procedures, which were included in the final list (Table 3). There was a strong and statistically significant correlation between the ranking based on the CAMES NAF in Round 2 and the participant’s ranking in Round 3 (rest = 0.78, p ≪ 0.001) (Fig. 2). There was no consensus on whether intubation is a skill that every ORL specialist should be competent in: only 12 out of 20 panellists (60%) found that intubation should be mastered by every ORL specialist. Finally, the authors did a literature search to identify possible simulation-based training models for the 13 included technical skills and procedures (Table 3).

Table 3 Final and prioritized list of technical skills for simulation-based training in otorhinolaryngology and available training modalities
Full size table
Fig. 2
figure 2

Correlation between ranking in Round 2 and Round 3

Full size image

Discussion

In this study, we conducted a nationwide needs assessment based on a Delphi methodology to, by consensus, identify and prioritize technical skills and procedures for simulation-based training in the ORL training curriculum. The Delphi panel included educational stakeholders from all training departments as well as the professional societies involved in planning of ORL residency training in Denmark. The panel was asked to consider four aspects: importance (i.e. that every ORL specialist should be competent in the procedure), the frequency, the number of physicians needed to train in the procedure, and patient safety and discomfort. Current feasibility for simulation-based training as judged by the authors was given only a minor consideration in the initial ranking as part of the CAMES NAF score. Ultimately, the panel was given the opportunity to prioritize and eliminate procedures for simulation-based training, which resulted in final consensus on 13 technical skills and procedures that should be practised in the future using simulation-based training methods.

Our study has several limitations. First of all, the study represents a national survey. Even though we ensured a broad geographic and stakeholder representation within Denmark, this single-country design potentially limits generalizability especially in places where traditions and training curricula differ substantially. Nonetheless, the use of a well-established approach to surgical training needs assessment [18, 32] can hopefully inspire ORL specific studies in other contexts as well. Next, there was a large time gap between the first and the second/third rounds due to changes in the author group. As the first round constituted a brainstorming phase whereas the following rounds were specific prioritization, we do not think this affects the results. It did however lead to the exclusion of the four panellists since they changed positions and were no longer educational stakeholders and may have contributed to the lower response rates overall. Several panellists reported that the surveys were very time consuming, which could also have contributed to the declining response rate [33]. Nonetheless, response saturation was achieved after relatively few responses and further responses contributed mainly to ensuring geographic and stakeholder category representation. Finally, in this study we only considered technical skills and procedures. We recognize that non-technical skills such as communication and teamwork could also be highly relevant for consideration in simulation-based training.

Our panel gave the highest priority of simulation-based training to emergency cricothyroidotomy—a rare event with very high stakes [22]. Such critical procedures that occur very infrequently represents the hallmark of simulation-based training: they cannot be practised reliably in the clinical setting as occurrence is rare, the overall demands on the medical professional in the real-life situation are extremely high, and the potential cost of failure is detrimental. The other procedures prioritized for simulation-based training in our study represent either (1) basic surgical skills such as suturing, knot-tying and instrument handling, and fine needle aspiration; (2) very high frequency procedures such as flexible fibre pharyngo-laryngoscopy, rhino/sinoscopy, and direct laryngoscopy, where diagnostic accuracy is important; (3) intermediate-frequency surgeries such as tracheotomy and septoplasty, where the initial learning curve is a challenge; and (4) more infrequent procedures such as flexible bronchoscopy, flexible and rigid oesophagoscopy, maxillary antrostomy, and removal of salivary duct stones, where patient safety and discomfort requires performance by an experienced physician. In contrast, procedures such as otomicroscopy/otoscopy, the Dix–Hallpike and Epley’s manoeuvres, and vHIT and VNG examination, were not found to necessitate simulation-based training, most likely because they can be practised relatively safely on peers or patients in the clinical context. Similar to other studies [32], we found a strong and positive correlation between the ranking based on the CAMES NAF and the final ranking assigned by the panel. However, the third round added valuable information especially in relation to eliminating procedures.

We found a discrepancy between the responses in the first round, where intubation was mentioned as a procedure only once out of the total 493 items, and in the second round, where intubation achieved moderate scores in importance and in the number needed to train but when combined with a high safety and feasibility for simulation-based training, it has ended among the ten highest ranked procedures. Because intubation is predominantly performed by anaesthesiologists and is not a part of the current Danish ORL training curriculum, we asked the panel specifically to consider the relevance of intubation in the context of ORL training in the final round. However, there was no consensus in the panel on this. In general, most of the identified procedures in Round 1 are mentioned in the Danish ORL curriculum, however, the level of required competency ranges between procedures from “having observed/theoretical knowledge of” to “are able to perform unsupervised”. In this Delphi study, we only wanted to identify and prioritize procedures for simulation-based skills training and not challenge the current curriculum.

Even though simulation-based training has revolutionized surgical training and enabled repeated and distributed practice [34] and mastery learning [35] in a safe learning environment as evidence-based approaches to surgical training, it represents only one tool in the teaching toolbox and does not constitute the one and only solution to all training needs. What simulation-based training excels at is bringing the novice one step up the learning ladder towards competency. Simulation-based training is therefore often an important first step before training in the clinical setting for further experience and skills development [36].

Most of the prioritized technical skills and procedures seem to have either proof-of-concept or commercially available/well-established simulation-based training models that can be implemented in a simulation centre setting (i.e. dry training). With the exception of emergency cricothyroidotomy training on manikins and VR simulation training of flexible bronchoscopy, the remaining simulation models have yet to have a firm evidence-base for their use in surgical skills training. However, it should be noted that the same goes largely for training on human cadavers and on patients.

Overall, it is our impression that models for simulation-based training exist for a majority of the total 50 technical skills and procedures in ORL identified by our panel in the first round. Nonetheless, genuine implementation of simulation-based training into training curricula remains inadequate: a recent Canadian survey of ORL programme directors and residents found a general favourable attitude towards simulation-based training but little actual adaptation into training programmes [37].

Finally, it is important to note that the relatively low number of skills prioritized by the panel reflects the intention of our general needs assessment: to identify skills and procedures that need to be mastered by all ORL specialists in Denmark and not skills exclusively relevant for a subspecialized segment of clinicians. Simulation-based training could still be a relevant modality for teaching general procedural knowledge and light hands-on experience relevant to every ORL specialist for a number of procedures such as hemithyroidectomy, functional endoscopic sinus surgery (FESS) and mastoidectomy. Finally, future targeted needs assessments should be conducted to define, for example sub-specialist training curricula where simulation-based training of more advanced procedures could also be relevant.

Conclusion

A systematic approach to curriculum development is important as evidence-base for educational decisions is increasingly required by stakeholders such as healthcare educational funding bodies and programme directors. In this study, we applied a previously reported approach to identify and prioritize technical skills and procedures for simulation-based training in ORL. Consensus by a representative national panel was iteratively obtained and the most important procedure for simulation-based training was found to be emergency cricothyroidotomy—a rare but critical emergency where immediate action is needed. This was followed by basic surgical skills, a number of very frequently performed diagnostic procedures, fewer intermediate-frequency surgical procedures, and finally, by a number of lower frequency procedures where patient safety and discomfort require them to be performed by an experienced practitioner. Altogether, this prioritized list could be valuable for key educational stakeholders when planning and implementing simulation-based training in the ORL curriculum.