Abstract
Purpose
To conduct a national needs assessment using a structured approach to identify and prioritize technical skills and procedures in otorhinolaryngology (ORL) for simulation-based training.
Methods
The study was designed as a national Danish survey of key educational stakeholders in ORL. A Delphi methodology with three rounds was used: the first round constituted a brainstorming phase to identify relevant procedures; the second round was a survey of importance, frequency, number of physicians needed to train, and patient safety/discomfort of the procedures, and feasibility of simulation-based training; and a final third round for prioritization.
Results
A total of 62 key opinion leaders were identified and 50 responded in the first round, constituting our panel. Fifty technical skills and procedures were identified in the brainstorming phase and were sent out for assessment, with responses from 56.5% of still eligible panellists. Thirty-six procedures were found important in ORL residency training by the panel. After final prioritization by the panel (response rate 43.4%), there was broad consensus (> 75%) on the need for simulation-based training of 13 technical skills and procedures, with the most highly ranking procedures being emergency cricothyroidotomy, flexible fibre pharyngo-laryngoscopy, and basic surgical skills.
Conclusions
As educational decisions are increasingly required to be evidence-based, this study represents a structured approach to identifying procedures for simulation-based training in ORL. This information can be valuable in the development and implementation of simulation-based training programmes in the ORL residency training curriculum.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Simulation-based training is increasingly used for surgical technical skills training in all surgical specialties including otorhinolaryngology (ORL): working hours restrictions, focus on patient safety, and a shift towards competency-based training in the surgical curricula are some of the main reasons for this. Furthermore, simulation-based training allows procedures to be practised repeatedly in a safe environment, offering an important alternative for training rare but critical procedures such as emergency cricothyroidotomy [1]. Finally, simulation-based training is supported by large amounts of evidence including positive and transferable effects on knowledge, skills and behaviour [2,3,4].
Reports on simulation-based skills training can be found for a large number of ORL procedures, representing a wide range of modalities including specific task trainers, manikins, animal and tissue models, cadavers, simulated patients, and virtual reality simulators, as summarized in numerous systematic reviews [5,6,7,8,9,10,11,12] and a book [13]. The incorporation of specific programmes of simulation-based training in a training curricula should be preceded by careful consideration of the educational challenge they aim to alleviate: a methodical approach for curriculum development such as the Kern six-step model calls for problem identification and general needs assessment, targeted needs assessment, the definition of goals and objectives, consideration of educational strategies, implementation, and evaluation [14]. Moreover, systematic implementation of simulation-based training into surgical curricula remains a major challenge [15, 16] and should be motivated by a documented need rather than unsystematically or by opportunity. The first step is to identify and prioritize among the countless surgical technical skills that can potentially be trained using simulation-based technology.
In this study, we conducted a national general needs assessment using a structured approach to identify and prioritize technical skills and procedures in ORL for simulation-based training.
Materials and methods
Study design
The study was designed as a national survey of key educational stakeholders in ORL. We used a Delphi process with three rounds planned á priori, so that information from each round could feed into the next round and be re-evaluated by participants. The Delphi method is used to systematically gather information and iteratively obtain consensus from a group of content experts [17], and has successfully been used for curriculum development in other surgical specialties [18, 19]. We conducted Round 1 in October–December 2016, Round 2 in October 2018, and Round 3 in November–December 2018. In each round, non-responding participants were further reminded/urged twice by e-mails 14 days apart. Ethical approval is not required for this type of study.
Participants
Participants were key opinion leaders actively involved in post-graduate otorhinolaryngological education and training, with representation of all ORL training departments in Denmark. These included all clinical professors, residency programme and course directors, heads of ORL departments, heads of clinical education, junior coordinators of clinical education, and board members of the Danish Society for Otorhinolaryngology—Head and Neck Surgery, the Association of Junior Otorhinolaryngologists, and Danish Association for ORL specialists in private practice. In total, this comprised 62 potential participants out of the 478 working members registered by the Danish Society for Otorhinolaryngology—Head and Neck Surgery in 2016 [20]. Participants were contacted by e-mail and the responding participants in the first round constituted the panel.
Delphi Round 1: brainstorming phase
Participants were asked to list in free text all the technical skills and procedures they would expect a resident to competently perform (i.e. safely and independently) at the end of their residency training in otorhinolaryngology. For example, this could be tonsillectomy or ultrasound examination of the neck. Panellists were encouraged to list as many procedures and technical skills that they found relevant and not be limited by current curriculum, educational challenges, practicalities, or training costs. All the suggested technical skills and procedures were reviewed by the authors (SA and JM) and non-technical skills such as communication skills or teaching experience were excluded. Procedures and technical skills were grouped if relevant, for example diagnostic bronchoscopy and bronchoscopy with foreign body removal, and the Dix–Hallpike test and Epley’s manoeuvre.
Delphi Round 2: needs assessment survey
Panellists were asked to assess the procedures identified in Round 1 according to:
-
(1)
perceived importance of competency in the procedure by every ORL specialist
-
(2)
frequency of the procedure at their department/clinic
-
(3)
number of physicians needed to train at their department/clinic, and
-
(4)
patient safety/discomfort if performed by a physician without adequate training in the procedure.
All four questions were assigned 1–5 on a Likert scale with lower scores indicating less importance/lower frequency/fewer physicians needed to train/safer procedure.
The feasibility of simulation-based training for each of the identified procedures was assessed by the authors (SA and JM) according to three criteria:
-
(1)
availability of simulation-based equipment for training of the procedure,
-
(2)
costs of simulation-based training (both establishing and running costs), and
-
(3)
overall suitability for dry training in a simulation centre environment.
Each of the criteria were assigned a 1–5 score on a Likert scale (lower scores worse) and the final feasibility score was calculated for each procedure as the grand mean of the scores on items 1–3 by both assessors.
First the importance score was examined: technical skills and procedures without certain agreement of importance (i.e. lower bound of 95% confidence interval < 3) were excluded. The included procedures were next ranked by the Copenhagen Academy for Medical Education and Simulation needs assessment formula (CAMES NAF) [21] (Table 2). The CAMES NAF score was calculated as the mean of Frequency of the procedure, Number of physicians needed to train, Impact on patient safety and discomfort, Feasibility for simulation-based training after linear data normalization to transform the scores to percentages.
Delphi Round 3: elimination and prioritization
The list of included procedures ranked in the descending order of the CAMES NAF score was sent to panellists for final prioritization: panellists were given the opportunity to revise the ranking order and/or to eliminate procedures from the list. Furthermore, due to a discrepancy between the responses in Round 1 and Round 2 as discussed later, panellists were specifically asked to consider if they found intubation a skill needed to be mastered by every ORL specialist after resident training. For the final list of technical skills that should be prioritized for practice using simulation-based training methods, procedures were included if there was consensus on simulation-based training of the procedure by > 75% of the panellists, ranked by the mean ranking of the respondents in Round 3. For the procedures on the final list, the authors performed a literature search to identify possible available simulation-based training models (Table 3).
Statistical methods
Mean scores, 95% confidence intervals, linear data normalization, NAF score, and rankings were calculated using Microsoft Excel version 15.41 for Mac (Microsoft, Seattle, WA, USA). Spearman’s rank-order was used to investigate the relationship of the rankings of the included technical procedures in Round 2 (determined by the CAMES NAF) and in Round 3 (determined by the participants) using SPSS (SPSS Inc., IL, USA) version 23 for MacOS X.
Results
A total of 62 key opinion leaders were identified and 50 responded to the Round 1 (81%) with all categories of key leaders and all regions represented in the panel (Table 1). The median age of the respondents was 47 years (range 28–69) with a median experience in the specialty of 15 years (range 1–36). Forty-three out of the 50 respondents (86%) had completed specialty training, whereas the remaining seven respondents (14%) were residents in training. Between Round 1 and 2, four participants had changed positions and were no longer considered educational stakeholders and were therefore excluded. In Round 2, the response rate was 56.5% of the still eligible panellists (26 of 46), and in Round 3, 20 panellists responded (43.4%). Figure 1 presents an overview of the Delphi process.
In the brainstorming responses from the first round, a total of 493 items were mentioned with many duplicate items. Of these items, 50 unique procedures and technical skills were identified and these were sent out for assessment of importance, frequency, number of physicians needed to train, and patient safety/discomfort in the second round. Thirteen procedures did not reach the pre-defined cut-off score for importance and five procedures were found only borderline important. In addition, one procedure was selected by the authors for specific consideration as the procedure (intubation) is currently not performed by ORL specialists/trainees in Denmark. After the second round, the list therefore consisted of 36 procedures (Table 2) which were ranked according to the CAMES NAF scores and sent out for final prioritization by the panel.
This resulted in broad consensus (> 75%) of importance for simulation-based training of 13 technical skills or procedures, which were included in the final list (Table 3). There was a strong and statistically significant correlation between the ranking based on the CAMES NAF in Round 2 and the participant’s ranking in Round 3 (rest = 0.78, p ≪ 0.001) (Fig. 2). There was no consensus on whether intubation is a skill that every ORL specialist should be competent in: only 12 out of 20 panellists (60%) found that intubation should be mastered by every ORL specialist. Finally, the authors did a literature search to identify possible simulation-based training models for the 13 included technical skills and procedures (Table 3).
Discussion
In this study, we conducted a nationwide needs assessment based on a Delphi methodology to, by consensus, identify and prioritize technical skills and procedures for simulation-based training in the ORL training curriculum. The Delphi panel included educational stakeholders from all training departments as well as the professional societies involved in planning of ORL residency training in Denmark. The panel was asked to consider four aspects: importance (i.e. that every ORL specialist should be competent in the procedure), the frequency, the number of physicians needed to train in the procedure, and patient safety and discomfort. Current feasibility for simulation-based training as judged by the authors was given only a minor consideration in the initial ranking as part of the CAMES NAF score. Ultimately, the panel was given the opportunity to prioritize and eliminate procedures for simulation-based training, which resulted in final consensus on 13 technical skills and procedures that should be practised in the future using simulation-based training methods.
Our study has several limitations. First of all, the study represents a national survey. Even though we ensured a broad geographic and stakeholder representation within Denmark, this single-country design potentially limits generalizability especially in places where traditions and training curricula differ substantially. Nonetheless, the use of a well-established approach to surgical training needs assessment [18, 32] can hopefully inspire ORL specific studies in other contexts as well. Next, there was a large time gap between the first and the second/third rounds due to changes in the author group. As the first round constituted a brainstorming phase whereas the following rounds were specific prioritization, we do not think this affects the results. It did however lead to the exclusion of the four panellists since they changed positions and were no longer educational stakeholders and may have contributed to the lower response rates overall. Several panellists reported that the surveys were very time consuming, which could also have contributed to the declining response rate [33]. Nonetheless, response saturation was achieved after relatively few responses and further responses contributed mainly to ensuring geographic and stakeholder category representation. Finally, in this study we only considered technical skills and procedures. We recognize that non-technical skills such as communication and teamwork could also be highly relevant for consideration in simulation-based training.
Our panel gave the highest priority of simulation-based training to emergency cricothyroidotomy—a rare event with very high stakes [22]. Such critical procedures that occur very infrequently represents the hallmark of simulation-based training: they cannot be practised reliably in the clinical setting as occurrence is rare, the overall demands on the medical professional in the real-life situation are extremely high, and the potential cost of failure is detrimental. The other procedures prioritized for simulation-based training in our study represent either (1) basic surgical skills such as suturing, knot-tying and instrument handling, and fine needle aspiration; (2) very high frequency procedures such as flexible fibre pharyngo-laryngoscopy, rhino/sinoscopy, and direct laryngoscopy, where diagnostic accuracy is important; (3) intermediate-frequency surgeries such as tracheotomy and septoplasty, where the initial learning curve is a challenge; and (4) more infrequent procedures such as flexible bronchoscopy, flexible and rigid oesophagoscopy, maxillary antrostomy, and removal of salivary duct stones, where patient safety and discomfort requires performance by an experienced physician. In contrast, procedures such as otomicroscopy/otoscopy, the Dix–Hallpike and Epley’s manoeuvres, and vHIT and VNG examination, were not found to necessitate simulation-based training, most likely because they can be practised relatively safely on peers or patients in the clinical context. Similar to other studies [32], we found a strong and positive correlation between the ranking based on the CAMES NAF and the final ranking assigned by the panel. However, the third round added valuable information especially in relation to eliminating procedures.
We found a discrepancy between the responses in the first round, where intubation was mentioned as a procedure only once out of the total 493 items, and in the second round, where intubation achieved moderate scores in importance and in the number needed to train but when combined with a high safety and feasibility for simulation-based training, it has ended among the ten highest ranked procedures. Because intubation is predominantly performed by anaesthesiologists and is not a part of the current Danish ORL training curriculum, we asked the panel specifically to consider the relevance of intubation in the context of ORL training in the final round. However, there was no consensus in the panel on this. In general, most of the identified procedures in Round 1 are mentioned in the Danish ORL curriculum, however, the level of required competency ranges between procedures from “having observed/theoretical knowledge of” to “are able to perform unsupervised”. In this Delphi study, we only wanted to identify and prioritize procedures for simulation-based skills training and not challenge the current curriculum.
Even though simulation-based training has revolutionized surgical training and enabled repeated and distributed practice [34] and mastery learning [35] in a safe learning environment as evidence-based approaches to surgical training, it represents only one tool in the teaching toolbox and does not constitute the one and only solution to all training needs. What simulation-based training excels at is bringing the novice one step up the learning ladder towards competency. Simulation-based training is therefore often an important first step before training in the clinical setting for further experience and skills development [36].
Most of the prioritized technical skills and procedures seem to have either proof-of-concept or commercially available/well-established simulation-based training models that can be implemented in a simulation centre setting (i.e. dry training). With the exception of emergency cricothyroidotomy training on manikins and VR simulation training of flexible bronchoscopy, the remaining simulation models have yet to have a firm evidence-base for their use in surgical skills training. However, it should be noted that the same goes largely for training on human cadavers and on patients.
Overall, it is our impression that models for simulation-based training exist for a majority of the total 50 technical skills and procedures in ORL identified by our panel in the first round. Nonetheless, genuine implementation of simulation-based training into training curricula remains inadequate: a recent Canadian survey of ORL programme directors and residents found a general favourable attitude towards simulation-based training but little actual adaptation into training programmes [37].
Finally, it is important to note that the relatively low number of skills prioritized by the panel reflects the intention of our general needs assessment: to identify skills and procedures that need to be mastered by all ORL specialists in Denmark and not skills exclusively relevant for a subspecialized segment of clinicians. Simulation-based training could still be a relevant modality for teaching general procedural knowledge and light hands-on experience relevant to every ORL specialist for a number of procedures such as hemithyroidectomy, functional endoscopic sinus surgery (FESS) and mastoidectomy. Finally, future targeted needs assessments should be conducted to define, for example sub-specialist training curricula where simulation-based training of more advanced procedures could also be relevant.
Conclusion
A systematic approach to curriculum development is important as evidence-base for educational decisions is increasingly required by stakeholders such as healthcare educational funding bodies and programme directors. In this study, we applied a previously reported approach to identify and prioritize technical skills and procedures for simulation-based training in ORL. Consensus by a representative national panel was iteratively obtained and the most important procedure for simulation-based training was found to be emergency cricothyroidotomy—a rare but critical emergency where immediate action is needed. This was followed by basic surgical skills, a number of very frequently performed diagnostic procedures, fewer intermediate-frequency surgical procedures, and finally, by a number of lower frequency procedures where patient safety and discomfort require them to be performed by an experienced practitioner. Altogether, this prioritized list could be valuable for key educational stakeholders when planning and implementing simulation-based training in the ORL curriculum.
References
Melchiors J, Todsen T, Nilsson P, Kohl AP, Bøttger M, Charabi B, Konge L, von Buchwald C (2016) Self-directed simulation-based training of emergency cricothyroidotomy: a route to lifesaving skills. Eur Arch Otorhinolaryngol 273(12):4623–4628
Cook DA, Hatala R, Brydges R et al (2011) Technology-enhanced simulation for health professions education. A systematic review and meta-analysis. JAMA 306(9):978–988
Mcgaghie WC, Issenberg SB, Barsuk JH, Wayne DB (2014) A critical review of simulation-based mastery learning with translational outcomes. Med Educ 48(4):375–385
Dawe SR, Pena GN, Windsor JA et al (2014) Systematic review of skills transfer after surgical simulation-based training. Br J Surg 101(9):1063–1076
Abou-Elhamd KE, Al-Sultan AI, Rashad UM (2010) Simulation in ENT medical education. J Laryngol Otol 124(3):237–241
Clifton N, Klingmann C, Khalil H (2011) Teaching Otolaryngology skills through simulation. Eur Arch Otorhinolaryngol 268(7):949–953
Deutsch ES (2011) Simulation in otolaryngology: smart dummies and more. Otolaryngol Head Neck Surg 145(6):899–903
Wiet GJ, Stredney D, Wan D (2011) Training and simulation in otolaryngology. Otolaryngol Clin N Am 44(6):1333–1350
Javia L, Deutsch ES (2012) A systematic review of simulators in otolaryngology. Otolaryngol Head Neck Surg 147(6):999–1011
Arora A, Lau LY, Awad Z, Darzi A, Singh A, Tolley N (2014) Virtual reality simulation training in otolaryngology. Int J Surg 12(2):87–94
Piromchai P, Avery A, Laopaiboon M, Kennedy G, O’Leary S (2015) Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev 9(9):CD010198
Musbahi O, Aydin A, Al Omran Y, Skilbeck CJ, Ahmed K (2017) Current status of simulation in otolaryngology: a systematic review. J Surg Educ 74(2):203–215
Malekzadeh S (2017) Simulation in otolaryngology. Otolaryngol Clin N Am 50(5):875–1036
Kern DE, Thomas PA, Hughes MT (2009) Curriculum development for medical education: a six-step approach. Johns Hopkins University Press, Baltimore
Deutsch ES, Wiet GJ, Seidman M, Hussey HM, Malekzadeh S, Fried MP (2015) Simulation activity in otolaryngology residencies. Otolaryngol Head Neck Surg 153(2):193–201
Frithioff A, Sørensen MS, Andersen SAW (2018) European status on temporal bone training: a questionnaire study. Eur Arch Otorhinolaryngol 275(2):357–363
Hsu C-C, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12:1–8
Palter VN, Graafland M, Schijven MP, Grantcharov TP (2012) Designing a proficiency-based, content validated virtual reality curriculum for laparoscopic colorectal surgery: a Delphi approach. Surgery 151:391–397
Nayahangan LJ, Konge L, Schroeder T, Paltved C, Lindorff-Larsen K, Nielsen B, Eiberg J (2017) A national needs assessment to identify technical procedures in vascular surgery for simulation based training. Eur J Vasc Endovasc Surg 53:591–599
Danish Society for Otorhinolaryngology—Head and Neck Surgery website. http://dsohh.dk/publikationer/beretninger/. Accessed 7 Jan 2019
Nayahangan LJ, Clementsen PF, Paltved C, Lindorff-Larsen KG, Nielsen BU, Konge L (2016) Identifying technical procedures in pulmonary medicine that should be integrated in a simulation-based curriculum: a national general needs assessment. Respiration 91:517–522
Smith ME, Leung BC, Sharma R, Nazeer S, McFerran DJ (2014) A randomized controlled trial of nasolaryngoscopy training techniques. Laryngoscope 124(9):2034–2038
Davis JR, Hale AL, Ewing JA, Lokey JS (2018) Instituting ultrasound-guided FNA for thyroid nodules into a general surgery residency program: what we learned. J Surg Educ 75(3):594–600
Naur TMH, Nilsson PM, Pietersen PI, Clementsen PF, Konge L (2017) Simulation-based training in flexible bronchoscopy and endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA): a systematic review. Respiration 93(5):355–362
Vanderbilt AA, Mayglothling J, Pastis NJ, Franzen D (2014) A review of the literature: direct and video laryngoscopy with simulation as educational intervention. AMEP 5:15–23
Malekzadeh S, Pfisterer MJ, Wilson B, Na H, Steehler MK (2011) A novel low-cost sinus surgery task trainer. Otolaryngol Head Neck Surg 145(4):530–533
Gardiner Q, Oluwole M, Tan L, White PS (1996) An animal model for training in endoscopic nasal and sinus surgery. J Laryngol Otol 110(5):425–428
Touska P, Awad Z, Tolley NS (2013) Suitability of the ovine model for simulation training in rhinology. Laryngoscope 123(7):1598–1601
van der Wiel SE, Küttner Magalhães R, Rocha Gonçalves CR, Dinis-Ribeiro M, Bruno MJ, Koch AD (2016) Simulator training in gastrointestinal endoscopy—from basic training to advanced endoscopic procedures. Best Pract Res Clin Gastroenterol 30(3):375–387
Allak A, Liu YE, Oliynyk MS, Weng KH, Jameson MJ, Shonka DC Jr (2015) Development and evaluation of a rigid esophagoscopy simulator for residency training. Laryngoscope 126(3):616–619
Pascoto G, Stamm A, Lyra M (2017) Sialendoscopy training: presentation of a realistic model. Int Arch Otorhinolaryngol 21(01):17–20
Nayahangan LJ, Stefanidis D, Kern DE, Konge L (2018) How to identify and prioritize procedures suitable for simulation-based training: experiences from general needs assessments using a modified Delphi method and a needs assessment formula. Med Teach 40(7):676–683
Keeney S, Hasson F, McKenna HP (2001) A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 38(2):195–200
Andersen SA, Konge L, Caye-Thomasen P, Sørensen MS (2015) Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol Head Neck Surg 141(10):913–918
Barsuk JH, Cohen ER, Feinglass J, Mcgaghie WC, Wayne DB (2017) Residents’ procedural experience does not ensure competence: a research synthesis. J Grad Med Edu 9(2):201–208
Bjerrum F, Thomsen ASS, Nayahangan LJ, Konge L (2018) Surgical simulation: current practices and future perspectives for technical skills training. Med Teach 40(7):668–675
Lui JT, Compton ED, Ryu WHA, Hoy MY (2018) Assessing the role of virtual reality training in Canadian Otolaryngology-Head and Neck Residency Programs: a national survey of program directors and residents. J Otolaryngol Head Neck Surg 47(1):61
Funding
None.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Andersen, S.A.W., Nayahangan, L.J., Konge, L. et al. Identifying and prioritizing technical procedures in otorhinolaryngology for simulation-based training: a national needs assessment in Denmark. Eur Arch Otorhinolaryngol 276, 1517–1524 (2019). https://doi.org/10.1007/s00405-019-05352-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00405-019-05352-5