Competency-based medical education (CBME) has dramatically changed medical education in Canadian anesthesiology programs. Competency-based medical education is an outcomes-based approach to the design, implementation, assessment, and evaluation of medical education.1,2 Society expects educational programs to ensure physicians deliver safe and effective care. The transition to CBME was intended to ensure that all graduating residents achieve the necessary competencies before independent practice.3 In an ideal situation, all the required competencies can be observed or assessed during routine clinical practice. In reality, anesthesiology training is unpredictable and heterogeneous, and many high-stakes situations may not be amenable or appropriate for trainee assessment.4 Simulation-based medical education (SBME), which is defined as any educational activity that uses simulation to replicate clinical scenarios, can be used to complement traditional methods of medical education and allow learners to practice and reinforce their knowledge, skills, and attitudes, especially in low-frequency, high-stakes clinical events.5,6,7

Simulation-based medical education was widely adopted in anesthesiology training programs well before the CBME model was introduced, primarily in response to decreasing tolerance for medical errors and greater emphasis on patient safety.8 Simulation-based medical education is already being used to teach and assess several anesthesiology competencies that are difficult to acquire, including procedural and communication skills, interprofessional learning, nontechnical skills (e.g., situational awareness, leadership, teamwork), and management of rare and emergent clinical situations.5,9 Simulation-based assessments have also been shown to provide valid competency assessments that correlate with clinical performance.10,11 In Canada, all anesthesiology residents must successfully complete five mandatory standardized mannequin-based simulation scenarios, called the Canadian National Anesthesiology Simulation Curriculum (CanNASC) scenarios, prior to certification by the Royal College of Physicians and Surgeons of Canada.12

In response to the implementation of CBME in anesthesiology, the demand for simulation-based training and assessment is anticipated to increase. Thus, there needs to be a greater understanding of how SBME is currently being used for teaching and assessment, the available resources, and any potential barriers to the use of this educational modality. We conducted this study to assess the content and assessment criteria of SBME in Canadian academic departments of anesthesiology.

Methods

Ethics approval for this study was obtained from the University of Alberta Research Ethics Board (Edmonton, AB, Canada; approval number, 93102).

Survey development

We developed an English-language survey in accordance with established guidelines.13 A search of MEDLINE, PubMed®, and Google Scholar identified four prior surveys of simulation use and resource studies.14,15,16,17,18 We created the initial survey questions based on themes and topics described in these articles, and modified the questions based on gaps identified within our Canadian context. Our collective experience includes graduate training in education, fellowship training in simulation and medical education, and experience developing and implementing both summative and formative simulation curricula in Canadian residency programs.

The survey captured demographic information, simulation centre characteristics, simulation faculty characteristics, training program characteristics, simulation content, and learner assessment. The investigators held group discussions, during which items were removed, added, or modified until consensus was achieved on survey question content, survey length, and appropriate response options. We used checkboxes for all question responses as a previous report has shown that closed-ended questions result in fewer incomplete questionnaires than open-ended formats.19 We piloted the survey with an anesthesiology faculty member with expertise in SBME who gave feedback about the flow, content, clarity, time required for completion, and user-friendliness. J. W. B, a methodologist with expertise in conducting survey-based research, reviewed the survey. The resulting feedback was reviewed and incorporated into the final version of the survey, which consisted of 21 questions (see Electronic Supplementary Material eAppendix).

Study population

All 17 Canadian academic departments of anesthesiology were invited to take part in the study. The simulation director/coordinator at each site was identified and either they or their delegate completed the survey.

Survey distribution

We administered and collected surveys from October 2019 to January 2020. An electronic consent form, which described the background and intent of the study, description of the survey content and length, and study investigator contact information, was provided to each institution’s designated SBME contact. A link to the survey was attached to each e-mail invitation using Google Forms (Alphabet Inc., Mountainview, CA, USA). The survey was made available online for ten weeks, with two reminder e-mails sent during the third and sixth weeks after the initial invitation. Involvement in the study was voluntary. All data were stored within a password-protected university-affiliated Google account. No incentives to participate were provided.

Statistical analysis

All survey responses were downloaded from Google Forms and analyzed using Microsoft® Excel version 16 (Microsoft Corporation, Redmond, WA, USA). Demographic data and characteristics of educational programs, simulation content, and simulation assessment were presented as frequencies with percentages and median with range where appropriate.

Results

Demographics

The response rate for the survey was 17/17 (100%). Among the respondents, 9/17 (53%) were simulation directors, while the remaining eight respondents were the local CanNASC leads of their respective simulation programs.

Simulation centre characteristics

The faculty of medicine funded 11 simulation centres (65%) within their university. The second and third most common funding sources were the provincial health authority (5 of 17; 29%) and department of anesthesiology (4 of 17; 24%). Approximately half of the simulation centres were accredited, most commonly by the Royal College of Physicians and Surgeons of Canada (Table 1).

Table 1 Funding and accreditation of Canadian simulation centres, n/total N (%) of programs
Full size table

Simulation faculty characteristics

There was large variability regarding simulation faculty number, roles, and formal training across programs (Table 2). The median number of faculty members involved in simulation at each site was eight (range, 4–20). Three programs (18%) reported no fellowship-trained faculty and eight (47%) had only one to two fellowship trained faculty. Most programs reported faculty who had completed a simulation instructor course or were trained by other faculty members. Table 3 reports the types of compensation received by simulation faculty. Of note, five programs (29%) did not offer any form of faculty compensation for those involved in curriculum design and development, or for running and debriefing simulation sessions. Seven of the programs (41%) were accredited by the Royal College of Physicians and Surgeons of Canada. The three most reported barriers to faculty involvement in SBME were lack of protected time (12 programs; 71%), lack of financial compensation (ten programs; 59%), and lack of appreciation for SBME (seven programs; 41%) (Table 4).

Table 2 Simulation faculty training and involvement, n/total N (%) of programs
Full size table
Table 3 Simulation faculty compensation, n/total N (%) of programs*
Full size table
Table 4 Barriers to simulation faculty involvement in SBME, n/total N (%) of programs*
Full size table

Simulation program characteristics

Fourteen programs (82%) provided a structured rotating annual simulation curriculum. Ten simulation programs (59%) were designed in collaboration with the Residency Program Committee, and eight (47%) were integrated into existing academic curriculums. Fifteen programs (88%) offered an introductory first-year intensive simulation course designed to help transition the learner into residency training. Every program offered both mannequin-based simulators and part-task trainers (Table 5).

Table 5 Training modalities used for simulation teaching, n/total N (%) of programs*
Full size table

Most residents received one to four days of simulation sessions per year during their first to third, and fifth year of residency (Table 6). Two programs (12%) offered 13–16 days of simulation for first-year trainees. Residents during their fourth year of training received five to eight days of simulation sessions per year. Outside of CanNASC, three programs (18%) did not offer any simulation sessions to their fourth- and fifth-year residents.

Table 6 Days of simulation training per year among residents, n/total N (%) of programs, excluding CanNASC
Full size table

Simulation curricular content

More than 90% of the programs included technical or procedural skills training, mannequin-based management of rare perioperative problems, and crisis resource management training during residency training (Table 7). Three programs (18%) did not provide interdisciplinary or interprofessional training or pediatric simulation. Ten programs (59%) did not include in situ simulation training. Only two programs (12%) did not offer training in interpersonal communication skills, and four programs (24%) did not offer simulation sessions related to ethical considerations in anesthesiology practice.

Table 7 Topics within each department’s simulation program, n/total N (%) of programs*
Full size table

In regard to part-task trainers, all programs provided advanced airway simulation-based training. The provision of simulation training for other part-task trainers, such as echocardiography, invasive access or monitoring, bronchoscopy, and regional anesthesia skills ranged from six (35%) to 13 (76%) programs (Table 8).

Table 8 Percentage of part-task trainers used, n/total N (%) of programs*
Full size table

Resident assessment

CanNASC is a mandatory summative simulation-based examination that all senior anesthesiology residents must successfully complete during their training; consequently, all programs use this form of summative simulation-based assessment. Outside of CanNASC, only four programs (24%) used simulation for summative assessment. Among these four programs, the format of assessment varied between mannequin-based (3/4; 75%), objective structured clinical examinations (2/4; 50%), and part-task trainers (1/4; 25%) (Table 9). The purpose underlying summative assessment also varied, with three programs (75%) focusing on the assessment of CanMEDS roles and promotion of residents to the next level of seniority.

Table 9 Characteristics of summative assessments, n/total N (%) of four programs
Full size table

Discussion

Simulation-based medical education is a core component of anesthesiology residency training; however, our survey found important variability in the design, content, and experience of simulation faculty across the 17 programs in Canada. Whether this variability affects the clinical competencies of trainees or, more importantly, patient outcomes, is uncertain.

From a simulation faculty characteristic perspective, most programs we surveyed had engaged three to five faculty to create scenarios and run and debrief simulation sessions, which is similar to centres in the USA.15 Also, as was the case in our study, inadequate financial compensation and time for SBME faculty were reported in three prior international and one Canadian survey exploring barriers to the use of simulation in anesthesiology.15,18,20,21 Many programs also reported a lack of appreciation or recognition for SBME among their nonsimulation faculty. Reasons may include inadequate faculty involvement in simulation activities, lack of awareness of the evidence supporting SBME, and an absence of local champions/leaders advocating for SBME.22 In our study, 18% of programs did not have any faculty who had completed simulation fellowships. Engagement of faculty who have completed formal fellowships in simulation education may champion and improve the profile of SBME within their department and has been shown to improve career satisfaction and scholarly success.23

A previous Canadian survey of anesthesiology residents from 2010 found perceived variability in the way simulation was delivered at each training program.16 Our survey of Canadian simulation directors or their proxy has confirmed these impressions. Our findings are also consistent with an American survey that found large variations in simulation-based training and assessment programs.15 The clinical experiences and exposures of residents during their training period can vary, and relying solely on real patient interactions may not guarantee residents will be exposed to all the necessary training requirements laid out by published national residency curriculums. Thus, simulation is an effective tool to complement our current educational programs to ensure the necessary competencies are achieved.5,6 In addition, integrating SBME into a standardized educational curriculum may support the sustainability and quality of programs.24 Recognizing the importance of SBME, Scottish junior doctors and North American emergency medicine programs have already attempted to standardize and implement national simulation curriculums for both teaching and/or assessment.25,26,27,28

Nevertheless, in light of the monetary costs and resource demands of simulation training, more research is needed on the optimal method of SBME delivery in academic centres before it becomes standard.

A 2010 Canadian survey explored residents’ experiences and attitudes towards SBME. This survey found that junior and senior residents received a median of two simulation sessions per year,16 whereas our survey found that many programs had significantly increased the number of simulation sessions over the subsequent ten years. In the 2010 survey, 81% of the residents agreed that an introductory simulator course focused on management of common intraoperative emergencies should be available, and we found that 88% of programs currently provide such a course.16 These encouraging findings highlight the progress SBME has made over the years, which may indicate a greater acceptance of SBME, less acceptance of patient risk during training, transformation of medical education delivery (i.e., CBME), and reduced costs of purchasing and maintaining simulation equipment.24

Eighty-two percent of the programs we surveyed offered interprofessional or interdisciplinary team training, which is a substantial increase compared with the 2010 Canadian survey,16 in which 76% of residents “rarely” or “never” practiced with individuals from other programs or specialties. Practicing within a team is important during residency training as team training has been shown to improve crisis resource management skills, and is associated with greater patient safety.27,28,29 Furthermore, team training within authentic clinical environments also referred to as in-situ simulation training, has been associated with increased detection of latent safety threats and improved patient outcomes.30,31,32,33 Team training requires participation by several disciplines and allied health care professionals and often requires endorsement and support from organizational leaders.33 Our survey found that team-based simulation training has become increasingly common, which suggests that greater acceptance of SBME has occurred in fields outside of anesthesiology.

Seventy-six percent of programs did not use simulation for summative assessments outside of CanNASC scenarios. Simulation is highly stressful for learners, and traditionally, simulation instructors often prioritize the importance of simulation for formative (learning) purposes and confidentiality as a way of maintaining a safe learning environment.34,35 These concerns compounded by the resource intensive nature of SBME, may partially explain why programs have been slow to adopt summative-based formats. In response to the resource constraints, a recent study by Fleming et al. found that the optimal number of raters and scenarios needed for competency-based assessments in a simulated setting were two and four, respectively.36 These types of innovative studies are needed to help optimize our current resource supply and educational demand.

The advent of CBME in anesthesiology is likely to shift the traditional paradigm of SBME and accelerate the adoption of summative-based SBME, as residents will require more frequent contextualized assessments of their level of competence, especially with rare clinical presentations.12,36 As depicted in Miller’s pyramid, simulation is an ideal tool for assessing a learner’s “shows how” level of competence, thus reflecting a more accurate picture of how a trainee behaves in their actual clinical setting.37 Nevertheless, before widespread adoption, more research is needed regarding proper implementation and validity evidence for summative simulation-based assessment.38

Limitations

This study has several limitations. First, we only surveyed academic centres primarily focused on SBME offered to residents, and our findings may not be representative of SBME offered through community centres or other learners, such as staff, fellows, or medical students. Many residents rotate through community centres during their training, and simulation training in these centres are often run independently from the resident’s formal simulation curriculum. Second, we relied on self-report and did not confirm information provided, which may have been affected by positive response bias. Third, not all respondents were local simulation directors. Recruitment was conducted through our national CanNASC working group, and in some centres, the CanNASC lead is not the simulation director/coordinator. In those instances, the CanNASC lead had the opportunity to clarify or seek answers from their local simulation faculty, including their simulation director. Lastly, our study was conducted prior to the onset of the SARS-CoV-2 pandemic and postpandemic surgical backlog. The pandemic imposed drastic changes and necessary adaptations in the delivery of medical education, including simulation-based training.39,40,41,42 Many centres had to divert simulation resources (space and personnel) towards team preparation and personal protective equipment training during the pandemic. In addition, strict workplace restrictions were imposed that made running routine simulation sessions impossible. In response to these changes, the simulation community began shifting more toward distant and remote simulation techniques.39,40,41,42 Furthermore, with our current national shortage of anesthesiologists and surgical backlog, a greater demand for clinical services may divert time and resources away from nonclinical activity, including simulation-based medical education.43 It is unclear how these factors have influenced our findings.

Conclusion

We found significant progress and innovation in the use of simulation within academic Canadian anesthesiology programs over the past ten years. Substantial variations in these programs, including faculty support, financial and human resources, educational content, and delivery of curricula still exist. Future work is needed to establish optimal strategies to integrate simulation within the CBME framework, explore the return on investment of simulation activities, and gather more validity evidence for SBME.