Introduction

Reduced clinical exposure has raised concerns regarding limited learning opportunities for trainees (Friedman et al. 2011; Nasca et al. 2010; Samkoff and Jacques 1991). Electronic-learning (E-learning) can be a cost-effective and flexible modality to supplement clinical learning (Cohen 1988; Cook et al. 2008, 2010). In medical education, numerous studies have shown that E-learning can be effective (Cook 2009; Cook et al. 2008, 2010; Custers et al. 1999; Fryling et al. 2011; Hodges and Williams 2007; McCullagh and Weiss 2001; Rosen et al. 2010; Taylor et al. 2005). However, despite calls for research to focus on the instructional design of E-learning, systematic research on how to incorporate learning strategies to optimize E-learning modalities remains scarce (Cook 2009; Cook et al. 2008; Custers et al. 1999; Driskell et al. 1994; Fryling et al. 2011; Hodges and Williams 2007; McCullagh and Weiss 2001; Rosen et al. 2010; Taylor et al. 2005).

A specific modality of E-learning is the podcast. Podcasts are defined as media recordings that can be accessed online or downloaded to a portable multimedia device (Rainsbury and McDonnell 2006). While early definitions of podcasts referred solely to audio recordings, increased access to bandwidth and the growth of streaming sites (e.g. YouTube) have expanded the definition to include other forms of media such as video (Kay 2012). Video podcasts have been referred to as audiographs, podcasts, vodcasts, webcasts, and/or video streams, leading to ambiguity in terminology (Kay 2012). A recent review of podcasting in higher education attempted to eliminate this ambiguity by describing the term ‘enhanced podcasts’ as containing “multimedia information such as slides, pictures, images, photographs, and short videos” (Fernandez et al. 2014).

In healthcare education, a common use of enhanced podcasts has been recorded lectures. Studies have shown that Podcast-based asynchronous video lectures are equally as effective in teaching medical content (Bhatti et al. 2011; Hew 2009; Griffin et al. 2009; Allen and Katz 2011; Bensalem-Owen et al. 2011). Further, a recent review of enhanced podcasts with video concluded that they led to enhanced study habits and motivation to learn (Kay 2012). However, the specific design characteristics that enhanced performance could not be elicited. As such, similarly to the message arising from the broader E-learning literature, further research is needed to determine how manipulations in the design of enhanced podcasts lead to improved learning. In this randomized controlled study, we examined the relative and combined contributions of two instructional design strategies—mental practice and modeling—on medical students’ learning of complex medical knowledge and skills via enhanced podcast.

Mental practice (MP) is “the cognitive rehearsal of a task in the absence of overt physical movement” (Bandura 1986; Driskell et al. 1994). Modeling, also known as observational learning, is the process of acquiring knowledge, skills, and attitudes through viewing examples of performance (Bandura 1986; Custers et al. 1999; Driskell et al. 1994; Feltz and Landers 1983; Feltz et al. 1988; Hinshaw 1991; Nasca et al. 2010; Rosen et al. 2010; Samkoff and Jacques 1991; Taylor et al. 2005). When individually applied, and in combination with modeling, MP has been shown to increase learning and retention in areas outside of health care (Cook et al. 2008; Custers et al. 1999; Driskell et al. 1994; Epstein 1980; Feltz et al. 1988; Feltz and Landers 1983; Hinshaw 1991; Jeannerod 1995; Mahoney and Avener 1977; Nasca et al. 2010; Rosen et al. 2010; Samkoff and Jacques 1991; Start and Richardson 2011; Taylor et al. 2005). The extent to which modeling and MP impact learning from E-learning modalities, including podcasts, remains to be determined. Although many learning design strategies exist, MP and modeling lend themselves well to the attributes of podcasting. Enhanced podcasts fit in with the dynamics of the modern mobile learner, being easily accessible and portable without the requirement of a continuous Internet connection (Cobcroft et al. 2006). MP and modeling are techniques that are ideally suited to such a tool for several reasons. MP does not require an extended amount of time and can be performed in a mobile environment (Driskell et al. 1994). Similarly, short modeling clips are relatively easy to build into enhanced podcasts and can be viewed by learners at their own pace on their mobile device. More importantly, both learning tools are argued to reduce the educational content cognitive load in an enhanced podcast by forcing intentional pauses and time for reflection (Li et al. 2015).

In this study, we focused on medical students’ ability to learn airway management, as it is a ubiquitous skill for all medical students. We hypothesized that, compared to traditional narrated PowerPoint podcasts, the inclusion of MP and modeling, either in isolation or combined, would result in enhanced knowledge retention and skill acquisition in airway management. Secondly, we hypothesized that there would be a synergistic effect when MP and modeling are used in combination, leading to greater knowledge and skill retention in comparison to the use of MP or modeling alone.

Methods

Participants

After Research Ethics Board approval from the University of Toronto and St. Michael’s Hospital (#26530 and #11-063c, respectively), 63 medical students from years 1 to 4 from the Faculty of Medicine at the University of Toronto (Toronto, Canada) were recruited for this study. Participants were naive to the scenario they would encounter in the simulator. Informed consent was obtained in addition to confidentiality agreements to prevent details pertaining to the clinical scenarios being disseminated.

Study design and intervention

This study used a randomized, controlled, repeated measures design. Participants were randomized (using a sealed envelope with allocation details) to one of four groups in which they watched a unique enhanced podcast focused on the decision-making and procedural skills required during airway management. Each group was assigned one of the four versions of the podcast outlining the approach to the airway of a hypoxic patient:

  • Control group Participants watched an enhanced podcast consisting of a PowerPoint presentation with voiceover narration (i.e. similar to a video-recorded traditional lecture).

  • Mental practice (MP) only group Participants viewed the same enhanced podcast as the control group, consisting of a PowerPoint presentation with voiceover narration. In addition, each student engaged in a guided ‘think aloud’ MP session with the primary investigator, where participants were prompted to audibly practice, using the first person perspective, the key learning concepts of the topic, as suggested by the literature (Cook 2009; Cook et al. 2008; Epstein 1980; Jeannerod 1995; Kim et al. 2006; Mahoney and Avener 1977; Rosen et al. 2010; Samkoff and Jacques 1991; Start and Richardson 2011). For instance, the primary investigator asked the participant: “From the perspective that you are holding them in your hand, please describe the types of non-invasive supplemental oxygen therapies and differences between each? What do you see and feel that differentiates them?”; “In the first person, as you approach the patient, go through your approach for an awake patient with stridor…” (see “Appendix”) Think aloud prompts came directly from the enhanced podcast content. The instructor did not provide feedback on the quality or content of the verbal MP session.

  • Modeling only group Participants viewed the same enhanced podcast as the control group, consisting of a PowerPoint presentation with voiceover narration, but the enhanced podcast also included video clips of recorded simulation demonstrations (role model) outlining the approach to the airway of a hypoxic patient with voiceover narration.

  • Combined MP/modeling group Participants viewed the same enhanced podcast as the modeling only group with the recording of the recorded simulation scenario with voiceover narration. In addition, each student engaged in a guided ‘think aloud’ MP session with the primary investigator, as described above for the MP group.

All four groups spent an equal amount of time interacting with the learning content, with the control, MP only and modelling only groups reviewing their podcasts so that the time spent learning was equal across the 4 conditions.

Outcomes measures

On day one, all students completed a multiple choice questionnaire (MCQ) that measured their baseline knowledge of the topic. Each student then individually watched the podcast according to their group allocation with the use of headphones and individual Apple iPad devices. Seven days later, participants were given a second MCQ to measure their knowledge retention. MCQs were first piloted on students who did not take part in the study to confirm that both MCQs matched in their level of difficulty. The students also participated in a 10-min, manikin-based, simulated airway crisis scenario to assess their clinical skills (ability to apply the knowledge acquired from the podcast).

Performance in managing the simulated airway crisis was assessed using three measures:

  • a checklist of pertinent actions (development outlined in the next section) to demonstrate an understanding of the principles and technical skills needed to complete the scenario

  • the Ottawa global rating scale for crisis resource management skills (a scale assessing non-technical skills for crisis management related to five domains; Kim et al. 2006)

  • number of critical errors made (defined as specific actions/lack of actions that would lead to severe morbidity or mortality for the patient—creation outlined in the next section)

We did not include a pre-test simulation scenario, as pilot airway management scenarios revealed that medical students were consistently unable to proceed without prior instruction. Two raters, blinded to the study hypotheses and group assignment, independently rated performance from the videotaped scenarios using the above outlined three measures.

Enhanced podcast content and outcome measure development

The core concepts taught in the enhanced podcast were identified by using pertinent chapters from the most recent version of an Anaesthesia textbook (Morgan et al. 2006). These concepts were included in the MCQ by the principal investigators. Similarly, the simulation scenario, its checklist, and vital critical errors were created to evaluate the core concepts taught in the podcasts. These concepts were then reviewed and modified in an iterative process until two faculty anaesthesiologists and a critical care physician reached consensus (95 % agreement) about the enhanced podcast content, multiple choice tests, simulation scenario, and its checklist. The agreed core content judged as appropriate for medical students included (1) recognizing and assessing a patient in respiratory distress (2) consequences of inadequate oxygenation/ventilation (3) basic approach to increase oxygenation/ventilation (types and use of supplemental oxygen, appropriate utilization of airway maneuverers and nasal/oral adjuncts), (3) indications plus technique for bag mask ventilation and (5) indications (not technique) for laryngeal mask and endotracheal tube insertion. The PowerPoint presentation and video segments on airway management were then developed, converted into podcast format, and transferred onto Apple iPad devices. IPad devices were used because of their mobility and popularity among medical trainees who use them to access medical content outside of the physical classroom (Bensalem-Owen et al. 2011; NUS/HEFCE 2010; Parson et al. 2009; Rainsbury and McDonnell 2006).

Statistical analysis

For MCQs, a two-way analysis of variance was conducted with time (Baseline/Retention) as the repeated measure and group allocation (control, MP, modeling, combined MP/modeling) as the subject variable. Separate one-way analyses of variances were then conducted on the baseline and retention MCQ scores to determine whether there were differences between the four groups. Post-hoc independent t tests, with Bonferroni corrections, were used for comparisons of scores between the groups.

For the key events checklist, Ottawa Global Rating scores, and the mean number of critical errors made, separate one-way analyses of variance were conducted with group allocation as the subject variable, followed by post hoc independent t tests with Bonferroni corrections. There was a high level of rater agreement, with intraclass correlation coefficients of 0.97 for the key events checklist, 0.90 for the mean number of critical errors, and 0.76 for the Ottawa global rating scale. As such, the average scores of the two raters were used in statistical analyses.

Our sample size calculation was based on an effect size of 1.0, which is considered as large and appropriate for educational intervention. Assuming an effect size of 1.0 and a power of 0.8, we calculated a sample size of 16 students per group (Cohen 1988).

Results

Sixty-three medical students were recruited for this study. There were no significant differences between the four groups in terms of (1) gender (χ2 = 2.76, df = 3, p = 0.43), (2) year of training (χ2 = 13.27, df = 12, p = 0.35), (3) previous airway management experience (χ2 = 7.72, df = 9, p = 0.56), (4) previous experience using podcasts (χ2 = 9.89, df = 9, p = 0.36), (5) previous experience using medical podcasts (χ2 = 15.9, df = 12, p = 0.196), or (6) previous experience in a manikin-based simulation environment (χ2 = 15.9, df = 12, p = 0.196).

Multiple choice questionnaires (MCQs)

The two-way ANOVA revealed a significant main effect of time (F(1,61) = 169.365, MSError = 0.01, p < 0.01), and group (F(1,61) = 3919.725, MSError = 0.01, p < 0.01). More importantly, there was a significant time by group interaction (F(3,61) = 6.822, MSError = 0.01, p < 0.01), indicating that the students’ improvements from the baseline to retention MCQs differed based on the group to which they were randomized. A one-way analysis of variance of the baseline MCQ scores with group as the subjects variable revealed that the groups did not differ at the baseline level (F(3,65) = 0.805, MSError = 0.01, p = 0.50). However, the analysis of variance of the retention MCQ scores showed a significant effect of group (F(3,65) = 19.797, MSError = 0.01, p < 0.01). The control group scored significantly lower in comparison to the other three groups (all ps < 0.01). There was no significant difference in scores between the MP and modeling only groups (p = 0.41). However, the combined MP/modeling group scored significantly higher than both the MP only and modeling only groups (both ps < 0.01) (Fig. 1).

Fig. 1
figure 1

Baseline and retention multiple choice questionnaire scores with standard error of mean (All MSError = 0.01)

Full size image

Key events checklist

The ANOVA revealed a significant main effect of group (F(3,59) = 7.249, MSError = 68.19, p < 0.01). The post hoc analyses revealed that the control group scored significantly lower in comparison to the other three groups (all ps < 0.05). There was no significant difference in scores between the mental practice and modeling only groups (p = 0.56). The combined MP/modeling group scored significantly higher than the MP only group (p = 0.01), but only showed a trend towards increased performance when compared with the modeling only group (p = 0.06) (Fig. 2).

Fig. 2
figure 2

Key events checklist scores with standard error of mean (MSError = 68.19)

Full size image

Ottawa global rating scale

The ANOVA revealed a significant main effect of group (F(3,59) = 4.822, MSError = 43.17, p < 0.01). Post hoc analyses revealed that the control group scored significantly lower than the modeling only and combined MP/modeling groups (both ps < 0.05). The MP group did not score significantly different than the control group, but did show a trend towards increased performance (p = 0.08). There was no significant difference between the MP only and modeling only groups (p = 0.51), or between the modeling only and the combined MP/modeling groups (p = 0.23). When comparing the combined MP/modeling with the MP only groups, there was a trend for the combined MP/modeling group to have a higher score (p = 0.06) (Fig. 3).

Fig. 3
figure 3

Ottawa global rating scale scores with standard error of mean (MSError = 43.17)

Full size image

Mean number of critical errors

The analysis of variance revealed a significant main effect of group (F(3,59) = 6.112, MSError = 0.03, p < 0.01). Post hoc analyses revealed that the control group scored significantly lower than the other three groups (all ps < 0.05). There was no significant difference in scores between the MP only and the modeling only groups (p = 0.88), between the MP only and combined MP/modeling group (p = 0.33), or the between the modeling only and the combined MP/modeling group (p = 0.26) (Fig. 4).

Fig. 4
figure 4

Mean number of critical errors made with standard error of mean (MSError = 0.03)

Full size image

Discussion

For medical educators, there is an inherent risk in creating E-learning applications focused on the technology itself, without using educational theories to guide their design (Custers et al. 1999; Driskell et al. 1994; Fryling et al. 2011; Hodges and Williams 2007; McCullagh and Weiss 2001; Morgan et al. 2006; Rosen et al. 2010; Spiro et al. 1992; Taylor et al. 2005). The medical education literature on E-learning modalities highlights different instructional strategies likely to enhance learning, but the question remains of which, and how many, of those strategies should be included in an E-learning module (Cook 2005; 2009; Cook et al. 2008; 2011). Our study addresses this literature gap by investigating the effect of two instructional designs that can easily be integrated into podcasts aimed at increasing both knowledge retention and psychomotor skill acquisition.

The results of this study show that the effectiveness of podcasts for knowledge and clinical skills acquisition was enhanced with the inclusion of either MP or modeling. Further, the combination of MP and modeling had synergistic effects on knowledge retention. For the acquisition of psychomotor clinical skills however, the combination of the two learning strategies had limited effects beyond those observed with either strategy alone. As such, these results suggest that increasing the complexity of instructional design, or the number of learning strategies, in E-learning modalities may not necessarily lead to enhanced learning or performance. It may be prudent to base instructional design on the type of content being taught to learners. Furthermore, there are variations in how a particular learning strategy can be integrated into educational strategies. We explore these concepts further in the following paragraphs.

Modeling can be a simple process of observing behavior without an accompanying description of the actions being modeled. Literature has shown this to be effective for learning when dealing with simple content (Custers et al. 1999; Decker 1980, 1984; Fryling et al. 2011; Hodges and Williams 2007; McCullagh and Weiss 2001; Nast et al. 2009; Rosen et al. 2010; Samkoff and Jacques 1991; Taylor et al. 2005). However, in order to increase learning of a complex clinical skill (i.e. airway management), we included voiceover narration summarizing the key elements of the task. With airway management being a complex task, the vital learning points might not translate to the learner with passive viewing of the simulated actions. As such, podcasts designed with narrated modeling can provide focus to the key actions (Arora et al. 2009; Decker 1980, 1984; Driskell et al. 1994; Immenroth et al. 2007; Nast et al. 2009; Samkoff and Jacques 1991; Sanders et al. 2004, 2008; Welk et al. 2007).

While the theory behind MP provides a framework for learning cognitive skills, the manner in which MP is employed has varied in healthcare education (Arora et al. 2009; Custers et al. 1999; Driskell et al. 1994; Feltz et al. 1988; Feltz and Landers 1983; Hinshaw 1991; Immenroth et al. 2007; Jeannerod 1995; Nasca et al. 2010; Rosen et al. 2010; Samkoff and Jacques 1991; Sanders et al. 2004, 2008; Taylor et al. 2005; Welk et al. 2007). One approach has been to create MP ‘scripts’ to teach psychomotor skills involved in surgical techniques. In studies where participants are given scripts outlining the steps of a surgical task for MP, this results in enhanced performance compared to control (Arora et al. 2009; Cook et al. 2008; Custers et al. 1999; Driskell et al. 1994; Feltz et al. 1988; Feltz and Landers 1983; Hinshaw 1991; Jeannerod 1995; McVee et al. 2005; Nasca et al. 2010; Rosen et al. 2010; Samkoff and Jacques 1991; Taylor et al. 2005). This form of MP is similar to mental rehearsal strategies successfully used by athletes to improve their sport performance. However, healthcare education requires the learning of more than motor tasks; students have to learn complex cognitive skills for situations that require decision-making. The management of complex clinical situations, such as airway management, is a dynamic process that requires anticipatory skills, constant changes in management, and continuous decision-making. As such, having a ‘fixed’ script for the learning of this form of dynamic cognitive skills may not be as effective.

The present study had learners engage in MP by creating personal mental images, and subsequently their own approach, of themselves managing an airway crisis. This was different than reading a script that affords only one approach in a passive manner. As educators utilizing MP, it may be prudent to actively engage learners in MP by guiding them to create their own mental images in the first person based on what has been taught. In such cases, it is argued that learners create their own symbolic connections of the complex content (Jowdy and Harris 1990; McVee et al. 2005; Ryan and Simons 1983). It has been shown that by making their own connections, learners are able to better solve problems during difficult scenarios because they anticipate the consequences of their actions (Decker 1980, 1984; Driskell et al. 1994; Epstein 1980; Feltz and Landers 1983; Glisky et al. 1996; Jeannerod 1995; Jowdy and Harris 1990; Mahoney and Avener 1977; Ryan and Simons 1983; Start and Richardson 2011). The role and effect of different forms of MP is an area for further inquiry.

The synergistic effects of MP and modeling appear to differ depending on whether the cognitive or the more psychomotor aspects of a skill are measured. In the context of this study, cognitive skill refers to the knowledge of the principles of airway management while psychomotor skill refers to the physical application of this knowledge (e.g. managing a difficult airway during the simulated scenario). The broader literature on MP and modeling reveals that MP and modelling appear to have greater learning effects on cognitive than on psychomotor skills (Decker 1980, 1984; Driskell et al. 1994; Epstein 1980; Feltz and Landers 1983; Glisky et al. 1996; Jeannerod 1995; Mahoney and Avener 1977; Start and Richardson 2011). For psychomotor skills, MP and modeling appear to lead to some improvements or learning, but actual physical practice appears necessary for greatest improvements in performance (Custers et al. 1999; Driskell et al. 1994; Feltz et al. 1988; Feltz and Landers 1983; Fryling et al. 2011; Kim et al. 2006; McCullagh and Weiss 2001). The combination of MP and modeling appears to enhance knowledge acquisition related to complex tasks through complementary mechanisms that allow the learner to gain a better understanding and memory for the various aspects of a task. However, the actual physical performance of such tasks requires more than just an understanding of the task. It also requires the physical feel and manipulation of clinical adjuncts, as well as the application of knowledge in dynamic settings that can involve time and/or emotional pressures.

Consistent with the broader literature, the results of the current study show that the combination of MP and modeling had synergistic effects on the more cognitive aspects of the task; the knowledge of airway management principles. However, the effects of the combined strategies on the more psychomotor skills (physical performance during the simulation scenario) were not as straightforward. Acquiring the skills necessary for the physical management of a difficult airway situation may be limited without the concomitant elements that would be present in a real or simulated situation (e.g. the “feel” of clinical instruments or a patient’s unique anatomy; time or emotional pressure). For these types of skills, any form of cognitive learning, whether in isolation or combined, may lead to improvements in performance (as observed in this study). However, they will have limited potential for the development of the ability to apply this knowledge in dynamic situations as well as the more psychomotor aspects of the task. As such, cognitive instructional approaches such as E-learning, even when enhanced with instructional design strategies, are unlikely to completely replace actual hands-on training. Rather, they can be viewed as supplemental learning tools to prepare for learning in a real or simulated clinical situation. The combination of MP and modeling followed by deliberate simulated practice is an avenue for future inquiry.

Our study has certain limitations. Firstly, it focused on the knowledge and clinical skills of airway management. We must therefore be cautious in generalizing these findings to other areas of healthcare performance. However, we believe that the task of airway management requires complex cognitive decision-making and clinical skills similar to the problem solving required in many areas of medicine. A second limitation of the study is the sample size. Given our limited sample size, it is possible that the study lacked the power to detect subtle synergistic effects of MP and modeling on the clinical applications of the acquired knowledge. Lastly, we assessed the effect of a single E-learning session. Further research is required to investigate whether the results would differ with repeated or distributed learning sessions.

In conclusion, the results of this study reveal that the effectiveness of enhanced podcasts for knowledge retention and clinical skill acquisition is increased with either mental practice or modeling. The combination of mental practice and modeling appears to have synergistic effects on knowledge retention, but conveys less clear advantages in its application through clinical skills. The latter result suggests that more is not always better in terms of integrating instructional design strategies into E-learning modules.