Introduction

Team-based learning (TBL) was developed by Dr. Larry Michaelsen for business school courses to eliminate lectures and use class time for students to solve authentic problems in small groups in the classroom. In a TBL session, there is an (1) advanced preparation assignment—the core of the “flipped classroom” model [1], (2) both an individual and a group readiness assurance test, and (3) a group application exercise that is based on one or more significant problems with specific choice options that teams solve and answer simultaneously [2]. Medical schools in the US began to use TBL in the early 2000s as many explored instructional strategies that would reduce lecture time without sacrificing content coverage, promote self-directed learning, and engage students in the classroom [3, 4]. A review of TBL used in health professions schools found that the majority of studies reported an increase in knowledge-based scores [5].

While there is evidence supporting improved academic performance in pre-clinical medical school courses, including pharmacology, anatomy, and pathology [6,7,8,9], there is little published on the use of TBL in the clinical years. In 2004, Levine et al. [10] demonstrated a significant increase in National Board of Medical Examiners (NBME) subject exam scores for the first 2 years after implementation of TBL into the psychiatry clerkship. Later, a survey of medical students from six different medical schools reported that TBL topic-based curricula in psychiatry clerkships were associated with improved performance on the NBME subject examination [11]. Based on 8 years of sustained improvement at one medical school, Spollen and Cluver [12] recommended replacing traditional lectures with TBL to improve NBME and USMLE step 2 performance.

After 2 years of TBL in a pediatrics clerkship, Warrier et al. [13] reported improved student performance on the NBME subject exam compared to the 3 years before implementation. Also, a year later the TBL students performed significantly better on two pediatric observed structure clinical examinations (OSCE), than their earlier peers, suggesting longer-term retention of knowledge [13]. A more recent study in a pediatrics clerkship found that after TBL implementation the medical school’s scores in the topic area of blood disorders increased and were sustained over 3 years [14].

Our medical school demonstrated promising results after incorporating TBL in pre-clinical courses beginning in 2002 [6, 7]. With such results evident in the pre-clinical curriculum, the leaders of the psychiatry and internal medicine clerkships became increasingly concerned about the passivity of lectures, the lack of student engagement during lectures, and the absence of any consistent and regular feedback on mastery of content during the clerkships. Consequently, in 2007–2008 the internal medicine and psychiatry clerkship directors shifted to a TBL curriculum for their didactics, decreasing formal curricular time from one full day (the common length across all BSOM clerkships) to less than a half day per week. Internal medicine created ten TBL sessions, each 2.5 h in duration during its 12-week clerkship, focusing on common adult disorders and diseases. The psychiatry clerkship created four TBL sessions, each 2.5 h in duration during its six-week clerkship, focusing on common psychiatric disorders. Additional didactic sessions average no more than 1 h a week in the formal curriculum, allowing time for self-directed learning. Prior to this shift to TBL, NBME subject exam scores in both of these clerkships were at the national mean.

After observing our students’ NBME scores following the transformation of the didactic curriculum of these two clerkships, we hypothesized that NBME scores would be more improved in internal medicine and psychiatry than other clerkships that continued to use a lecture-based didactic curriculum. To our knowledge, our study is the first to compare NBME results in TBL-based clerkships to lecture-based clerkships for the same cohorts of students.

Materials and Methods

Psychiatry and internal medicine clerkship TBL sessions adhered to the key components of TBL (advanced preparation assignment, readiness assurance tests and application exercises) in order to generate high student accountability, teamwork, and in-depth learning. Students were assigned to the same team for the duration of the clerkship.

The other clerkships utilized traditional lecture based didactics and other small group teaching. Family medicine was not included in this analysis, as that clerkship did not use the NBME subject examination during the time period studied.

The internal medicine and psychiatry clerkships at the Wright State University Boonshoft School of Medicine (BSOM) analyzed NBME subject test scores for the 11-year period from 2004 to 2005 through 2014–2015, which includes the 8 years of TBL curriculum and 3 years prior to the shift, as the baseline data set. The two-tailed, paired t test was used to determine if the differences between the BSOM means and the national means differed before and after the shift to TBL in 2007–2008. Based on previous cited research [6, 7, 9, 10], we hypothesized that the use of TBL in the clerkship curriculum would have a positive impact on the BSOM NBME scores as compared to the national NBME mean scores. To reject the null hypothesis (i.e., BSOM scores are not significantly different than the national mean scores), the results of the two-tailed t test were considered statistically significant at p < .05. Data are presented as mean ± standard error. This study was determined by the Wright State University Institutional Review Board to be exempt.

Results

Table 1 shows the BSOM and national NBME exam means for the academic years 2004–2006 (prior to implementation of TBL), and 2007–2014. Prior to TBL implementation, BSOM internal medicine and psychiatry clerkships had NBME subject exam means that were similar to the national mean. However, for the 8-year period, students in the two TBL clerkships scored higher on the NBME subject examinations than corresponding national samples. There were no significant differences between BSOM and national end-of-clerkship NBME for pediatrics and women’s health over the 8-year period with the same cohorts of students (Table 1, Fig. 1). However, the difference between BSOM and national NBME performance for surgery was statistically significant. It is important to note that when analyzing data from 2007 to 2008 through 2013–2014, the difference between BSOM and national NBME performance on the end-of-clerkship exam in surgery was not significant (74.89 ± 0.70 vs. 73.99 ± 0.59, p = 0.06). The authors associate this unanticipated result with the implementation of some TBL sessions throughout the surgery didactics during academic year 2014–2015. Though not all surgery didactic sessions were converted to TBL as was the case with both the internal medicine and psychiatry clerkships beginning in 2007–2008, these results support the need to further investigate the mean performance of BSOM students versus the national NBME mean as additional TBL sessions are added to not only surgery, but to other clerkships in the future.

Table 1 BSOM and national NBME subject exam means for 2004–2006 and 2007–2014 academic years
Full size table
Fig. 1
figure 1

Differences (mean ± standard error) between BSOM students and corresponding national samples on the end-of-clerkship NBME tests over the eight-year period of TBLs. BSOM students scored significantly higher in the internal medicine and psychiatry clerkships (p < 0.001). There was a small but statistically significant difference in the surgery clerkship (p = 0.03), and no differences in pediatrics or women’s health. **p < 0.001, *p = 0.03 compared to national samples

Full size image

It should be noted that BSOM students’ scores for USMLE step 1 and step 2 CK were at or near the national average during the time period included in this study.

Discussion

Our knowledge-based results for NBME end-of-clerkship examination scores provide evidence to support using TBL as a key instructional strategy for clinical clerkships. Several factors may explain how TBL, in lieu of didactic lectures, positively impacted student performance on these examinations. First, since students no longer had several hours of lecture each week, time for self-study was created. Second, faculty could focus on clarifying misconceptions and supporting critical thinking during TBL sessions, rather than delivering content. Third, with the transformation to TBL, clerkship leaders redesigned the curriculum to focus on students applying content to solve authentic clinical problems. Fourth, since part of the clerkship grade was based on weekly individual and group readiness assurance tests, students had to stay current with weekly reading assignments. In general, TBL maintains student accountability for learning as an individual and for being prepared to contribute to the learning of teammates. Fifth, the immediate feedback provided by the individual and group readiness assurance tests allowed students to focus on knowledge gaps prior to the end-of-clerkship examination. Finally, the application exercises with each session required students to solve difficult clinical questions in the context of real-life patient information, and to do so, they needed to find and use evidence to support their decisions.

Several features of TBL utilize methods that have been validated to be important aspects in the neurobiology of learning; these include repetition, reward and reinforcement, stress, and active engagement [15]. Repetition occurs through the process of taking the individual readiness assurance test and then taking the same test as a small group. Through the use of the Immediate Feedback Assessment Technique (IF-AT) cards [16], groups receive immediate feedback on their answers and then rewards in the form of points for correct answers. While excessive stress can negatively impact learning, the grading schema of a TBL session simultaneously provides positive stress and ample accountability. TBL learners typically score higher on the group readiness assurance test than the preceding IF-AT; thus, simultaneously success is rewarded and new learning occurs.

The group readiness assurance test and the group application exercises, the two major components of a TBL session, promote active engagement. Further, peer evaluation enhances student engagement for the duration of the TBL process. Although group activity is not unique to TBL, research has found that the process of learning in groups produces enhanced learning [17]. Van Meter and Stevens [18] maintain that the “structure of collaborative discourse” (i.e., peers responding to the comments made by their peers) has a positive impact on individual learning. Indeed, TBL requires considerable discourse between students and faculty, likely further enhancing learning.

In the same year TBL was implemented into the 12-week internal medicine clerkship (2007–2008), three in-house multiple choice question (MCQ) examinations were also introduced into that clerkship. The psychiatry clerkship did not add a MCQ examination until academic year 2013–2014. Consequently, the MCQ tests may have had an additive effect on NBME end-of-clerkship scores. However, the late addition of a local MCQ exam by the psychiatry clerkship (6 years after TBL) suggests that the impact on the NBME scores may not have been powerful. In addition, many NBME subject exam takers across that nation may have had clerkships with in-house MCQ tests similar to BSOM [19]. Also, psychiatry, which is BSOM’s shortest clerkship at six-weeks, out-performed the three clerkships that were eight-weeks in duration (surgery, pediatrics, and women’s health). Further, the evidence is mixed on the relationship between clerkships of varying lengths and NBME subject examination performance [20,21,22,23,24,25].

Strengths and Limitations

The major strength of this research is the fact that within 1 year of changing the didactic curriculum to TBL in both clerkships, there was considerable improvement in the end of clerkship NBME subject examination scores, and those improvements have been sustained for 8 years, in comparison to the three clerkships that did not change didactic curricula. Since the implementation of TBL sessions in the surgery clerkship, NBME subject examination scores have also improved above the national mean, providing additional support to our observation that a shift to TBL yields improvements in NBME subject examination scores.

A major limitation is related to confounding variables that were not included in this analysis, specifically the potential influence of the BSOM internal medicine MCQ examinations on NBME subject exam performance.

Practical Implications and Future Research

This study adds to the evidence that utilizing flipped classroom models, specifically in this case, TBL, improve learner outcomes in the area of knowledge. Future research will be focused on examining changes in student outcomes as other clerkships move to TBL curricula. Additionally, non-cognitive factors, such as transferability of student engagement on clinical teams could be examined.

Conclusions

This knowledge-based data of NBME end of clerkship examination scores provides evidence to support the shift to TBL as the primary pedagogical method of teaching and learning within the clinical clerkships. The utilization of TBL requires students to be accountable for content on a weekly basis, providing opportunity for information to be tested in a formative manner prior to the end of clerkship NBME examination.

While we cannot conclusively state that changing the didactic structure of a clinical clerkship to the TBL method is the only factor related to sustained improvement on the NBME subject examinations in the internal medicine and psychiatry, our results over 8 years provide support for the use of TBL in the clinical years. As we develop our new curriculum at BSOM, the WrightCurriculum, with engaged and self-directed learning modalities as the primary teaching and learning methods throughout the 4 years, this data provides evidence within our own school to shift pedagogy in the other core clinical clerkships.