Introduction

Clinical rotations (inpatient or outpatient) allow postgraduate medical education (PGME) trainees to acquire complex clinical skills required for the independent practice of medicine. Teaching in such settings, however, presents many challenges, including limited teaching time and increased demands placed on physician educators [1]. Faculty may assume that trainees will arrive with the requisite medical knowledge necessary to successfully engage in patient care on busy clinical rotations. However, this is not always the case [2]. While medical knowledge is emphasized across undergraduate medical education programs, knowledge pertaining to specific inpatient or outpatient rotations may be lacking in trainees. Moreover, significant differences exist between the content and structure of postgraduate education in internal medicine training programs across Europe and the USA [3].

Medical educators have noted a “knowledge gap” between expected and actual knowledge of new postgraduate trainees [2, 4]. Unfortunately, this gap can persist throughout training, placing trainees at risk for poor performance ratings and even failure within their programs [5]. The acquisition and retention of new medical knowledge may even be diminished and/or delayed by an initial medical knowledge gap [5, 6].

Recent studies have sought to enhance our understanding of the problems facing PGME trainees in ambulatory or outpatient settings, with multiple studies suggesting that lack of faculty time for teaching may be a limiting factor [7,8,9]. Studies have shown that adaptive, web-based learning can optimize residents’ learning processes by decreasing the amount of time spent studying [10, 11]. Adapting instruction to an individual trainee’s knowledge baseline and utilizing active online learning experiences build upon both adult learning [12, 13] and instructional design principles [9]. In addition, self-assessments, when embedded within web-based learning management systems, can actively engage trainees and improve learning outcomes [14]. Self-assessments in adaptive learning platforms assist learners in re-calibrating their own performance, which is considered critical in effective self-assessment and directing one’s own learning [15].

Spaced education, a learning method using information delivered repeatedly over time [16], has been shown to significantly improve both short- and long-term retention of medical knowledge [17,18,19,20,21]. Generally, studies in medical education have focused on the benefit that spaced education can bring to the process of learning when learners are assumed to have the same baseline of core medical knowledge. However, one cannot expect the acquisition of new information to be similar when baseline medical knowledge is different.

The purpose of this paper is to describe the use of a self-directed, adaptive spaced education module to increase the internal medicine residents’ medical knowledge present at the beginning of new clinical rotations. While this paper describes outcomes of an innovation related to medical knowledge enhancement during an outpatient rotation, online adaptive, spaced learning can be used prior to any inpatient or outpatient rotation.

Materials and Methods

Design

This prospective study examined the effects of an educational innovation on residents’ medicine knowledge acquisition.

Human Subjects Approval

The study was approved by the Institutional Review Board at the Cleveland Clinic.

Assignment to Groups

Participants were randomly assigned to either the intervention (n = 27 residents) or non-intervention group (n = 27 residents).

Participants and Setting

The educational innovation and assessment of residents’ medical knowledge took place at the Internal Medicine Residency Program at Cleveland Clinic. Fifty-four of 57 (94.7%) postgraduate year 1 (PGY1) residents participated in the curriculum innovation project.

Self-Directed, Adaptive Spaced Education Module Development

The Internal Medicine Residency Program’s curriculum committee approved the pre-rotation curriculum topics. Topics included are the following: diagnosis and management of common medical conditions such as hypertension, diabetes mellitus, and hypercholesterolemia; the Center for Disease Control’s (CDC’s) recommendations for immunizations; and the United States Preventive Services Task Force (USPSTF) screening guidelines. All curriculum topics covered concepts expected to be mastered by interns at the beginning of the outpatient clinical rotation. Two faculty members (AB and AG) created questions to be used within the online, adaptive, spaced repetition–learning module. After items were created, they were reviewed by three other faculty members (SM, AS, and NM) to ensure that they were appropriate for PGY-1 residents.

Adapting to the Level of the Learner

Using the spaced education function in Moodle, AB and AG created online flip cards (Fig. 1). After displaying each question, the module prompts users to pause, self-reflect, and judge whether they have answered the question correctly. After uncovering the correct answer, users (residents) are then prompted to select whether they answered correctly or incorrectly. Any question a user (resident) reports as incorrect is then automatically repeated at shorter time intervals (2–4 days), as compared with questions that a resident reports as correct (8–16 days). Questions that are answered correctly 3 times in a row are automatically retired from the system and no longer repeated. Thus, in this adaptive model, the time required to complete the spaced education program varies depending upon each resident’s individual, self-assessed performance. Flip cards include concepts that need to be mastered in order to correctly answer the pre- and post-intervention medical knowledge assessments. This requires the learner to understand the topics and not simply memorize the questions (Fig. 2). The performance of this adaptive module was tested for feasibility prior to use by residents.

Fig. 1
figure 1

The online spaced education cards. The resident is asked to assess if he/she knows the answer to the question presented in the card outlined in green. Once the resident flips the card, the correct answer is revealed in the card outlined in blue. The resident can remove the card if the answer was correct or keep it in the deck and move to the next card if the answer was incorrect

Full size image
Fig. 2
figure 2

Comparison between the question used to test the medical knowledge during the pre- and post-intervention assessment (a) and the flip cards used to increase the medical knowledge (b)

Full size image

Knowledge Assessment

AB and AG created the 40-item pre- and post-intervention medical knowledge assessments and the 8-item independent post-intervention assessment. All questions were subsequently reviewed by three other internal medicine faculty members (SM, AS, and NM) in order to determine whether content domains appeared to be adequately addressed. AB, AS, and NM are knowledgeable in Accreditation Council for Graduate Medical Education (ACGME) residency curriculum requirements, including medical knowledge and patient care competencies and milestones. AG, who has an EdD in education, provided feedback regarding appropriateness and difficulty of items. Based upon reviewers’ feedback, items were then revised prior to use within the assessment forms.

Medical knowledge was assessed at baseline (pre-intervention) and after 3 months (post-intervention) and included questions covering topics included in the pre-rotation curriculum and the spaced education program. Eight additional questions covering topics not included in the curriculum and the spaced education program were added to the post-intervention test in order to measure the medical knowledge acquisition unrelated to the curriculum tested (“independent assessment” of medical knowledge) (Fig. 3). Topics included are the following: screening and treatment for sexually transmitted diseases, dietary guidelines, treatment of headache, and pain medications. Both groups participated in the medical knowledge assessments at baseline and 3 months later. An electronic audience response system (Socrative) [22] was used to record the answers. Residents did not have access to the correct answers.

Fig. 3
figure 3

Study design. PGY-1, postgraduate year one; USPSTF, United States Preventive Services Task Force; CDC, Center for Disease Control; ABIM, American Board of Internal Medicine

Full size image

Procedures

The intervention group received an electronic version of the clinical rotation curriculum as portable document format (PDF) files and then participated in the interactive, adaptive, online module delivered via Moodle, a free, open-source learning management system. [23] The Moodle “course” was hosted locally by the Cleveland Clinic Education Institute. The non-intervention group also received a PDF of the material included in the clinical rotation curriculum with no other accompanying information. Both groups received curricular materials prior to the start of the outpatient rotation and participated in the same didactic sessions during the outpatient rotations. The residents in the two groups were exposed to patients with acute and chronic medical conditions commonly encountered in outpatient clinics.

Data Collection

The following demographic data were collected: age, gender, degree conferred by medical school (medical doctor or doctor of osteopathic medicine), place of graduation (United States or International medical graduates), and years since graduation from medical school. The following academic data were collected: United States Medical Licensing Examination (USMLE) Step 1, Step 2; In-Training Examination (ITE) scores; and scores from team-created assessments of medical knowledge (pre- and post-intervention and independent assessment of medical knowledge).

Statistical Analysis

The characteristics of study participants were depicted using standard descriptive statistics, specifically Pearsons χ2 for categorical variables and t test for continuous variables were used to analyze the covariates of interest overall and by the two study groups. Graphical methods were used to describe examination performance within each group expressed as mean percent correct ± standard deviation [SD]. The models were adjusted for relevant confounding variables, including demographics and academic performance variables. All statistical tests were 2 sided, and p < 0.05 was considered significant. IBM Corp (2016) SPSS Statistics for Windows, Version 24.0, Armonk, NY, was used for all analyses.

Results

Demographic data of the 54 internal medicine postgraduate year 1 residents participating in the study are depicted in Table 1. There were no statistically significant differences between intervention and non-intervention groups in terms of age, degree conferred, place of graduation, USMLE Step 1, Step 2, and ITE scores, and years since graduation from medical school. There were two times more men in the non-intervention group compared with the study group (74% vs. 37%).

Table 1 Baseline characteristics of Cleveland Clinic internal medicine interns participating in the study
Full size table

The results of the pre- and post-intervention assessment scores are presented in Table 2. There was no difference between pre-intervention and independent post-intervention assessment scores between the two groups (49.1 ± 9.3 vs. 48.8 ± 9.4 and 57.2 ± 17.4 vs. 57.5 ± 18.8 respectively). PGY-1 residents assigned to the intervention group had a statistically significant increase in the post-intervention scores compared with the control group (73.3 ± 17.9 vs 57.9 ± 9.6, p = < 0.001) (Table 2).

Table 2 Pre- and post-intervention knowledge assessment scores in the intervention and control groups
Full size table

The increase in the medical knowledge for the intervention group was 24.2 ± 15.4 compared with 8.6 ± 9.9 for the control group (p = < 0.001) (Fig. 4). The effect size using Cohen’s d was 1.07, which is generally considered to be a large effect [24]. For both groups, the medical knowledge assessed by the independent post-intervention 8 questions not included in the assigned curriculum was similar (57.2 ± 17.4 vs 57.5 ± 18.8, p = 0.95).

Fig. 4
figure 4

Change in residents’ medical knowledge

Full size image

In the multivariable adjusted linear regression models adjusted for demographics and other confounding variables, the residents who received the spaced repetition–learning module had a significant increase in medical knowledge of 21.4% [95% CI (11.9–30.9)] (Table 3). A significant interaction was observed by age, with the effect of intervention being more pronounced in older residents (p for interaction = 0.02).

Table 3 Effect of self-directed, adaptive spaced education module on the increase in medical knowledge
Full size table

Discussion

Results of this intervention indicated that a self-directed, adaptive spaced education module was effective in increasing medical knowledge of new PGME trainees prior to an outpatient rotation (i.e., decreasing their knowledge gap). While spaced education (i.e., educational sessions which are spaced and repeated over time) has been explored within urology [17,18,19,20] and internal medicine residency programs [25], this appears to be the first description of an adaptive, online, self-directed spaced education module to enhance medical education in internal medicine residents. Moreover, the online platform utilized for adaptive learning is open source and available to programs of any size. Results of the intervention are consistent with the literature in other fields, specifically in psychology and urology, showing that repeated reinforcement of learning improves knowledge acquisition and retention, particularly for medical residents and interns [17,18,19,20,21, 25]. These findings are particularly important in light of knowledge gaps, or the difference between supervisors’ expectations of beginning trainees’ medical knowledge and their actual medical knowledge [2, 4,5,6]. There has been much discussion surrounding this issue, with an array of solutions presented, including a renewed focus on the fourth year of undergraduate medical education [26], intensive pre-residency training [27], and a more stringent assessment process that would direct curriculum redesign.

The approach described in this paper takes into account both the systems issues surrounding curriculum redesign, as well as adult learning principles [28]. With the spaced education module, trainees were prompted to self-reflect and assess whether they knew the correct answer. The online system adapted to the specific needs of each intern, which affected medical knowledge acquisition. Increased utilization of the available resources may improve residents’ performance. However, the Moodle version used for this intervention did not allow us to track the time learners actively spent in the course activities.

Limitations of this study include the relatively small sample size and focus on short-term knowledge gains. In addition, although faculty reviewed assessments used in this curriculum innovation, no formal validity evidence was gathered prior to use. Results are also limited by the implementation of this innovation within one area of curriculum content for internal medicine residents. Another limitation is that both the intervention and control groups participated in the same didactic activities, where learners could potentially interact and discuss curriculum and Moodle questions. However, they did not have access to the correct answers for the questions used in the knowledge assessment. Also, while the random assignment resulted in differences in gender distribution across control and intervention groups, to our knowledge, there is no body of empirical evidence suggesting that gender affects performance on spaced education outcomes. In addition, the statistical difference continued to be significant after adjustment for possible confounding variables.

Conclusions

An online, adaptive spaced education module was found to be effective in enhancing medical knowledge of internal medicine trainees prior to the start of a busy outpatient clinical rotation. We think this intervention can be adapted to specific rotations and settings, including inpatient settings. The module, built within an open-source platform (Moodle), allows for a clear definition of objectives by explicitly stating performance goals (e.g., 100% performance on medical knowledge). In addition, trainees’ supervisors can decide upon the sequence of module delivery and use the results to feed into other system decisions, such as individual work assignments. In fact, some authors noted that for online learning to become more efficient, “instructional resources must be adaptable to varying contexts, learners, and educators [9].” We believe that implementing such a module is an efficient method of improving medical knowledge and retention among beginning postgraduate trainees, and may be feasible for a variety of PGME programs and institutions. In addition, we think this methodology could be used in medical schools to teach and formatively assess knowledge associated with Core Entrustable Professional Activities (EPAs) for entering residency.