Introduction

Continuing medical education is known to improve physician knowledge base, competence, and performance in practice, although it is still unknown whether this translates into healthcare outcome improvement [1–3]. Didactic lectures derived from learning theory as developed in the United States remain the most common approach to CME worldwide, but its implementation has been sporadic and its efficacy is unknown in most low- and middle-income countries. Its effectiveness is even less certain given language barriers and the lack of standardized curricula in residency. CME is traditionally defined as an educational activity that maintains, develops, or increases the knowledge, skills, performance, and relationships a physician uses to provide services for patients, the public, or the profession [4], yet such activities and forums are non-existent in poor countries like Haiti. The Haitian Annual Assembly for Orthopaedic Trauma (HAAOT) is a CME conference that was started in 2013 through a partnership with Partners in Health, the Haitian Ministry of Health, the SIGN Foundation, Foundation for Orthopaedic Trauma, and American Academy of Orthopaedic Surgeons to bridge this gap. The conference consists of a blend of lectures and case-based presentations by visiting foreign surgeons and local Haitian orthopaedic surgeons and residents. While the response to the conference has been enthusiastic as it is the only orthopaedic-focused conference available to surgeons in Haiti, its short-term and long-term effects on participant knowledge have not been studied until now.

Given recent research that has shown that lecture-based learning is a less effective form of teaching when compared to problem-based learning, we sought to explore if a lecture-based course like HAAOT would still have a tangible positive effect on participants’ knowledge base of essential orthopaedics [5–8]. To increase audience interaction, we also used an audience response system (ARS) instantaneously to gauge audience attentiveness as they answered assessment questions. Since this was the first year an ARS was used in the conference, we evaluated short-term outcomes and hypothesized that HAAOT could improve participant’s short-term performance on knowledge-based assessments.

Materials and methods

Setting

The Haitian Annual Assembly for Orthopaedic Trauma (HAAOT) was created in 2013 to address previously non-existent CME in orthopaedic surgery. The conference consists of original research paper presentations, case discussions by Haitian residents and didactic lectures presented by foreign visiting attending surgeons on relevant topics in orthopaedic trauma. The conference is hosted in Port-au-Prince, the capital city, to facilitate increased attendance.

Participants

Study participants included Haitian and foreign (United States/Ireland) orthopaedic residents and attending surgeons drawn from multiple academic centres within their respective countries. All orthopaedic attending surgeons had completed residency training in their home country. Foreign orthopaedic surgeons were used as a positive control for comparison with the Haitian participants.

Electronic invitations were sent to all three orthopaedic residency programs and most of the practicing private orthopaedic clinicians in the country. Two of the three programs are located in Port-au-Prince and the other is in Cap-Haitian, which is located about 251 miles (156 km) north of the capital city.

Survey design

Pre- and post-test questions were designed for each didactic lecture by expert US orthopaedic attending surgeons or extracted from US orthopaedic in-service training exams. The entire test consisted of 40 scored questions (1 min per question for 30 multiple-choice and 30 s per question for 10 true–false), which was designed to have >90 % power to detect a difference of 5 % between the pre- and post-test. Questions reflected the variety of lectures and were chosen to sample across the scope of key areas in orthopaedic surgery relevant to Haiti (e.g. dislocations, osteomyelitis, etc.).

All questions were created as Microsoft Office PowerPoint slides that were integrated with an audience response system (TurningPointResponseCard RF, Turning Technologies) that was rented for $200 from Harvard Medical School Media Services using departmental funds. This system uses radiofrequency to capture audience responses privately. A set of five demographic questions (gender, attending/resident, post-graduate years, prior HAAOT attendance, and hospital affiliation) was used to train participants in the use of the ARS, and all questions were projected in both English and French to account for the differing language preferences of the audience. These questions allowed us to register each response unit to a unique individual while maintaining anonymity.

Determining participant scores was challenging due to the logistics of using an audience response system during a CME conference where participants could miss questions due to momentary absence. We conservatively counted non-answers as incorrect to avoid bias in finding improvement.

Statistical analysis

Cross-sectional descriptive statistics were calculated to quantify demographics. Two-sided paired t tests were used to compare pre- and post-test scores. Post-hoc Bonferroni unpaired two-sided t tests were used following a statistically significant ANOVA to compare among demographic groups.

Results

Demographics

In total, 85 orthopaedic surgeons attended HAAOT (57 Haitian, 28 foreign). Table 1 provides more detailed demographic information. To summarize, among Haitians, 11 were attending physicians, 12 were residents, and 27 were unreported at the time of inquiry. Half (50 %) of all participants had previously attended HAAOT.

Table 1 Demographic Information about participants
Full size table

Response rate: Median response rate using the ARS during the conference was 77.4 % (76.5–85.9 %) and was similar among Haitian and Foreign respondents reflecting active participation. See Table 2 for more detailed daily information on response rates.

Table 2 Response rate by day among Haitian and Foreign participants
Full size table

Pre-test scores

Pre-test scores averaged 21 % for Haitians and 39 % for Foreigners (p < 0.0001) (See Fig. 1). Baseline scores were similar among Haitian attendings and residents averaging approximately 20 % (See Fig. 2a, b).

Fig. 1
figure 1

Pre- and post-test scores among Haitian and Foreign participants for each conference day. *p < 0.05 when comparing between pre- and post-test

Full size image
Fig. 2
figure 2

Pre- and post-test scores for each conference day among Haitian a attendings, and b residents. *p < 0.05 when comparing between pre- and post-test

Full size image

Pre-post differences

Scores improved by 8 % for Haitians (p < 0.0001) and 10 % for Foreigners (p < 0.01) following didactic presentations. Among subgroups, Haitian attendings improved on average by 18 % compared to 6 % for residents (p < 0.0001). Haitian attending improvement trended toward significance when compared to Foreign participants (18 vs. 6 %, p < 0.08). These trends were consistent over all three conference days (See Table 3 and Fig. 2a, b).

Table 3 Score improvements by demographic group
Full size table

Discussion

The delivery of surgical care has become an increasingly relevant issue in low-income countries. Beyond simply hosting visiting surgeons or training local surgeons through medical school and residency, all countries are starting to recognize that continuing medical education for established providers and recent trainees is essential to building, maintaining, and improving surgical capacity [9]. At the same time, international CME is also recognized as more than conferences and courses, but as a structured curriculum requiring accreditation and evaluation of surgeon performance and healthcare outcomes [9]. As technology continues to modify the way education is delivered, we also need to continue to provide learner-centred methods that engage participants in CME.

In Haiti, orthopaedic surgery became acutely sensitive to the challenge of orthopaedic surgical delivery after the tragedy of the 2010 earthquake and the orthopaedic injuries it caused. Haitian orthopaedic surgeons still take care of the complications from the 2010 earthquake, but both the Haitian government and local providers have placed a renewed focus on the rising burden of orthopaedic disease from injury. The HAAOT CME conference was designed to build surgical capacity to meet this need, yet while it has been received favourably by the Haitian government and providers, little is known about the true effect of CME courses in low resource settings. Thus, a key result of our study has been to show that an international CME conference hosted in a low-income country improves short-term performance on knowledge-based assessments. We also present a generalized methodology for evaluating CME conferences using pre-post testing with audience response systems. An added benefit of ARS is that it takes advantage of new technology to further engage participants.

Our results in Haiti are relevant in context and scope when compared to prior work in international CME. The most similar international study in current literature is by Ali et al. who found that the implementation of an Advanced Trauma Life Support course in Trinidad and Tobago among untrained providers led to a pre-post improvement in test scores of 22.0 % that was sizeable even when compared to Nebraska physicians who were used as a control [10]. They found that all participants improved in terms of total score, cognitive, and attitudinal effects. This effect size was similar to another recent study by Kelly et al. in 2014 who found that a short one-week course in musculoskeletal medicine to 154 students using brief didactic talks, case-based small group work, and physical examination skills demonstration resulted in a 27 % improvement in standardized cognitive test scores immediately after the course. After one year, they found that improvement still averaged 21 % [11]. Our study represents a novel hybrid approach between existing studies. We found an improvement of 8 % among all Haitians, but show that Haitian attendings improve by 18 % (relatively in line with the studies previously described) and Haitian residents improve by only 6 %. This dichotomy in improvement may suggest that attending physicians in low-income countries like Haiti, who are more chronologically distant from standardized training (e.g. residency), benefit to a greater extent from CME curricula. Alternatively, this result may occur because residents display a “floor effect” where they lack sufficient knowledge to assimilate information provided in the lectures and are thus unable to show a change in their understanding on the post-test. Furthermore, our estimates of improvement may be lower than other studies since our investigation explores the differential effect of CME by training level.

In addition to reporting on the outcomes of an international CME course in Haiti, our study also presents a systematic methodology for measuring participant performance using an audience response system. The ARS system also has several advantages; it facilitates audience participation and allows for immediate feedback with higher response rates than would be likely from a paper survey. Prior efforts at encouragement of evaluation measurements in workshops have shown adoption rates as low as 28 % [12]. The simplicity of our approach with its high participation rate over 70 % may allow for greater ease in the future adoption of evaluation methods that can help improve the overall quality of global CME efforts.

Limitations

While we show that a CME conference can lead to statistically significant improvements in short-term knowledge-based assessments, our study’s chief limitations are the lack of long-term performance data and the lack of direct correlation with clinical outcomes. However, there is reason to remain optimistic about our study’s findings since prior work has shown sustained improvements in outcomes on knowledge-based assessments [4]. Furthermore, other studies have shown that implementation of systematic education programs improves alignment of surgeon practice with clinical guidelines and lowers annual mortality [13, 14].

Although unlikely, given our study setting, it is also possible that participants inadvertently exchanged their personal ARS devices. To account for this possible confounder, we also compared performance using unpaired two-sided t tests between pre- and post-tests among relevant groups. Our findings remained identical and statistically significant when comparing performance improvement among all participants (Haitians and foreigners) (p < 0.05). Sub-group analysis is not possible in this setting given the new assumption and small sample size.

Conclusions

Our study is the first to present improved short-term knowledge performance during an orthopaedic CME conference in a low-income country. We also demonstrate how an audience response system can facilitate participant engagement and evaluation measurements. While it is unknown whether these improvements will persist over the long-term or improve patient outcomes, we demonstrate that orthopaedic surgeons in this cohort significantly benefit from CME in the short term, and that these gains are greater among attendings than residents. These findings suggest CME conferences in low-income countries will be an effective tool to build surgical capacity and increase surgeon’s knowledge and that audience response systems are useful to engage participants and track outcomes.