INTRODUCTION

Professional organizations have called for a redesign of ambulatory medicine training.1–4 Despite the need for reform, ambulatory training models have remained largely stagnant. Newer training models are needed so that best practices in ambulatory structure and scheduling can be defined.

A longstanding problem with training involves the conflict residents encounter when attending to inpatient and outpatient responsibilities simultaneously. The Accreditation Council for Graduate Medical Education now requires programs to minimize this conflict.5

Historically, trainees attend continuity clinic once a week while on inpatient rotations. Alternative models such as long blocks and short weeks of ambulatory interspersed with inpatient rotations can protect a trainee’s inpatient and outpatient experiences and meet the ACGME mandate.6,7 Examples of the latter can be referred to as “X + Y” models such as 4 + 1, 3 + 1, 6 + 2, etc.

To our knowledge, only one study has assessed this templated model.6 Mariotti found increased resident satisfaction with ambulatory training when switching to a 4 + 1 model. This study was limited by its examination of one clinic site, use of non-validated survey items, and lack of assessment of residents’ workflows. More studies are needed to assess the generalizable impact of ‘X + Y’ models on ambulatory training.

In July 2010, we adopted a 4 + 1 model of training. We hypothesized that this new model would: 1) improve residents’ learning opportunities, 2) reduce reported fragmentation of the care residents deliver, and 3) improve satisfaction with ambulatory experiences. A secondary outcome included the self-reported number of patients in each resident’s ambulatory panel for whom the resident felt they were the primary care physician (PCP). We anticipated that residents, faculty, and administrative staff would hold similar perceptions of the new model’s benefits. Finally, we hypothesized that improvements would be achieved across all three ambulatory training sites being used in our program.

SETTING AND PARTICIPANTS

The Hofstra North Shore-LIJ internal medicine residency is a large university-program in New York with 146 trainees. We use three diverse ambulatory training sites: 1) A non-hospital-based patient-centered medical home (PCMH) using an EMR; 2) A hospital-based ambulatory clinic (HBC) using paper charts; and 3) Various community private practices (PP), with 50 % using an EMR.

Participants were all 82 PGY-2/3 residents who experienced both the traditional and 4 + 1 model, 25 faculty members most frequently on service with residents (hospitalist and ambulatory faculty, generalists and subspecialists) and all eight administrative staff responsible for ambulatory patient scheduling and general oversight for the residency program. Interns were excluded because they had no experience in the traditional model of training.

PROGRAM DESCRIPTION

Prior to July 2010, residents attended weekly continuity clinic while on inpatient rotations. After July 2010, a 4 + 1 templated schedule that mirrors Mariotti was implemented.6 Trainees devote 4 weeks exclusively to inpatient experiences, followed by 1 week of exclusive outpatient experiences, in a repetitive 5-week cycle. Trainees do not attend to inpatient/outpatient responsibilities simultaneously. Residents were divided into five cohorts, each attending ambulatory clinic in staggered weeks (see Fig. 1 in the online appendix).

PROGRAM EVALUATION

The new model was studied quantitatively using surveys and qualitatively using a nominal group technique (NGT).8 This mixed-methods approach gathered data on all three primary outcomes: 1) learning opportunities, 2) perceptions around the care delivery experience, and 3) satisfaction with ambulatory experiences. Our secondary outcome (self-reported empanelment) was gathered through survey data.

All data were analyzed using SAS 9.2 (Cary, NC). Learning opportunities and faculty teaching were measured through the previously validated Wayne State Learning Environment subscales. We utilized Chi Square/Fisher’s Exact tests for dichotomous data or t-tests/ ANOVAs for continuous/Likert data.

Nominal Group Technique (NGT) Administration

The modified nominal group technique (NGT) is a curriculum evaluation tool that provides semi-quantitative, rank-ordered feedback on participants’ perceptions of medical education programs. The eight-step technique includes: presenting large group with evaluation question, small group formation to identify strengths/weaknesses, round robin phase with facilitator helping theme small group comments together, clarification phase, voting phase, small group data scoring, large group data combining, and wrap-up discussion.8 We modified the technique by omitting the scoring step due to time constraints.

In separate meetings, residents, faculty, and staff were asked the following question: “Compared to last year’s traditional model of ambulatory training, where residents attended clinic one half-day per week, please comment on the strengths and weaknesses of our 4 + 1 model being used this academic year.” Meetings were conducted in March 2011 to allow sufficient time (9 months) for participants to experience the new model. Participants were provided with food during the meetings. Residents were also provided with gift cards, but were unaware of this compensation a priori.

Survey Administration

We created a mandatory anonymous 38-item paper survey, which was submitted to a secure box in our administrative offices. The survey measured four constructs: learning opportunities (items 1–9), quality of faculty teaching (items 10–20), fragmentation of residents’ care delivery experience (items 21–27), and satisfaction with ambulatory experience (items 28–33). We added four demographic items and one item asking residents to report the number of patients in their PCP panel. The learning opportunity and faculty teaching items were analyzed as subscales per the methods defined by Roth.9 Of the remaining 18 items, 14 came from published studies.10,11

The survey used a Likert scale with: 1 = strongly disagree/never; 5 = strongly agree/quite often. Five questions had dichotomous responses of yes/no answers. The survey was administered in May/June 2011 in a retrospective look-back fashion.12

RESULTS

The survey response rate was 96 % (78/82 residents). NGT participants included 20 randomly selected residents (100 % of those approached participated), 18 faculty most frequently on service with housestaff (72 %), and all four Clinic/four graduate medical education (GME) staff (both 100 %). The number of continuity clinics in the academic year increased from 90 (traditional model) to 150 (4 + 1 model) for PGY-1’s and from 180 (traditional model) to 300 (4 + 1 model) for PGY-2/3’s.

Survey data for all primary outcomes can be seen in Table 1. Results stratified by site can be seen in Table 2 in our online appendix. NGT results can be found in Table 3 in the online appendix. Our secondary outcome (self reported empanelment) is depicted graphically in Fig. 2 in the online appendix.

Table 1 Survey Results Comparing Traditional Versus 4 + 1 Model
Full size table

There was excellent alignment between survey (Table 1) and NG data (Table 3 in the online appendix). The strengths of the new model, as assessed by our primary and secondary outcomes, were seen in all three ambulatory care sites (Table 2 in the online appendix). Residents, faculty, and staff held similar views on the 4 + 1 model’s benefits. The Learner Subscale showed increased learning opportunities reported by residents in the 4 + 1 model (mean rating 4.02 vs. 3.71, p = 0.007). There was no difference in quality of faculty teaching as assessed by the Faculty Subscale (mean rating 4.28 vs. 4.21, p = 0.55).

All seven items assessing fragmentation of the care delivery experience in the inpatient and outpatient setting significantly improved after adopting the 4 + 1 model, including fragmentation of inpatient care (90 % to 3 %, p < 0.0001), inpatient handoffs (65 % to 6 %, p < 0.0001), and competition between inpatient/outpatient responsibilities (92 % to 5 %, p < 0.0001). Residents also reported fewer interruptions (3.51 to 0.11, p < 0.0001) to attend to inpatient duties per clinic session.

Residents’ overall satisfaction, sense of ownership for patients, and confidence in practicing general outpatient medicine all improved after transitioning to the 4 + 1 model (all p < 0.0001). The proportion of residents reporting stress while attending clinic decreased from 96 % to 1 % (p < 0.0001). Finally, residents reported serving as PCP for a larger number of patients (15.11 vs. 6.14) in the 4 + 1 versus the traditional model (p < 0.0001). This trend was seen in each of the three training sites.

Weaknesses of the 4 + 1

While the benefits of the 4 + 1 intervention were strong, certain weaknesses emerged in NGT data. Perceived difficulty caring for patients needing return visits sooner than every 5 weeks was the most frequently cited critique by residents, faculty, and staff. A related issue was faculty’s difficulty in making follow-up point-of-care teaching points with residents. In addition, the GME staff voiced concern with the increasing complexity of resident evaluations, due to five resident cohorts with different rotation dates.

DISCUSSION

Newer models of training are needed to improve ambulatory experiences during residency. Although several programs have endorsed the ‘X + Y’ model of training, only one has published its evaluation of this model.6 Our study supports using a 4 + 1 training model across diverse settings of ambulatory clinical care. Our main objective, to decrease the conflict between inpatient and outpatient responsibilities, was achieved. Reducing this conflict resulted in three main benefits: 1) improving residents’ learning opportunities; 2) reducing fragmentation in the care residents provide patients; and 3) improving satisfaction with ambulatory experiences.

Our traditional model revealed problems with the learning environment that mirrored prior studies.9 On a 5-point-scale, trainees at Wayne State rated their learning opportunities as 3.6 on average, close to our mean value of 3.71. After transitioning to the 4 + 1 model, the Learner Subscale rose in all three ambulatory settings.

Similarly, our trainees suffered from fragmentation in their care delivery with a frequency mirroring a national study of 14,000 trainees (40 %–75 %).10 After shifting to the 4 + 1 model, residents rarely experienced fragmentation issues (1 %–6 %). As with the prior Mariotti study, we confirm an increase in trainee satisfaction in the 4 + 1 model.6

Finally, while our secondary outcome was measured through self report only, we were pleased to see an increase in empanelment. NGT data supported this finding. It is worth noting that all outcomes were highly significant and supported by NGT data from residents, faculty, and staff.

When trainees attend to inpatient and outpatient responsibilities simultaneously, it is the inpatient setting that often “wins” the demand for resident’s time and attention. This conflict leads to a devaluation of ambulatory training.13 Reducing this conflict through X + Y models can give trainees a fair chance to enjoy ambulatory medicine.

We were pleased that our model’s impact was seen across three diverse ambulatory care sites. Importantly, the model can be adapted to sites that vary considerably in the numbers of residents they can accommodate (45 residents at PCMH, 18 at HBC, and 15 at PP). We were not surprised that the improvements at our HBC were slightly larger than at other sites. We believe HBCs may disproportionately suffer from conflict in the provision of inpatient and outpatient care, since residents can (and do) attend to inpatient tasks during clinic sessions, something that is less likely to occur when a clinic is off-site.

The 4 + 1 model was not without its problems. Caring for patients who need follow-up sooner than every 5th week was a challenge. Organization of group practices, across firms, where co-residents provide follow-up care quickly may help mitigate this problem. Also, evaluating five cohorts of trainees with different rotation dates became difficult. Current electronic evaluation software may need modification to better accommodate newer models such as ours.

There are several strengths to our study. To our knowledge, we are only the second authors to report outcomes from a templated ‘X + Y’ model of training. We confirm the improvement in satisfaction found by Mariotti and do so with a larger sample size (n = 78 vs. 48) and better response rate (96 % vs. 68 %).6 Our results extend the existing literature in four important ways. First, we studied additional outcomes of importance to graduate medical education (GME), namely learning opportunities (improved), quality of faculty teaching (no change) and fragmentation of resident’s care delivery (improved). We used questions from validated survey instruments and previously published studies to allow for national comparisons on these outcomes. Second, we studied our intervention’s impact in three vastly different ambulatory settings. Our results were robust across all sites, lending support for the model’s adaptability and enhancing our study’s generalizability/external validity.

Third, we evaluated our new model through a mixed methods approach, thereby obtaining quantitative and qualitative data measuring impact across a range of dimensions. We used a modified NGT instead of a traditional focus group to report qualitative results. Unlike focus groups, NGT members participate equally, discuss intervention strengths/weaknesses, and rank-order items thereby providing a more comprehensive picture of an educational innovations’ impact. Finally, we included a broad representation of stakeholders. Few evaluations of educational innovations examine impact from different viewpoints. To our knowledge, administrative staff has long been a silent voice in studies concerning GME, despite their important role in programmatic structure and function.

Several methodological weaknesses must be acknowledged. First, our findings represent the retrospective results of a single residency program. While we looked at different ambulatory settings, our findings may not be generalizable to other programs. Second, data collection was focused on self-report and oriented around perceived benefits of the intervention. Third, the survey wasn’t administered to faculty/staff. Providing compensation at our NGT meeting may have biased responses in a positive direction, although none were aware a priori. We modified the NGT by omitting the small group scoring step. Of note, the rank-ordering of responses, a key component of NGT methodology, was not compromised. Finally, it was beyond our scope to study patient outcomes or measure empanelment through a data-driven analysis of EMR’s/chart reviews, important areas for future research.

In conclusion, we find that minimizing the conflict between inpatient and outpatient responsibilities can be achieved through templated training models following an ‘X + Y’ schedule. We endorse a 4 + 1 model as an improved model of ambulatory training that can increase resident time in ambulatory continuity clinic, enhance learning opportunities, reduce fragmentation of care residents provide, and improve satisfaction with ambulatory experiences. Because such models are becoming more popular in GME, more studies will be needed to assess their impact on trainees and patients.