Throughout the course of an academic year, over 21,000 residents in Internal Medicine provide ongoing comprehensive care to a panel of ambulatory patients.1,2 Residents provide most of this care during 1 half-day weekly continuity clinic in sites that include community, hospital-based or Veterans Health Affairs clinics, faculty group practices, and private physician offices. Yet, reports from the last 15 years note how most Internal Medicine residents feel unprepared to provide outpatient care at the completion of training. Recently, 4 important reports call for urgent reform to the ambulatory education of residents, express concern that residents too often train in “dysfunctional” ambulatory clinics, and argue that residents should train in high functioning outpatient settings in order to learn how to deliver care effectively and efficiently.3–6

However, little systematic and methodologically rigorous information has been gathered on the quality of care provided by residents in ambulatory training sites.7 A few studies have examined some aspects of the quality of care delivered in residency clinics but were limited to single institutions and small numbers of patients.7–10 Less is known about the characteristics of the clinical microsystems, i.e., the working front-line units in which residents provide patient care.

To this end, this study uses a web-based tool developed by the American Board of Internal Medicine (ABIM) for its Maintenance of Certification™ (MOC) program and adapted for residency practices. The ABIM, in collaboration with the Alliance for Academic Internal Medicine (AAIM), implemented this pilot study using the Preventive Cardiology Practice Improvement Module (PC-PIM) to learn more about the practice environment and quality of care provided in 23 ambulatory training sites of 15 diverse Internal Medicine training programs.

The goals of this paper are as follows: (1) to describe the patient population cared for by trainees in internal medicine residency clinics; (2) to assess the quality of preventive cardiology care provided to these patients; (3) to characterize the practice systems that currently exist in Internal Medicine residency clinics; and (4) to examine the relationships between quality, practice systems, and features of the program [size, type of program, and presence of an electronic medical record (EMR)].

METHODS

In February 2004, a request for applications was issued for a joint ABIMF/AAIM project, titled the Resident and Faculty Practicum in Practice-Based Learning and Improvement. From 24 applicants, 15 residency programs (comprising 23 unique ambulatory training sites) were funded for an 18-month feasibility project to implement the PC-PIM in training programs. Each program received approval from their institutional review board. Designated program champions participated in a 2-day orientation and quality improvement training session in June 2004. This paper reports results of clinical, patient survey, and practice system data collected during the initial phase in fall 2004.

Study Participants

The 15 residency programs were selected based on size and type of program, geographic location, qualifications of project champion, strength of support letter, and assessment of potential for completion. Seven programs utilized more than 1 training site; therefore, the unit of analysis is the 23 clinic sites.

Instrument

The PC-PIM is a web-based tool whose purpose is to help physicians better understand and make routine use of the patient and systems data collected from their practice in an effort to improve the quality of care delivered to patients and is closely linked to the Accreditation Council for Graduate Medical Education’s competency goals of practice-based learning and improvement and systems-based practice.1 The tool identifies relevant process and outcome measures based on evidence-based national guidelines of care with broad acceptance from most constituencies.11–14 Participating residents performed chart reviews of a subset of their patients to provide data for calculating measures of preventive cardiology care; the program obtained surveys from patients to assess the presence of and satisfaction with preventive services in the resident-staff clinics, and residents completed one site-level survey that described the practice systems. Power analysis for estimating required sample size to detect differences among the sites for type and size of site and type of medical record determined that 7 chart reviews and 5 patients’ surveys per resident would be sufficient. Patients included were required to have been in the practice for at least 1 year, seen in the last 12 months, and management decisions about their preventive cardiology care made by providers in the practice. The PC-PIM can be viewed in its entirety at www.abim.org/online/pim/demo.aspx.

Patient Survey

Residents, assisted by a research assistant, recruited 5 of their patients to complete a survey.15–16 Questions addressed patients’ perspective on care, self-perceived health status, and two subscales: (1) satisfaction with the practice including overall satisfaction with delivery of preventive cardiology care, specific information about prevention, or side effects of prescription medication; (2) access to practice including ease of obtaining appointments, referrals, and prescription refills.

Chart Review

Residents were expected to abstract charts for 7 of their patients. These were not necessarily the same patients who were surveyed. The abstraction form contained the following: (1) patient demographics; (2) the presence or absence of cardiovascular disease (CVD); (3) the presence or absence of risk factors for coronary heart disease (CHD); 4) whether patient barriers to self-care were present, absent, or not known; (5) the presence or absence of processes of care performed (e.g., lipid testing according to guidelines, blood pressure recording, prescribing aspirin); and (6) clinical outcomes such as the result of most recent lipid profile. Although residents were instructed to abstract information from charts, they were not required to strictly report what was recorded and, therefore, could supply answers from knowledge of the patient or by inferences made from other chart information. For instance, a resident could have answered that he/she advised a patient to stop smoking without it being formally recorded in the chart.

Individual measures such as hypertension were scored dichotomously for each patient; a “1” was equivalent to “yes” and signifies its presence, and a “0” was equivalent to “no” and signifies its absence. For each site, the percent of patients with hypertension was calculated by summing the “1s” and dividing by the total number of patients with recorded data for the variable (0s or 1s) and converting the fraction to a percent [e.g., (5/20) × 100 = 25%].

Summary measures consider the individual measures in a particular category together. The average percent of measures present in a category was calculated and used as an overall assessment of patient health. Summary measures were calculated first at the patient level, then the site level. For example, the summary measure “prevalence of risk factors for CHD” consisted of 9 individual measures. For each patient, it was scored by summing the number of risk factors present (1) and then dividing by the total number of risk factors with recorded data (0s or 1s) and converting the fraction to a percent. For example, 33.3% represents that 3 of the 9 risk factors were present for the patient. For each site, the average overall patient percents represent the site mean, i.e., the average percent of risk factors for the site.

Practice System Survey

The site practice champion and the residents completed a survey assessing key structural elements of the clinic’s practice systems. The Practice System Survey was developed by two of the authors (FDD and LAL) [AU1]based on the principles of the Wagner Chronic Care Model, the Institute for Healthcare Improvement Idealized Office Design project, and Putting Prevention into Practice monograph from the Task Force On Clinical Preventive Services.17–19

Six broad categories of practice system elements were included: care management (26 questions), patient-activation and communication (22 questions), modes of communication (9 questions), practice-based learning and improvement (5 questions), information management (12 questions), and environment (10 questions). The response categories for the scale for each question were “yes” signifying that the particular element was operating well in the practice, “maybe” signifying that the element could be improved, and “no” signifying that the element was not available or operational. Subscale scores were derived as the percent of questions that received “yes” responses out of the total number of non-missing responses.

Statistical Analyses

Descriptive statistics including means, standard error of the mean, and range were used to describe the study variables. The unit of analysis was the site level. Differences in processes, outcomes, and systems of care were compared for program type (community or university), presence of an EMR (yes or no), and size of program (small or large) using multivariate analysis of variance with a p value < .01. Pearson correlations were computed between all site level measures. Cronbach’s alpha reliability coefficient measured the internal consistency reliability of the patient survey, the practice system survey, and its subscales.

Risk adjustments were done using generalized estimating equations (patients nested within site) with logistic regression for binary measures and normal linear regression for continuous measures with the assumption that the pattern of correlations among patients nested within sites is exchangeable among sites. Risk adjustments were done for the dependent variables: clinical outcome measure goals [diastolic and systolic BP, low- (LDL) and high-density lipoprotein (HDL) cholesterol, and triglycerides], patient satisfaction, and patient access to practice. The independent measures were patient age, patient gender, and physician’s assessment of patient’s factors limiting self-care.

RESULTS

Participating Programs

A total of 709 (96% of all) residents participated, with an average of 31.3 residents (SD = 19.2) per site. Participants were distributed evenly across the 3 years of training (31%, 33%, and 34%, respectively). Overall, 57% of residents were male. Type of residency program was self-described by program directors; 8 of the programs (with 13 sites) were community, 6 (with 9 sites) were university, and 1 was military. Ten sites reported having an EMR. “Size” was dichotomized at the median of 32 residents with 11 sites above the median (range, 4–73). A total of 4,783 charts were abstracted, an average of 6.7 per resident. A total of 3,092 patients responded to the patient survey, an average of 4.4 per resident. The chart review sample was similar to the sample of patient survey responders in terms of gender (44.6% male vs 42.4% male, NS) and prevalence of current smoking (22.0% vs 23.2%, NS) but was slightly older (average 56.0 years vs 52.7 years, p = .05).

Patient Sociodemographics and Risk Factors

Characteristics of the residents’ patients are shown in Table 1. The average patient age in the chart review was 56.0 years, and male patients constituted 44.6% of the sample (94.3% represented the single Veterans Health Administration (VHA) site). The patients in this study had a low overall incidence of CVD compared to patients seen by MOC candidates who took this same module most likely because the latter were sicker patients seen mostly in cardiology practices.20 Approximately 1 quarter of patients had social factors, problems with adherence, or other medical conditions that limited self-care, and 14% had psychiatric illness or cognitive impairment. On average, 46.9% of patients (range, 31.8–77.8%) had one or more of the 4 factors limiting self care: 21.3% (range, 11.4–34.3%) had one factor, 13.8% (range, 4.1–22.6%) had two, 8.6% (range 2.7–16.4%) had three, and 4.1% (range 0.4–8.2%) had all four factors.

Table 1 Chart Review: Patient Characteristics (n = 23 Sites Based on Data for 4,783 Patients)
Full size table

Table 1 also reports 5 CVD categories and 9 possible risk factors for CHD. The prevalence is a measure of the health of the patients and reflects an average (in percent) of how many of the categories or risk factors patients have. Patients had an average of 3 to 4 (41.2%) of the 9 potential risk factors for CHD, whereas 65.9% had hypertension only. The wide range across sites in risk factors is striking, often between 30 and 60 percentage points for a given risk factor. On average, only 4.8% of patients (range, 0.4–16.7%) were very healthy having none of the 9 CHD risk factors, whereas 55.0% (range, 29.2–75.7%) had four or more.

Processes of Care

Table 2 summarizes performance on processes of care measures derived from chart abstracted data. Preventive intervention measures showed a wide range of compliance with guidelines. For example, beta blockers were prescribed when indicated between 33% and 100% of the time. Charts documented recommended lifestyle modifications approximately half the time. Mean site performance for lifestyle modifications was lowest for increased consumption of fruits, vegetables, and/or soluble fibers (44.9%) and highest for smoking cessation support (73.8%). Overall, patients received an average of 57.4% of the appropriate interventions.

Table 2 Chart Review: Processes of Care (n = 23 Sites Based on Data for 4,783 Patients)
Full size table

Risk-Adjusted Outcomes of Care

Table 3 provides a summary of 3 types of risk adjusted outcome measures—those based on clinical endpoints derived from chart abstracted data (adjusted for age, gender, and factors limiting self-care); those based on patient satisfaction; and those based on access to practice derived from patient survey data (both adjusted for age, gender, and patient’s rating of health). For clinical outcomes, the mean site performance ranged from 48.3% for systolic BP <130 mm Hg to 78.9% for diastolic blood pressure below 85 mmHg; on average, patients achieved between 3 and 4 (65.1%) of the 5 clinical outcomes. For patient satisfaction, the mean site data showed overall satisfaction with care rated excellent by an average of 34.4% of the patients and fair or poor by only 10.7%. Reported satisfaction with access to care was high; average ratings were 78.5% for “not a problem.” For the patient satisfaction subscale, Cronbach’s reliability was .86 and for access to care it was 0.73.

Table 3 Risk Adjusted Outcomes of Care (n = 23 Sites)
Full size table

Practice Systems of Care

Table 4 summarizes data from the practice systems survey. Cronbach’s alpha reliability coefficient for the 6 subscales are .85, .78, .80, .70, .88, and .81, respectively, whereas the overall practice system score has a Cronbach’s alpha coefficient of .94.

Table 4 Practice Systems of Care (n = 23 Sites)
Full size table

For each category, on average, about half were reported as operating well in practice. However, as is apparent from the overall score range (18.0–81.0%), some sites had few, whereas others had many systems elements considered ideal for delivering quality care.21–26 More quality environment elements were working well (54.8%) than care management elements necessary in caring for complex patients (41.3%). Even “low-tech” elements were often reported as functioning poorly; for example, 10 clinics (45%) did not have well-functioning standard problem lists in medical records.

Variation Among Program Sites

A final set of analyses explored if variability in program site performance on appropriate intervention (Table 2), adjusted outcomes of care (Table 3), or presence of systems of care (Table 4) was related to size and presence of an EMR. Smaller clinic sites (n ≤ 32 residents) had higher modes of communication scores than larger sites (69.9% vs 48.6%, P = .001). Clinic sites with an EMR had higher information management (80.7% vs 27.34%, P < .001) and modes of communication scores (79.4% vs 43.3%, P < .001).

Figure 1 shows how sites varied across 6 summary variables. Five patient variables include the mean percentage of prevalence of CVD (Table 1), the mean percentage of prevalence of risk factors for CHD (Table 1), the mean percentage of appropriate interventions (Table 2), adjusted mean percentage of clinical outcomes meeting targets, and adjusted mean percentage of satisfaction with practice as excellent (Table 3). The sixth variable is the percentage of system elements that are operating well in practice (Table 4). Prevalence of CVD was related to prevalence of risk factors for CHD (r = .77, P < .001) and to adjusted outcomes of care (r = −0.58, P = .004). Prevalence of risk factors for CHD was also related to appropriate interventions (r = .58, P = .004). It may be that the more risk factors a patient has, for example in sites 19–23, the more attention the practice pays to the need for interventions such as diet, exercise, and smoking cessation. Notably, systems of care and adjusted patient satisfaction were not related to any other summary variables.

Figure 1
figure 1

Overall measures by prevalence of risk factor for CHD. Pink diamond Patient satisfaction—excellent, green triangle clinical outcomes—target met, blue diamond intervention—appropriate, orange square overall systems score, brown circle prevalence of CVD, dark blue dash prevalence of risks factors for CHD.

Full size image

DISCUSSION

The quality of the ambulatory care experience and the characteristics of patients of internal medicine residents are described using a web-based tool that assessed the quality of the clinic’s microsystem and the preventive cardiology care provided in their weekly continuity of care practices. Quality of care was measured by a set of evidence-based process and outcome measures, patient satisfaction, and presence of practice-based systems known to support quality care. The variability in performance on these measures differed for certain program features (size, type, and presence of an electronic record).

Most notable was the variability in clinic practices among the 23 training sites. The good news is that nearly half (10 of 23) of the clinic sites have EMRs, compared to less than 20% of practicing physicians nationally.27 As anticipated, when an EMR was present, there was better overall information management in the clinic (81% vs 27%) as well as better systems for patient access and communication between visits (79% vs 43%). Interestingly, there was also better access and use of multiple modes of communication in smaller as compared to larger sites (70% vs 49%). Surprisingly, neither process nor outcome quality measures differed with the presence or absence of an EMR.

The low frequency of functional basic “technologies” in many of the sites is salient. For example, the mean score for systems to facilitate information management about chronic illnesses (e.g., problem lists, templates, and reminders) was only 48%, with a sobering range of 0% to 92% among sites. Likewise, the mean score for patient activation and communication, a measure of systems for promoting self-care, was only 46%, with a range of 22% to 83%. Literature suggests that a better understanding of the elements that comprise functional systems will help to improve processes of care.17 Residents then who work in an effective system and understand the essential concepts of systems-based practice are more likely to be adequately prepared to engage in or implement more effective systems upon graduation from residency.

Process and outcome measure performance of quality in preventive cardiology care varied widely and was sometimes disturbingly low. Appropriate use of guideline-based medications ranged from 33% to 100%. Compliance with lifestyle interventions showed the greatest variability across sites and processes (7% for dietary sodium restriction to 91% for smoking cessation support). Overall, patients received an average of 57% (range, 30% to 77%) of the appropriate interventions, surprisingly similar to a national analysis of quality at the patient level.28

This study importantly examined quality from the perspective of patients’ care experience and showed relatively little variability in responses between programs (Table 4). Patients rated access relatively high (79%), but results show ample opportunities for improvement in patient communication and satisfaction.

What do these results tell us about the current state of residency experience in preventive cardiology care in the ambulatory setting? Although direct comparisons are not possible,29 many challenges in delivering high quality care in training sites mirror those in the external practice environment.24,25 Variability in practice-based systems and patient populations among ambulatory training sites has heretofore not been systematically documented in a multi-site study. The extent of variability is of concern and, although perhaps expected, has been underappreciated.

Interestingly, this variability did not track with the type of residency program nor were patient satisfaction and practice systems of care related to other quality measures. Factors such as faculty supervision, faculty or trainee knowledge, local micro-systems, institutional competence and attitudes, practice patterns, or some unmeasured aspect of the patient population (i.e., presence or absence of insurance) were not studied and may explain some of the variability.

This study has several limitations. First, the programs were not randomly selected, and these findings may not be able to be generalized to other training programs, but this is one of the first studies to examine care and experience across multiple and diverse residency programs. Furthermore, this selection bias would conceivably favor programs with better performance, but the wide range of performance speaks to the contrary. Second, most data came from patient charts, selected and abstracted by residents, but there was no obvious incentive to over-report performance because resident-level identification was not tracked and data were not verified. However, in a study of ABIM’s diabetes practice improvement module, strong agreement was found between trained abstractors and physician audits of the same medical records.30 Furthermore, low rates of compliance for many measures further suggest that significant gaming did not occur. Third, these programs did receive modest compensation for participation. Fourth, the available risk adjusters are inadequate; socioeconomic and race classifications and a more robust measure of co-morbidity are necessary.

To date, this is one of the largest studies of ambulatory care practices among a diverse group of residency programs, involving over 700 residents and 4,700 patients. The study provides a methodological approach for gaining insight into the current state of practice in the residency sites including aspects of the clinic’s environment and the quality of preventive cardiology care. More research is needed to help faculty and institutions understand the variation in systems and patient care in the training environment with the ultimate goal to deliver effective care to the unique population.