INTRODUCTION

Quality measurement and reporting programs are now widespread.1–4 Clinical performance measures such as those defined in the Health Plan Employer Data and Information Set (HEDIS) are used increasingly for public reporting5–7 and pay for performance programs.8–10 These widely recognized clinical measures of quality include both processes of care (e.g., rates of screening hemoglobin A1c exams) and outcomes of care (e.g., proportion of patients achieving specified levels of hemoglobin A1c control).

Valid measures of the patient’s experience of care have also appeared as an integral part of quality measurement and improvement initiatives by provider organizations, large payers’ pay for performance programs,8 and public reporting efforts.5 The Ambulatory Care Experiences Survey (ACES) is one such measurement tool developed specifically for use in the assessment of patient experiences of primary care,11 providing an assessment of both organizational features of care and the quality of clinical interactions.12

As physician groups and individual physicians strive to deliver care that is both clinically effective and patient-centered,13,14 it is increasingly important to improve our understanding of the relationship between measures of clinical performance and patient experience. Physicians may express concern that conflicts arise as systems are implemented to standardize care across all patients to assure outstanding clinical performance, while the delivery of patient-centered care may ultimately rely on customizing care to suit the needs of a particular patient.15 At the same time, consumers of performance data, including purchasers, insurers, and patients, would benefit from knowledge of whether different methods of performance assessment provide equivalent information or whether multiple independent tools are required to obtain a complete picture of health-care delivery.

We used statewide data in Massachusetts to evaluate the association between clinical measures of quality and measures of patient experiences at the level of the physician practice site and the individual physician.

METHODS

Overview

To evaluate the correlations between clinical performance measures and patient experience survey measures, we separately analyzed data from (1) a statewide cohort of 373 practice sites and (2) a cohort of 118 individual physicians that are part of one large physician organization in Eastern Massachusetts.

Practice Site Data

Data on practice site performance were obtained through the Massachusetts Health Quality Partners (MHQP), a broad-based coalition of physicians, health plans, and purchasers.6 MHQP derives its patient samples from the enrollment databases of five participating health plans, which together cover over 4 million Massachusetts residents, or 85% of managed care enrollees in Massachusetts. As part of the MHQP initiative, primary care physicians were linked to distinct practice sites for public reporting purposes. Practice sites were defined as a physical location where a physician confirmed he or she practiced as of December 31, 2004. To be included in the sample, practice sites must have had at least 3 eligible physicians with a panel size of at least 50 eligible patients across the five health plans. The eligible practice sites together represented 92% of licensed primary care physicians in the state.

MHQP collects clinical performance measures according to standard HEDIS specifications (Table 1) using either claims data alone (process measures) or a hybrid method of claims data and medical record review (outcomes measures). To calculate HEDIS performance scores for each physician, the denominator is equal to the number of opportunities an individual physician has to satisfy a given HEDIS measure, and the numerator is equal to the number of times that HEDIS measure is satisfied. Because the numerators for the process measures are derived from claims data alone, they are multiplied by a claims adjustment factor, which is derived by comparing the difference between the health plan’s HEDIS performance rate based on administrative data alone with the health plan’s rate derived from the hybrid method (combination of claims and medical record review). There is no claims adjustment factor applied to clinical outcomes measures as they are derived solely from medical record review.16

Table 1 Measures of Clinical Quality
Full size table

Patient experience data were derived from MHQP’s administration of the ACES instrument, which produces summary measures covering quality of clinical interactions and organizational features of care (Table 2).12 The survey sample consisted of adults 18 to 64 years of age with at least 1 year of continuous enrollment in one of five participating health plans, and at least one visit with the primary care physician in the last 12 months. The survey was fielded between July and September 2005 using a three-stage mail protocol,17 beginning with a mailing containing a cover letter on their health plan stationery and a copy of the survey, followed by a reminder post card (2 weeks later) and a second survey packet to non-respondents (1 week later). The cover letter provided the option of completing the survey online and afforded a web address and unique respondent log-in information. The overall response rate was 39% (n = 61,683), producing patient experience data for 373 (75%) practice sites statewide. Prior analyses of this survey process have confirmed that survey non-response does not pose a threat to the validity of the findings, with no differences across physician practices in individuals’ propensity to respond to the survey according to age, race, education, and socioeconomic status.12 We limited our analyses to practice sites with data available on both patient experiences and HEDIS measures (n = 334).

Table 2 Measures of Patients’ Experience
Full size table

Individual Physician Data

Data on individual physician performance were obtained from Harvard Vanguard Medical Associates, an integrated group practice consisting of 14 ambulatory health centers in eastern Massachusetts, with approximately 120 primary care physicians caring for approximately 300,000 adult patients. HVMA has used a fully functional electronic medical record system, EpicCare Ambulatory (www.epicsystems.com) since 1997, which allows for the electronic capture of all aspects of outpatient encounters, including clinician notes, diagnostic codes, procedure codes, and all laboratory results. The HVMA electronic record system assigns a unique primary care physician to each patient, and data from this medical record system have been used extensively in formal analyses of quality of care.18–22 We supplemented electronic medical record system data with claims data to assess colorectal cancer screening and diabetic eye exams, as these services are sometimes received outside of the HVMA network. We applied HEDIS criteria to identify appropriate denominator and numerator populations using diagnostic and billing codes within the electronic medical record during 2005.23 As HVMA does not maintain membership enrollment similar to health plan enrollment databases, we restricted the inclusion of patients in all denominator populations to those with a primary care physician visit within the past 36 months.

HVMA administered the ACES instrument to patients of all adult primary care physicians on a quarterly basis with sample sizes defined to achieve a 0.70 Spearman Brown-derived reliability at the individual physician level. All primary care physicians with at least a 50% full-time equivalent practice are included in the survey process. Surveys were administered to patients who had an office visit within the past 12 months with the primary care physician and used the same three-stage data collection protocol as used by MHQP. This study used survey data obtained between January 2005 and December 2005, and had a 40% (n = 9,120) response rate. Patient experience data were available for all 119 (100%) primary care physicians maintaining at least a 50% full-time equivalent practice at HVMA.

Composite Measure Creation

We created three composite measures of clinical quality and seven composite measures of patient experiences (Tables 1 and 2). Composite measures of clinical quality were developed by first separating process of care measures from clinical outcomes, as these represent distinct concepts of quality.24 Finally, we categorized process measures based on whether they focused on preventive health or chronic disease management, as these activities likely represent different care processes and may be targeted by independent quality improvement strategies.

The patient experience composites include three measures of the quality of clinical interactions (doctor/ patient communication, clinical team interactions, and health promotion support) and four measures of organizational features of care (integration of care, office staff, visit-based continuity, and organizational access).12

Numeric composite scores for both patient experience and clinical quality measures were calculated using the adjusted half-scale rule to produce ratings on a scale from 0 to 100, with higher scores representing either better patient experiences or superior clinical quality. This method has been described previously,25 and involves first transforming the individual item to a 0 to 100 scale based on the number of responses. HEDIS scores are already ranked as such, while patient survey responses require conversion (e.g., a six-point Likert response item would generate a score of 0, 20, 40, 60, 80, or 100). Scores were adjusted by adding or subtracting a constant for each item so that all items have the same mean (the average of the overall mean of all items in the scale), satisfying a key assumption of the half-scale.

Correlation Analyses

We calculated Spearman correlation coefficients between each clinical quality composite measure and each patient experience composite measure at the practice site and individual physician level. We further defined the convergence estimates for each of these correlations to account for the limitations in the reliability of the composite scores. These convergence estimates, or adjusted correlation coefficients, represent the true correlation that would be observed between the two composite scores in the absence of measurement error associated with creating the composites. The convergence estimates were calculated using the Spearman-Brown reliability coefficients for the patient experience composite (α1) and the clinical quality composite (α2), along with the Spearman correlation coefficient (ρ) in the following manner: \({\text{Convergence = }}\rho /\!{\sqrt {\alpha _{{\text{1}}} \times \alpha _{2} } }\).

This study was approved by the Human Studies Committees at Harvard Vanguard Medical Associates, Tufts-New England Medical Center, and Harvard Medical School. All analyses were carried out using the STATA statistical package, version 9.0.

RESULTS

Clinical performance for the process of care prevention measures was generally comparable for the practice group and individual physician level data, with the exception of colorectal cancer screening and the diabetes care process measures (Table 3). The aggregate patient experience ratings were comparable for the practice group and individual physician level data, with the exceptions of health promotion support and office staff (Table 4).

Table 3 Clinical Quality Composite Measures
Full size table
Table 4 Patient Experiences of Care Composite Scores
Full size table

Among 42 possible HEDIS-ACES composite correlations (21 involving practice sites and 21 involving individual physicians), the majority were positive in both site level (71%) and physician level (67%) analyses (Table 5). There were no significant correlations between composite measures of patient experiences and clinical outcomes at either the practice site or individual physician level.

Table 5 Adjusted Correlations Between Patient Experiences of Care and Clinical Quality Composites*
Full size table

Within the practice site cohort, the HEDIS process-health prevention composite was positively correlated with one ACES measure of clinical interactions (clinical team interactions,adjusted ρ = 0.15, p = 0.02) and one ACES measure of organizational features of care (integration of care, adjusted ρ = 0.13, p = 0.04). The HEDIS process-disease management composite was positively correlated with the same ACES measures of clinical team interactions (adjusted ρ = 0.13, p = 0.01) and integration of care (adjusted ρ = 0.15, p = 0.03). The HEDIS process-health prevention composite was also negatively correlated with with one ACES measure of organizational features of care (visit-based continuity, adjusted ρ = -0.12, p = 0.03).

Within the individual physician cohort, the HEDIS process-health prevention composite was positively correlated with two ACES measures of clinical interaction quality (doctor/patient communication, adjusted ρ = 0.284, p = 0.02; and clinical team interactions, adjusted ρ = 0.512, p = 0.01), and one ACES measure of organizational features of care (organizational access, adjusted ρ = 0.390, p = 0.01). Three additional correlations between the HEDIS health prevention composite and ACES composite measures were of borderline statistical significance (health promotion support, adjusted ρ = 0.257, p = 0.06; integration of care, adjusted ρ = 0.254, p = 0.07; and office staff, adjusted ρ = 0.373, p = 0.06). The HEDIS process-disease management composite was negatively correlated with one ACES measure of organizational features of care (clinical team interactions, adjusted ρ = -0.468, p = 0.02).

DISCUSSION

Enhancing the quality of care in the ambulatory setting includes assuring the delivery of clinically effective care in a patient-centered manner.26,27 In this statewide evaluation, we found that measures of patient experiences were positively, but modestly correlated with process measures of clinical quality at both the practice site and individual physician level, with only one-third of these correlations achieving statistical significance. There were no significant correlations between patient experiences of care and clinical outcomes among practice sites and individual physicians.

Our study is consistent with prior research on the association of clinical performance and patient experience measures at the level of health plans, which demonstrated that better performance on some, but not all, patient reports of care experiences were associated with higher scores on clinical HEDIS measures.28 The magnitude of the correlations we observed were generally modest and similar in magnitude to previously reported correlations at the health plan level, which ranged from 0.24 to 0.38.28 The absence of overwhelmingly strong correlations in our study suggests that clinical care delivery and patient experience represent sufficiently distinct activities that ongoing quality measurement programs should include independent measurement of both domains in order to obtain a comprehensive evaluation of care. Patients using such data to select a primary care physician may need to make trade-offs between technical performance and interpersonal performance.29 Similarly, pay for performance programs will likely need to incorporate both aspects of health-care delivery to achieve broad-based improvements in care.

The modest correlations we observed suggest that improvements in clinical quality will not automatically produce improvements in patient experiences and vice versa.4,30 Monitoring both patient experience and clinical quality can ensure that efforts to improve patient experiences of care do not come at the expense of assuring the highest possible clinical performance.15 Importantly, the majority of all of the correlations examined were positive, with no evidence to support the notion that delivering high quality clinical processes is somehow in conflict with positive patient experiences in the primary care setting.

We found no association between patient experiences of care and clinical outcomes. Earlier research suggests that better patient experiences may improve patient outcomes through improved adherence.31–35 Our null findings may reflect aspects of our study design, such as the cross-sectional nature or the limited sample sizes for the outcomes measures. However, the absence of correlations between patient experiences of care and clinical outcomes parallels findings from recent studies demonstrating the lack of a consistent relationship between improvements in clinical process measures and clinical outcomes.3,21,36 Rather than reflecting a limitation of measures of patient experiences, our findings reinforce the inherent difficulty of linking process to outcome and that existing measures of process, including patient experiences of care, may not represent the full spectrum of factors that contribute to improved outcomes.

Our findings are strengthened by several aspects of the study design. Our data focused on care delivered by doctor’s offices and individual clinicians, including a statewide sample of practice sites and a large number of individual physicians across eastern Massachusetts. We were also able to include a wide range of measures of clinical quality and outcomes and patient experiences that were similar at both the practice site and individual physician level. However, our findings should be interpreted in the context of study design limitations as well. The availability of a large number of quality measures necessitated multiple testing, increasing the probability of detecting significant correlations by chance alone. However, our study goal was focused more on overall patterns rather than the importance of any single correlation.

While we included practice sites throughout Massachusetts, our sample was dependent on enrollment in one of five major commercial health plans and so may not generalize to other populations, particularly those lacking health insurance or the elderly. At the individual physician level, we were only able to include data from a single multispecialty practice group, though it is important to note that this is the largest ambulatory practice group in the state, providing care for over 300,000 patients and employing over 100 primary care physicians across 14 health centers in a mix of urban and distant suburban settings. In addition, while actual performance on HEDIS and ACES measures may not generalize across different health settings, there is no evidence to suggest that the relationship between these two domains of care differs across settings.

We are unable to draw conclusions regarding the mediators of the correlations we identified. In particular, we do not have information regarding structural characteristics of the practices, as well as individual characteristics of the physicians and patients in our study. It is very likely that specific attributes of a practice or physician mediate the strength of correlations between patient experiences of care and clinical measures of quality, and future research will be needed to explore this important question.

The definition and collection of patient experience measures were identical at the practice site and individual physician level; however, the collection of the clinical HEDIS measures differed between the two levels. At the practice site level, we relied mainly on health plan administrative data supplemented in some cases by chart review, while at the individual physician level we used mainly electronic medical record data. This resulted in a limited number of practice sites available for analysis of clinical outcomes, which are reliant on medical chart review. However, since our analyses were focused on determining correlations within these two levels (practice site and individual physician) and not between these two levels, the analyses are internally consistent at each level and are not substantially affected by differences in data collection for the clinical HEDIS measures. Finally, we were unable to correlate HEDIS chronic disease measures to patient experience measures for patients with those specific conditions, a technique that may have identified stronger associations.

In conclusion, we found that patient experiences of care were modestly correlated with clinical process measures and not correlated with clinical outcomes. Improving both patient experiences of care and clinical process measures should remain high priorities for health-care providers; however, the limited strength of the associations implies that distinct efforts in these areas are required to both monitor and improve these two domains of quality.