The Patient Centered Medical Home (PCMH) is a model of comprehensive primary care delivery that many believe will improve the quality, access, and efficiency of our health care system.1 An important setting to test the PCMH model is safety-net clinics, including clinics that are part of Federally Qualified Health Centers (FQHC’s) as well as other community-based clinics. Since these clinics provide primary care for over 18 million medically underserved patients, implementing the PCMH model could reduce health disparities.2,3 Several initiatives are already implementing the PCMH model in safety-net clinics, including the Centers for Medicare and Medicaid Services’ (CMS) Federally Qualified Health Center Primary Care Practice Demonstration in 500 FQHCs.4

Implementation of the PCMH in safety-net clinics would benefit from an evaluation tool that measures PCMH adoption. While good tools are available that are useful for many purposes, each has limitations for research evaluation in safety-net clinics. For instance, tools used in PCMH pilots do not fully cover domains relevant for the safety-net setting or are burdensome to complete.5,6 One of the most widely used tools for PCMH recognition is the National Committee for Quality Assurance (NCQA) Physician Practice Connections ®-Patient-Centered Medical Home™ (NCQA PPC-PCMH).7 However, the 2008 and 2011 NCQA versions may have limited feasibility in a research setting because the required supporting documentation frequently takes 20–30 h or more to complete. In addition, some have argued that the NCQA PPC-PCMH overemphasizes health information technology.8 Also, this instrument consists primarily of dichotomous items, which are not optimal for detecting variation in PCMH. The NCQA has a research version of its tool with lower response burden and more scaled options, but it lacks some key domains for the safety-net setting such as interpretation services and access to specialty care for uninsured patients.9–11 Also, the tool is not publicly available. Similarly, the Patient Centered Medical Home Assessment (PCMH-A), created by the MacColl Institute for Healthcare Innovation at Group Health and Qualis Health, asks about obtaining specialty referral, but not specifically for uninsured or Medicaid patients.12 Table 1 summarizes currently available tools that have been used in medical home evaluations or those that have been specifically designed for safety-net clinics.5–7,13,14

Table 1 Patient-Centered Medical Home (PCMH) Adoption Tools Used in Research Evaluation or Designed for the Safety-Net Setting: Attributes Suitable for Longitudinal Analysis in Safety-Net Setting
Full size table

For longitudinal research and formative quality improvement, an ideal tool should be comprehensive, have a low response burden, and be publicly available. In addition, inclusion of scaled response options would improve detection of variation and may help detect change over time. Also, to provide a concise comparison of PCMH adoption, the tool should provide a summary score. Finally, to capture the comprehensive care many safety-net clinics try to provide, the scale should include items about providing language services and coordinating care for underserved patients.15 Based on these criteria, we developed the Safety Net Medical Home Scale to evaluate PCMH interventions in safety-net clinics. We tested it for reliability and convergent validity. Then we used the scale to describe PCMH adoption in safety-net clinics beginning a PCMH intervention. Finally, we determined clinic-level factors associated with PCMH adoption.

METHODS (ADDITIONAL METHODOLOGIC DESCRIPTION FOUND IN APPENDIX 1: TECHNICAL APPENDIX-AVAILABLE ONLINE)

We conducted a cross-sectional survey in 65 safety-net clinics in five regions (Colorado, Idaho, Massachusetts, Oregon, Pennsylvania) and developed a measure of PCMH adoption based on this survey.

SURVEY SAMPLE

The sample was 65 safety-net clinics taking part in the Safety Net Medical Home Initiative, a 5-year demonstration project supported by the Commonwealth Fund to implement the PCMH in safety-net clinics.16 Implementation is led by Qualis Health and the MacColl Institute using “change concepts” for practice transformation that are based on PCMH principles and tailored to the safety-net setting.17

Clinics were chosen by an application process. The Commonwealth Fund, Qualis Health, and the MacColl Institute requested applications from existing Regional Coordinating Centers (RCCs) that either had existing relationships with safety-net clinics or partnered for the purpose of this demonstration. RCC’s were required to have stakeholder committees, previous experience in quality improvement, and support from their state Medicaid offices. Clinics were also required to have previous experience with quality improvement interventions. Five hundred fifty-four clinics affiliated with 42 RCCs in 31 states submitted applications. The final study sample included 65 clinics affiliated with 44 distinct health center organizations. The centers were affiliated with five RCCs located in five states: Colorado, Idaho, Massachusetts, Oregon, Pennsylvania. (Table 2)

Table 2 Characteristics of Clinics
Full size table

SURVEY INSTRUMENT

We used the “Commonwealth Fund Organizational Survey of Federally Qualified Health Centers.” This survey was developed by investigators at the Commonwealth Fund and University of Chicago with input from a national advisory committee of health center clinicians, administrators, and policymakers. The survey was designed to provide an overview of health center capability and provide individual questions to create a PCMH evaluation scale. The survey included new items and items adapted from previous surveys, such as the NQF-Endorsed™ Definition and Framework for Measuring Care Coordination, Electronic Health Records in Ambulatory Care, and Adoption of Health Information Technology (HIT) among Community Health Centers.18–20 Pre-testing was done with six respondents, and the survey was revised to improve its clarity, comprehension, flow, and timing. The resulting 12-page survey includes 92 items with a variety of ordinal response options (34 dichotomous, 27 3-point Likert, 6 4-point Likert, and 25 5-point Likert). It covers the following domains: access to care, language services, quality improvement, and patient information systems. 21

SURVEY ADMINISTRATION

The survey was distributed by mail at the baseline of the PCMH initiative between July and December 2009. It included instructions to be completed by health center leadership with help from other staff as necessary.

SCALE CREATION

We (the manuscript authors) used the Commonwealth Fund survey items to develop the Safety Net Medical Home Scale (SNMHS) in an iterative, consensus process. We first categorized survey items onto eight domains of the 2008 NCQA PPC-PCMH instrument: Access and Communication, Patient Tracking and Registry, Care Management, Patient Self-Management Support, Electronic Prescribing, Test Tracking, Referral Tracking, and Performance Reporting and Improvement.22 In addition, some survey items were categorized into a new domain, External Coordination, because arranging specialist and inpatient care is believed by many to be essential for patient care and the PCMH.23 Next, domains were consolidated using content validity and internal consistency reliability as criteria. Also, content validity was used to identify core items of key importance to the PCMH. This conceptual mapping yielded 52 items, including 16 core items, that were organized into six domains: Access and Communication (12 items, 4 core items), Patient Tracking and Registry (7 items, 3 core items), Care Management (8 items, 2 core items), Test and Referral Tracking (4 items, 2 core items), Quality Improvement (10 items, 2 core items), and External Coordination (11 items, 3 core items).

To calculate domain scores and a global score, we rescaled item scores. First, we rescaled each of the 52 selected items to have a potential range from 0 to1. Then the 16 core items were differentially weighted by rescaling these items to have potential range of 0 to 2. Scores for each domain were generated by summing the rescaled weighted items in the domain and then rescaling the total score to have a potential range 0 to100 (worst to best). Finally, the total PCMH score was calculated as the mean of the six domain scores, yielding a total score with potential range 0 to 100.24

CLINIC CHARACTERISTICS

The Commonwealth Survey provides clinic-level data on urban versus rural setting, presence of electronic medical health record (EMR), region, the total number of provider Full Time Equivalents (FTE), participation in financial incentives programs for quality improvement, presence of nurse shortages, and percentage of patients with limited English proficiency [Appendix 1 (available online)]. In addition, clinic-level data, such as FQHC designation and payer mix of patients, were collected by Qualis Health. Characteristics of the clinics’ patient populations were obtained from Uniform Data System (UDS) reports that FQHCs are required to report.2 The UDS data were only available at the center level.

STATISTICAL ANALYSIS

Scale Properties

Reliability

To test internal consistency reliability, we calculated the Cronbach’s alpha for all items in each domain and all 52 items that comprise the total scale.

Convergent Validity

The clinics completed two survey tools that have content validity for PCMH concepts: the Assessment of Chronic Illness Care (ACIC) tool and the PCMH-A.

The ACIC is a tool to measure the adoption of the Chronic Care Model, which is conceptually aligned with the PCMH model.25,26 The survey consists of 34 items in seven domains with a score range of 0–11. The ACIC was distributed by Qualis Health and MacColl Institute to clinics as part of the baseline assessment in 2009 with instructions to be filled out by front-line staff. The response rate was 93%. The PCMH-A 12 is composed of 33 items in eight domains ranging in score from 1–12. The survey was distributed by Qualis Health and MacColl Institute to the clinics as part of the baseline assessment with instructions to be filled out by front-line staff. The response rate was 100%. We computed the correlation between the total SNMHS score and the scores for each of these surveys.

Degree and Variation of PCMH Adoption

We computed descriptive statistics for SNMHS domain scores and total SNMHS score.

Sensitivity Analysis

While the external coordination domain is essential to patient care, centers may have limited control of obtaining off-site specialty care for their underserved patients.27 Therefore, we performed a sensitivity analysis by computing an alternative scoring algorithm that omitted the external coordination domain.

Correlates of PCMH Adoption

We explored the association between PCMH adoption and clinic characteristics that may influence the capability to provide high quality care. Clinic characteristics examined were geographic setting, presence of EMR, provider shortage, nurse shortage, participation in financial incentives for practice improvement, total provider Full Time Equivalent (FTE), and characteristics of patients served at the clinics (race, ethnicity, English proficiency, insurance). Provider FTE was analyzed as a categorical variable, with categories <4, 4 to 8, and >8 FTEs. Patient characteristics were aggregate percentages and were modeled as continuous covariates. All other covariates were dichotomous.

We first tested for differences in total SNMHS score by region using analysis of variance. We then fit mixed effects linear regression models with the total SNMHS score as the dependent variable and each clinic characteristic as the independent variable. Region was included as a random effect to account for the nesting of clinics within RCCs. To determine the final multivariable model, we used a backwards-stepwise regression approach to remove variables that had the highest p-value and did not significantly change the other coefficients, until all variables had a p-value less than 0.10. All analysis was conducted using Stata 11.0 (College Station, TX). This study was approved by the University of Chicago Institutional Review Board.

RESULTS

Response Rate: After two waves of administration, the response rate was 100%. In most cases (71%) more than one staff member contributed to survey completion. The primary respondent included Executive Director (21%), Medical Director (20%), Director of Nursing (3%), Chief Financial Officer (2%), and other clinical operations personnel (54%). Eighty-two percent of clinics were FQHC or FQHC Look-a-Like, while 18% were other community clinics serving underserved populations.28 Seventy-one percent had an EMR, and 42% were in a small town, rural, or frontier location (Table 2).

SCALE PROPERTIES

Reliability

The SNMHS demonstrated high internal consistency reliability: Total Score (Cronbach’s alpha = 0.84), Access and Communication (alpha = 0.68), Patient Tracking and Registry (alpha = 0.89), Care Management (alpha = 0.60), Test and Referral Tracking (alpha = 0.73), Quality Improvement (alpha = 0.73), and External Coordination (alpha = 0.78).

Convergent Validity

The total SNMHS score correlated with the total ACIC score (R = 0.64, p < 0.001) and the total PCMH-A score (r = 0.56, p < 0.001).

SNMHS PERFORMANCE

The mean of the SNMHS total summary score was 61 (SD 13) on a measurement scale of 0 to 100. (Figure 1). Removing the External Coordination domain resulted in a slightly lower mean total summary score of 60 out of a possible 100. External Coordination (66) and Access and Communication (65) had the highest mean scores, and Quality Improvement (55) and Care Management (55) had the lowest mean scores.

Figure 1
figure 1

Safety Net Medical Home Scale Total Summary Score and Domain Scores (N = 61). The horizontal line in the box represents the median value, and the edges of the box are the first and third quartiles. The lines extending from the edges of the box indicate the 10th and 90th percentile. Domain scores calculated from Safety Net Medical Home Scale [Appendix 1 (available online)]. Total score calculated from mean of other six domains. A clinic's data were included in a domain score if the clinic responded to more than 50% of the items in the domain and more than 50% of the core items in that domain. In addition, to be included in the total PCMH score, clinics had to have more than 50% of items answered for all domains and more than 50% of core items present for all domains. For surveys that had missing items but had fewer than 50% missing items in a domain (n = 9), we imputed the missing values based on the average score of the rest of the domain items. Four centers were excluded from analysis because of missing data, leaving 61 (94%) clinics with total scores calculated.

Full size image

Analysis of individual survey items revealed that clinics performed well in tracking laboratory tests, collecting and reporting clinical outcomes at the provider level, and collecting patient satisfaction surveys at the group level (Table 3). However, clinics scored lower at providing same or next-day appointments, generating a list of patients who are due for tests or preventive care, tracking specialist referrals, and arranging timely specialty appointments for the uninsured and patients with Medicaid insurance.

Table 3 Responses to Core Items by Domain (n = 65)
Full size table

CORRELATES OF PCMH ADOPTION

The SNMHS score varied by region (range 51 to 67, p = 0.01). In univariate analyses, the total SNMHS score was positively correlated with having more than eight provider FTEs, city/suburban location, and participation in financial incentive programs (Table 4). Characteristics of patients served at the clinic level, including insurance, race, and ethnicity, were not associated with PCMH adoption. In multivariable analysis, having more than eight provider FTEs and participation in financial incentive programs remained significant.

Table 4 Clinic Correlates of Safety Net Medical Home Scale Total Score
Full size table

DISCUSSION

The Safety Net Medical Home Scale is a comprehensive, practical scale to measure PCMH adoption in the safety-net setting that adds to strengths in previous scales.14,29,30 The SNMHS has a low response burden, yet still provides a thorough measurement of PCMH domains. Scaled responses provide increased discrimination to detect variation in the sample. While the scale has not been tested in a longitudinal setting, we believe the scaled responses would be more sensitive to change over time than dichotomous responses. In addition, our scale demonstrated reliability as measured by Cronbach’s alpha and convergent validity through correlation with the ACIC and the PCMH-A scales. Overall, we believe our scale is a useful addition because it is brief, comprehensive, publicly available, and suitable for longitudinal analysis.

In our study, the average baseline SNMHS total summary score of 61 suggested that clinics have good foundations for the PCMH model. However, as in studies of PCMH adoption in private practices, room for improvement exists in all domains.31–34 In addition, we found that smaller practices may need special attention to help them succeed in building PCMHs. Some PCMH activities, such as improved access to after-hours clinical advice, may require additional staffing and could be especially challenging for practices with few personnel.35

Going forward, the scale could be used for several purposes. While tools such as the NCQA research version may be appropriate for many PCMH evaluations, we believe our scale is especially relevant for the safety-net setting. The items capturing language services and coordinating care for underserved patients will help evaluate PCMH interventions in safety-net clinics, such as the CMS FQHC Primary Care Practice Demonstration4. Evaluating this intervention with a medical home tool that does not capture these services could lead to an inappropriate assessment of medical home adoption. In addition, the Safety Net Medical Home Scale may identify areas that require broader coalitions of stakeholders to address. This sample of clinics scored well in the External Coordination domain. However, improving this domain requires external partnerships with other providers in the local health system since the amount of charity care that specialists and non-profit hospitals provide depends on Medicaid reimbursement rates, disproportionate share funding, and legal requirements. 27 Therefore, the Safety Net Medical Home Scale could provide information useful for guiding the wider health care system about how to establish incentives and allocate resources to coordinate care for underserved populations. Finally, the scale should be tested in non-safety-net settings, since the scale has content validity for most outpatient settings.36,37 A good setting to start may be other small practices that may face similar challenges to safety-net clinics.

Our scale has several limitations. First, while the SNMHS has a lower burden than the NCQA recognition process, it relies on self-report. We have validated the SNMHS against the ACIC and PCMH-A surveys, and in the future we will correlate the scale with measures of clinical performance and efficiency. In addition, since the tool only goes to a single respondent per organization, there may be variation in the survey’s reliability across sites. While the survey will go back to the same person over time to increase reliability, it will be difficult to test the reliability of responses since medical home adoption will likely have changed at the end of the intervention. Brevity comes at the price of completeness, and our scale focused on measuring NCQA PPC-PCMH domains plus other domains that seem particularly important for safety-net clinics. Thus, other factors relevant for PCMH development may not be measured. For instance, PCMH domains such as whole person-orientation and team approach to care are not fully captured, and concepts critical for PCMH implementation such as engaged leadership are not completely measured either. Also, the cross-sectional design limited the ability to test specifically for sensitivity to change. However, we did attempt to enhance sensitivity over prior tools. In our convergent validity analyses, the ACIC and PCMH-A do have content overlap with the SNMHS, so some correlation would be expected. However, these instruments were designed for different purposes, include different domains, and the items do not completely overlap. Thus, they are reasonable scales to use to test for convergent validity with the SNMHS.

In addition, the baseline assessment of this clinic sample may not be generalizable. These clinics have established partnerships with regional coordinating centers, which may reflect local policies and culture towards care coordination, and also expressed an interest in PCMH adoption. Also, there are limitations to using UDS data in our analysis of correlates of PCMH adoption. UDS data are center specific, whereas the scale data are site specific, so there is the potential for mismatch when pairing these data sources. However, we used UDS data only for the racial and ethnic composition of the clinics’ patient populations.

In summary, we have developed the Safety Net Medical Home Scale and demonstrated its ability to provide a detailed description of PCMH adoption in a sample of safety-net clinics. The Safety Net Medical Home Scale can be a valuable tool for research evaluations of PCMH adoption in safety-net clinics.