Introduction

Many papers examining implementation of evidence-based programs have reported on fidelity to intervention protocols and consistency of delivery of key intervention program components [1, 2]. This makes good sense, given that one of the primary reasons for program failure is the lack of ability to replicate results of successful programs when applied in community and clinical settings [3]. That is, failure to implement the intervention as designed. This, however, is only part of the picture, as implementation science and behavior change literatures have also documented the importance of context and of tailoring interventions to local settings. Indeed, there is evidence that successful implementers are those who know how to successfully adapt an intervention to specific settings and circumstances [4, 5]. Therefore, there is a need to study how programs, policies, and quality improvement efforts are adapted from an original protocol or plan, and for frameworks to understand and measures to assess these adaptations [6]. The purposes of this paper are to investigate (a) the characteristics and perceived impact of different types of adaptations in the context of quality improvement efforts based upon the patient-centered medical home (PCMH) model [7,8,9]; (b) associations between those adaptation characteristics and outcomes on the Practice Monitor, an assessment tool measuring level of implementation of key elements of PCMH transformation; and (c) the fit of a proposed expanded conceptual model of adaptations, the PCMH Adaptations Model, for this new context.

Adoption of the PCMH model [7] has been among the most influential recent developments in primary health care. PCMH is essentially a philosophy about care bound by a set of principles and achievements that, taken together, focus on the goal of improving the quality, efficiency, and patient- and system-specific outcomes of primary care. Features of the medical home include care that is patient-centered, comprehensive, coordinated, accessible, and committed to quality and safety [8]. Meeting, documenting, and submitting evidence of the achievement of these principles can result in recognition as a PCMH-designated practice, which can result in improved reimbursement or other positive outcomes [9]. Evaluations of PCMH efforts have found that generally, PCMH-designated practices produce improved outcomes compared to non-designated practices [10,11,12].

A regional quality improvement and PCMH transformation program in Colorado offered an opportunity to apply and expand concepts of adaptations from strictly evidence-based interventions to a new content area, practice transformation. Most studies of adaptation have focused on tightly defined, evidence-based, and typically “protocolized or manualized” interventions [13, 14]. This paper extends the study of adaptations to quality improvement efforts that are less tightly structured, but still designed to employ standard methods and processes, such as team-based care and the development of patient registries to guide efforts to improve patient health outcomes in primary care.

To study adaptations in this PCMH context, we began with the framework of Stirman et al. [13], which is based upon a review of the literature on adaptations. Stirman et al. developed a model to describe adaptations or modifications that organizations make to fit an evidence-based intervention to their setting. Their model was developed based upon a systematic review of adaptations to evidence-based interventions. This and other studies have illustrated that there are several challenges to diffusion and implementation of evidence-based interventions and that both planned and unplanned adaptations are frequent [15, 16]. Further, various factors may function as either facilitators or barriers to implementation of evidence-based interventions, depending on context. These data suggest that adaptations are widespread, underscoring the importance of understanding the effectiveness of various adaptations. Despite this, relatively little is known about adaptations, their context, positive and negative outcomes associated with various levels of fidelity or adaptation, and the robustness of adaptation concepts across different contexts and types of interventions [13]. To address these issues, the Robert Wood Johnson Foundation funded grants to study local adaptations under a program envisioned by Leviton and colleagues [17]. This report is the result of one of these projects.

Methods

This study was funded by the Robert Wood Johnson Foundation (Grant #71732) and approved by the Colorado Multiple Institutional Review Board at the University of Colorado Denver. Data were collected in 2015 and early 2016, and analyses occurred in 2015 and 2016.

Recruitment and setting

Participants were recruited from Colorado primary care practices that participated in a year-long practice transformation initiative consisting of trained practice facilitators providing regular technical assistance on quality improvement (QI) and PCMH concepts. Prominent goals and frequent activities in the initiative focused on the development of team-based care and patient registries to guide chronic disease management. Patient registries are an important component of the PCMH model because they allow the care team to identify, maintain, and track patients with a particular risk factor or condition, such as uncontrolled hypertension or diabetes, and perform specific activities to improve patient health outcomes. Most participating practices specialized in family medicine, and just over half were part of a group (e.g., network, hospital system, independent practice association). Key informants who took part in interviews represented both clinical and non-clinical roles and held positions such as practice administrator, office manager, coordinator, director, or physician.

For this study, a research assistant emailed and phone called practices after an initial introduction by the initiative’s practice facilitators, offering interviews to the QI team leader or coordinator at the 56 practices that completed the initiative.

PCMH adaptations model

Initial pilot work and discussions with PCMH colleagues revealed that although useful, the Stirman et al. model might not include all the factors involved in understanding adaptations made during quality improvement efforts for PCMH transformation and would need to be modified to enhance applicability to this topic and context. To address this, we added components from the RE-AIM planning and evaluation framework [18, 19] and others based upon pragmatic issues in primary care settings to help us better understand the characteristics and purpose of adaptations, creating the “PCMH Adaptations Model.” Questions were framed in a “who, what, why, when, how” format intended to be user-friendly and understandable by practice personnel. Interview questions from the Stirman et al. model included the domains of Who: the individual(s) making the decision to modify, and What, consisting of both the type of content modification and context of the adaptation. To this model, we added domains based on the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [18,19,20] and our study team’s experience related to PCMH improvement efforts. This included the domains of Why: the reasoning behind the adaptation (e.g., to increase reach, effectiveness, adoption, implementation and/or maintenance; adding response options to allow for the possibility that external factors influenced the adaptation’s development), and Impact: the perceived positive and/or negative outcomes of the adaptation. Interviewees were instructed to provide their own assessment of the impact of the adaptation on their original program goals, acknowledging that this was their personal perception of impact. To complete our PCMH Adaptations Model, we added questions to represent the How domain: the adaptation’s basis, and When: the point during the program at which the adaptation was made. Table 1 presents all questions and response options by domain from our structured interview for adaptations based on the model.

Table 1 PCMH adaptations model, corresponding interview questions, and response options
Full size table

We tested this model and structured interview method in a pilot study of eight primary care organizations that participated in the Robert Wood Johnson Prescription for Health program [21]. The interview questions and procedures worked well for these respondents and confirmed that the additional items helped to address the PCMH topic and context. Minor wording modifications to clarify questions and response options were made based on these interviews.

Data collection

We used a structured interview guide based on our PCMH Adaptations Model domains (Who, What, When, How, Why, Impact) designed to document how practices in the PCMH initiative adapted their original plans. A study investigator (JH) and research assistant (TH) trained in qualitative data collection and analysis conducted telephone interviews with practice key informants. The first several study interviews were conducted including the study PI (RG) to ensure fidelity to the interview protocol and provide any necessary adjustments to interview methods. The remaining interviews were conducted individually by the investigator or research assistant. Interviewers first discussed the concept of adaptation with participants, defined in the interview as “a change to your original plans or goals of the initiative,” and asked participants to identify up to two separate adaptations. This definition of adaptation in the context of PCMH transformation was added based on pilot interviews and has implications for participants’ characterization of the perceived positive or negative impacts of the adaptation. Interviewers then walked through each domain of the PCMH Adaptations Model and asked questions about each of these domains for up to two adaptations that the interviewee identified. Each interview question had set response options, and participants were asked to identify a primary and secondary selection from these options (Table 1), along with providing a narrative explanation for their selections.

Interviews took place over a ten-week period from January to April 2015 and lasted between 25 and 60 min. Interviewers took extensive notes supplemented by audio recordings. Participants received a $50 gift card for their participation.

Measures

There are two types of data included in this paper: (1) descriptive information on the characteristics of adaptations provided in interviews and (2) exploratory analysis of associations between adaptation characteristics and outcomes data for the practice changes made through the PCMH transformation. These data on perceived outcomes were collected via a separate assessment tool, the PCMH Practice Monitor described below, as part of an independent evaluation of the PCMH initiative conducted by different research staff and separately from the PCMH adaptations interview.

The study investigator and research assistant conducted basic descriptive analysis to determine the types of adaptations identified by interviewees and the frequency of corresponding response options, by domain of the PCMH Adaptations Model. Responses were organized into a Microsoft Excel spreadsheet according to categories based on the following domains of the PCMH Adaptations Model: Who, What, Why, When, How, and Impact.

Improvement outcomes

Assessing outcomes in practice improvement in the PCMH initiative was complex and challenging because practices focused on different disease conditions and employed different change strategies. We developed a composite outcome measure that was independent of adaptations and related to individual practice goals and improvement progress based on previously collected assessment data. This measure was computed using relevant scores from the separately administered Practice Monitor [22], a practice-based assessment for participating primary care clinicians and staff at baseline and 12-month follow-up during the initiative (and conducted by different staff independent of the structured adaptation interviews). The Practice Monitor measures progress on 11 domains—Staff Engagement, QI Processes, Data Capacity, Population Management, Patient & Family Engagement, Team-Based Care, Coordination of Care, Cost Containment, Access & Continuity, Behavioral Health Integration, and Leadership—that align with key elements of PCMH transformation [23]. Each domain consists of between five and nine items on which practices are scored on a five-point scale ranging from 0 (not at all) to 4 (completely). The study team carefully reviewed and discussed a priori the items in each domain and determined that the team-based care and data capacity domain items best aligned with practice goals and activities in this study and therefore were the most relevant domains to analyze for association with adaptations. The team-based care domain includes five items measuring, for example, the maintenance of care teams and daily team huddles. The seven items constituting the data capacity domain include the frequency of reporting and reviewing quality measures, the ability to extract data from the medical record system for patient registries, and data accuracy. Team-based care and data capacity domain summary scores were computed by transforming responses to a scale of one to five, calculating the sum of responses to each item in the domain, and scaling to a summary score out of 100 possible points to facilitate comparison across scales. Practice Monitor summary scores reflect pre–post assessments in key criteria for PCMH recognition and are to be distinguished from the ongoing adaptations involving the process of implementation assessed in this study.

“Success” in this initiative was challenging to define. Practices addressed different chronic illnesses and types of improvement, so it was not possible to select a single common outcome as in most randomized controlled trials. Some practices were new to the facilitated practice improvement initiatives, while others had been previously involved in PCMH initiatives and were focusing on new areas of change after previously making substantial gains. Thus, a simple change score on the Practice Monitor subscales was not a good indicator of “success” that could be applied to all practices, since practices’ baseline levels of experience, skills, PCMH status, and activities varied. Operationalization of the outcome measures from the Practice Monitor subscales is described below.

Analyses

The study team conducted basic descriptive analysis (frequencies of response options selected by adaptation topic) to identify the most common characteristics of adaptations in the areas of team-based care and chronic disease registries. These descriptive analyses were used to inform selection of a subset of adaptation characteristics for analyses of relationship to outcomes. Topical categories of adaptations that emerged from initial descriptive analysis indicated the most relevant domains of Practice Monitor data to include in this analysis. To maintain a reasonable number of potential predictor variables, it was necessary to include in quantitative analyses all adaptations described rather than considering QI team and registry adaptations separately, and to reduce the adaptations dataset by collapsing similar response options. The entire study team utilized a group discussion process to decide which interview response options to combine, based on existing research frameworks, interview respondents’ conceptualization of response options, and the frequency and relationship of selected response options (e.g., deleting those options almost never selected or always selected). These decisions were made independently of and before considering outcomes. For instance, in response to the Why question of “Which of the following was the primary reason behind this change?” response options related to elements of the RE-AIM framework (reach, adoption, implementation, maintenance) were combined because they clustered together, while the “effectiveness” response option was left independent due to its high selection frequency. Each adaptation characteristic selected for exploratory analysis was coded yes/no (endorsed versus not endorsed) for each practice (N = 27) for either adaptation described in the structured interviews—that is, a practice would be endorsed as “yes”’ for the What—Format response option if this was their primary or secondary response option selected for either of the two adaptations they described.

Outcome variables were created based on Practice Monitor scores for the data capacity and team-based care subscales as described above at baseline and 12-month follow-up. Exploratory analyses on the larger sample of practices (N = 58; this number is inconsistent with the interview sample due to two interviews taking place with representatives of multi-specialty clinics which completed two separate Practice Monitors) in this initiative using hierarchical agglomerative cluster analytic approaches suggested three general patterns of change: (1) lower baseline scores with less improvement (n = 3) or decline (n = 1); (2) lower baseline scores with substantial improvement (n = 23); and (3) high baseline scores that were maintained over time (n = 31). However, except for the single practice that actually declined, these distinctions are somewhat arbitrary, so a continuous outcome variable was operationalized by calculating total area under the curve (AUC) for baseline to follow-up scores for each subscale and across subscales, thus capturing overall capacity plus improvement spanning the 12-month timespan of the PCMH initiative. AUC has been commonly used in other scenarios to combine multiple measures over time to reflect some total quantity over a designated time span, and the combined measure can be an outcome or predictor [24,25,26]. Kendall’s tau was used to examine the bivariate associations between the dichotomized adaptation variables (present/absent) and the continuous AUC outcome variables. Due to the small sample size and limited number of variables that could be included in a multivariable model, multivariable linear regression using a forward stepwise approach with 0.1 to enter the model (in order of lowest p value) was used to explore the relationship between adaptations and the AUC outcomes.

Results

Twenty-seven practices participated out of 56 total possible initiative-participating practices. Most participating practices specialized in family medicine, with small representation from internal medicine. About half were identified as part of a group such as a network or hospital system. Practices which declined to respond or participate were relatively similar to those that took part in interviews: most specialized in family medicine (p = 0.19), followed by internal medicine; about two-thirds were part of a system or group (p = 0.60), compared to just over half of participating practices. Measured by the number of clinicians at the practice, comparison indicates that non-participating practices may have been slightly smaller, with a mean of just under three providers compared to just under four among participating practices (p = 0.09). See Table 2 for comparison of select characteristics of participating and non-participating practices.

Table 2 Characteristics of participating and non-participating practices
Full size table

Adaptation topics

A total of 49 adaptations were described by the 27 practices. Responses to probes at the beginning of the interview to elicit the topical focus of the adaptation indicated that changes to registry and changes to the QI team were by far the most common categories of adaptations, though these were also the focus of much of the program guidance received, so this may be a result of the initiative in which the practices participated. The most common adaptation topic involved the registry (37%, n = 18 adaptations), characterized by a change in the process, goals, or outcomes of developing a registry. For example, multiple practices experienced a change in the vendor or software utilized for the electronic health record or for patient registries during the initiative, as a result of factors outside of the QI team’s control. Others experienced challenges obtaining access to accurate data and registry reports to guide improvement efforts. These issues led to delays and forced practices to adapt and utilize whatever information they could access to guide PCMH transformation efforts.

Modifications to the QI team and movement toward team-based care aspects of the PCMH, described as a change in team structure, membership, or meeting times, made up one-third of adaptations (33%, n = 16). Examples of these adaptations include adding meetings aligning with organizational structure at practice and leadership levels to coordinate QI efforts, or reducing the number of QI team members involved in all meetings due to levels of staffing, resources, or physician engagement.

Characteristics of adaptations

Review of all 49 adaptations, across topic categories, indicated that within the Stirman et al. framework domain of What, personnel changes (e.g., adding or removing roles from meeting attendance, revising the personnel responsible for tasks) occurred most often (n = 27, 55% of adaptations), followed by changes in format (e.g., revising the frequency and schedule of meetings, utilizing Microsoft Excel for patient registries instead of the EHR system) (n = 23, 47% of adaptations). Within the Who domain, the entire team was most often responsible for initiating or carrying out the adaptation (n = 25, 51%), and adding a component (What) was the primary type of adaptation involved (n = 18, 37% of adaptations). Coding on contextual and RE-AIM factors of How and Why showed that most adaptations were based primarily on pragmatic or practical issues (n = 24, 49% of adaptations) and performed with the intention or hope of increasing either or both program effectiveness (n = 22, 45%) or implementation (n = 21, 43%). The vast majority took place (When) during the early (21, n = 43%) or middle (21, n = 43%) stages of the program. See Table 3 for frequencies of the most commonly selected response options in each domain across all adaptations.

Table 3 Counts of most frequent response selections for registry and QI team adaptations
Full size table

Characteristics of adaptations reported for modifications to the registry in comparison to QI team were similar for many domains including Who (most often made by the team), When (most often in the early or middle stages of improvement activities), and Impact (the vast majority of adaptations being perceived as having positive impact in comparison to having not made the adaptation at all). There were some differences in response patterns between registry and QI team adaptations on the How domain, with pragmatic and data-based responses chosen more often for registry adaptations, and feedback, practical, and vision reasons being similarly common with one another for QI team adaptations. Format changes were more frequent for registry than QI team adaptations, though as mentioned above, this was still a fairly common characteristic described in both categories of adaptations. In terms of specific type of adaptation, tailoring was much more common for QI team modifications than registry adaptations. Finally, for the Why question, in order to increase effectiveness was selected much more often for QI team adaptations, while implementation was a more common reason within the category of adaptations to the registry. Frequency counts of the most common response options in each domain of the PCMH Adaptations Model are presented for all adaptations, and separately for categories of registry and QI team adaptations in Table 3.

Relationship of adaptation characteristics to outcomes

In order to maintain sufficient ratio of subjects to variables for analyses of association, characteristics of adaptations were considered without regard for the type of adaptation (e.g., QI team or registry); that is, analyses included all 49 adaptations described. Further, these analyses were performed at the practice level (N = 27), consistent with practice-level outcomes on the identified Practice Monitor summary outcome measures of data capacity and team-based care. For analyses, adaptations characteristics were coded by whether each practice selected a given response option for either of the adaptations they described (i.e., endorsed vs. not endorsed). Table 4 summarizes results of analyses relating adaptation characteristics to Practice Monitor outcomes of (a) data capacity, (b) team-based care, and (c) both of these outcome categories combined.

Table 4 Associations between practice-level adaptations and practice monitor outcomes
Full size table

There were no significant negative associations between identified adaptations and outcomes. Because of limited power, bivariate associations are noted for all p values < 0.15 (instead of the customary 0.05). The strongest bivariate associations were between adaptations to promote the impact or success (Why) of the intervention and better team-based care (p = 0.0018). Adaptations initiated and/or carried out by the team (Who) were associated with both better team-based care (p = 0.0136) and data capacity (p = 0.1022). Adaptations that occurred in the early or middle stages of the intervention (When) were associated with better team-based care (p = 0.0586) and data capacity (p = 0.0370). Adaptations to modify content of the intervention to fit the target population (What) and adaptations of tailoring to make the intervention better fit the practice and its resources (Why) were associated with better team-based care (p = 0.1176; p = 0.1437).

Responses from the category of Impact were not included in analyses relating adaptations to outcomes because of potential confounding of characteristics of adaptations (perceived Impact) with their (separately measured) outcomes. Multivariate analyses, although considered preliminary due to the small number of practices, revealed that variables from the categories of When and Why contributed the most to the summary outcome measures (see Table 4).

Discussion

This paper extends application of adaptation models from specific interventions to general program plans, quality improvement, and the PCMH movement. We found it helpful to add components to the original Stirman et al. framework to fit the PCMH, and possibly other primary care contexts. Consistent with literature in other content areas, adaptations to PCMH transformation were common [13, 14, 27] and may have positive results.

Across all 27 practices’ adaptation activities, regardless of topic, practices most commonly made adaptations derived from the Stirman et al. framework: modifications to personnel (Who) from those originally responsible (endorsed in at least one adaptation by 21 practices) as well as change in program format (What) (endorsed in at least one adaptation by 20 practices). Results from the RE-AIM elements indicated that adaptations were most often made (Why) for pragmatic reasons (endorsed in at least one adaptation by 22 practices) in an effort to increase effectiveness (Why-endorsed in at least one adaptation by 19 practices). Other interview items relevant to PCMH indicated that the majority of adaptations occurred around early to middle stages (both endorsed in at least one adaptation by 21 practices) of PCMH enhancement efforts. Despite the overall similarities, there were some differences between adaptations to the registry and quality improvement team in the domains of How, What, and Why; contributing to our understanding of the specific characteristics of adaptations of various topics.

Preliminary analysis of the association between adaptation characteristics and Practice Monitor outcomes indicated that the adaptations most strongly related to relevant outcomes on Practice Monitor domains were the following:

  • in response to the Who question, based on the original Stirman model, adaptations initiated or carried out by some or most of the team;

  • in response to Why questions, based on the RE-AIM framework, adaptations performed to enhance effectiveness or implementation;

  • in response to When questions, added to the original Stirman model to fit primary care interventions, adaptations performed in early or middle stages of the program.

These significant associations share similarities with interviewees’ most frequently selected response options in these domains:

  • most or all of the Team was the most common party responsible for initiating or carrying out the intervention (Who);

  • improving effectiveness and implementation were by far the most common reasons to perform adaptations (Why);

  • adaptations taking place in the early or middle stages of the program made up the vast majority of adaptations (When).

These commonalities between descriptive analyses and analysis of association with improvement outcomes as measured by a separate standardized tool reinforce the relevance of these domains of the PCMH Adaptations Model. The association of relevant improvement outcomes with response options originating from domains of all sources (the original Stirman model, the RE-AIM framework, and the study team’s additions for the PCMH context) suggests that maintaining as well as extending the original Stirman et al. model categories to include these additional factors was warranted. The PCMH Adaptations Model can help practices to better anticipate, categorize, and understand the types of adaptations commonly made in PCMH transformation efforts and aid decision-making by providing insight as to how these adaptations might relate to potential outcomes.

In the context of practice improvement and PCMH transformation, adaptations identified by practices were defined as changes in their original plans to enact improvement activities, often mid-course corrections to uncontrollable outside factors rather than changes to specific QI tools or elements of the PCMH model. It may be beneficial for future research to explore adaptations to more specific and strictly defined elements of PCMH transformation, such as enhanced patient access or coordinated care, or other primary care or organizational change efforts, to enhance knowledge of the outcomes associated with these types of adaptations or facilitate use of the PCMH Adaptations Model by focusing the discussion of adaptations to a more specific topic area.

Our PCMH Adaptations Model was able to characterize important adaptations, but challenges and limitations were also noted. Providing an example of what we meant by an adaptation at the start of the interview helped interviewees understand the concept, but carries the possibility of biasing respondents’ recollection of their practices’ adaptations. Still, some respondents had difficulty differentiating adaptations to planned goals of the change program from more general changes in their clinic. There were also relatively high numbers of response options to choose from in many domains, ranging from 5 (When) to 11 (What—type), making it difficult for some respondents to differentiate or choose among responses. The terminology of some questions and response options presented challenges in interpretation. For example, what is a “component” when the term is applied to a quality improvement effort (vs. a clearly defined research protocol)? Based on these findings and lessons learned from these interviews, we have recently modified the interview framework to include more open-ended responses, which are then coded into the various response categories. This revised interview is being used in recently initiated investigations and is publically available from the senior author.

One limitation is the relatively small sample that involved only a limited, although diverse, selection of practices in Colorado that participated in one PCMH transformation initiative. Participants self-selected, and while participating and non-participating practices had similar specialties and only minor differences in size, we cannot determine the extent to which the results are generalizable to other practices. Given the slightly greater mean number of providers in participating practices, it is possible that practice size may be a factor in practices’ level of implementation and types of adaptations made throughout PCMH transformation. The focus in the PCMH initiative on QI team activities and chronic disease registries, among other topics, likely influenced participants’ identification of adaptations. Interviews occurred in some cases one year or more after the conclusion of program participation, so participants’ recollection of specific details and information may have been incomplete. As previously described, assessment of the impact of adaptations on program progress and goals was based on interviewees’ perceptions rather than objective outcomes data. Finally, in most cases we interviewed only one representative per practice, this being the leader or coordinator of practice QI activities, whose experiences and perspective of adaptations and perception of their impact may have been different from other team members.

Due to these limitations, the perceived impacts of adaptations as described using the structured PCMH Adaptations Model interview guide should be interpreted with caution. This interpretation should include the understanding that these are the subjective interpretation of interview participants. Associations between various adaptation characteristics and improvement outcomes as measured by the data capacity and team-based care domains of the Practice Monitor demonstrate more concrete assessment of progress toward program goals for improvement. Although admittedly exploratory, use of cluster analytic and regression approaches helped to understand the complex relationships among adaptations and outcomes. Future investigations should determine if these results can be replicated with larger samples and other primary care, quality improvement, and health care team interventions.

In summary, despite challenges associated with the nature of PCMH transformation as a topic and the user-friendliness of some interview questions, the PCMH Adaptations Model provided a useful framework with which to identify and characterize patterns of adaptations made in the course of PCMH transformation. We identified common types of adaptations and found differential relationships of different types of adaptations to improvement. Future research is needed to further investigate and improve the PCMH Adaptations Model, to replicate these results, and to test their generalizability.