In the United States, jails and prisons house individuals with multiple health problems, including drug use, mental health disorder, and HIV/AIDS.1 , 2 Relevant research indicates that 1.3% of the total prison population were reported to be living with HIV or AIDS in 2010; moreover, the rate of diagnosed HIV cases was disproportionally higher (more than five times) than in the general population for that same year.3 These statistics point to a critical public health issue because most incarcerated adults will return to the community at some point.4 – 7 Thus, there is a need to improve availability of and access to innovative and effective HIV services in the areas of prevention education, testing, and linkages to care for criminal justice (CJ) populations.

In phase 2 of the Criminal Justice Drug Abuse Treatment Studies (CJ-DATS), the National Institute on Drug Abuse (NIDA) initiated a 5-year implementation science cooperative agreement that brought together nine research centers (RCs) to investigate strategies for implementing improvements to HIV services delivery in CJ settings.8 Numerous strategies exist for implementing evidence-based practices (EBPs) in various settings (e.g., learning collaboratives, audit and feedback, rapid cycle testing),9 , 10 and strategies for implementing evidence-based HIV practices in particular have been tested (e.g., the Replicating Effective Programs)11; however, few of these strategies have been tested in CJ contexts using specific tools for improving services in the field.12 Furthermore, implementation outcomes are scarce for large-scale multi-site studies. Efforts by CJ-DATS researchers to address this gap culminated in the development of the HIV Services and Treatment Implementation in Corrections (HIV-STIC) protocol.13

The primary goal of HIV-STIC was to experimentally test an organizational process improvement strategy for enhancing services delivery in the areas of HIV prevention, detection, and treatment, to adults under correctional supervision. In the current study, we examine fidelity or adherence to the process improvement strategy outlined in the HIV-STIC protocol, aimed at integrating new or improved evidence-based services into routine healthcare in CJ settings.14 In other words, we are not evaluating fidelity as an implementation outcome (i.e., fidelity to the implemented improvements)15; rather, we are examining fidelity to the HIV-STIC protocol by participating sites. Evaluating fidelity to the protocol is essential, so that process improvements can be reliably linked to the mechanisms and procedures being tested,14 , 16 thus paving the way for researchers and practitioners to use these tools.

Developing the HIV-STIC Process Improvement Strategy

HIV-STIC researchers selected the NIATx (formerly the acronym for Network for the Improvement of Addiction Treatment)17 process improvement approach as the template for developing the HIV-STIC process improvement strategy. NIATx was founded in 2003 as a partnership between the Robert Wood Johnson Foundation, the Center for Substance Abuse Treatment, and addiction treatment agencies across the USA (see Evans et al. for a detailed description of NIATx).18 The NIATx strategy represents “the first widespread application of process improvement techniques to the organization and delivery of treatment services for alcohol and drug dependence.”19 (p.2) The NIATx model is based on five core principles: (1) understand and involve the customer, (2) fix key problems, (3) pick a powerful change leader, (4) get ideas from outside the organization, and (5) use of rapid cycle testing to test and refine ideas and implement effective changes.18 – 20

The NIATx approach uses several key components—most of which were incorporated in the HIV-STIC protocol. These components include establishing a local change team of agency stakeholders, representing various levels of expertise and authority. Consistent with NIATx standards, each HIV-STIC change team received direction from an external NIATx-trained coach to support the team and its leaders in facilitating implementation of specific process improvements in service delivery. Early in the facilitation effort, the external coaches (experts in NIATx) provided teams with background on the NIATx model and the five NIATx action phases of change: (1) identify needs, (2) decide on desired change goal, (3) decide how desired change will be measured, (4) test changes, and (5) sustain desired change. These phases are foundational to the activities undertaken by change teams during the HIV-STIC study and central to the fidelity being examined here.

HIV-STIC researchers selected NIATx as the process improvement model for the protocol, concluding that with modification, NIATx could be a good fit for correctional settings. This assumption was based in part on the involvement of change teams, with staff representing various levels of responsibility within the agency, and because the research to date has demonstrated the effectiveness of the NIATx approach at initiating and sustaining process improvements in substance abuse treatment settings.21 , 22 Furthermore, in one of the first applications of NIATx to a CJ context,23 outcomes demonstrated the efficacy of the NIATx model for improving organizational and administrative processes in drug courts. Wexler and colleagues23 reported reduced wait times and no-shows, as well as increased admissions and participant engagement with treatment.

Modifications to NIATx for HIV-STIC

A description of the HIV-STIC design is not complete without a brief recounting of the modifications made to adapt NIATx for a correctional environment (see Belenko et al. for a comprehensive review of the CJ-DATS HIV-STIC protocol).13 HIV-STIC retained key components of the NIATx model: change teams, Executive Sponsors, Change Team Leaders (CTLs), baseline data collection, rapid cycle testing, NIATx-trained coaches to guide and monitor team activities, and the development of a sustainability plan. However, the research environment of HIV-STIC differed in three distinct ways from those in which NIATx has historically been applied. First, HIV-STIC extended the NIATx mandate to improve processes by including the option for HIV-STIC sites to implement new practices. Next, HIV-STIC expanded the “single” organization context for improvements, to focus on improvements across the care continuum. This modification importantly called for coordination of changes across different organizations. The third major distinction between NIATx and HIV-STIC shifted from working on only one improvement aim and one goal at a time (NIATx) to providing HIV-STIC sites the opportunity to work on one or more aims or new services for a specified goal. Additional changes to the NIATx model were incorporated to make it more applicable to the CJ setting:

  1. 1.

    Primary leadership roles defined by NIATx (i.e., Executive Sponsor and the CTL) were expanded by adding a third key leadership role—the Facility Sponsor—to help ensure buy-in from local facility leaders or management (e.g., medical directors, prison wardens). The Facility Sponsor, specified as someone who possessed sufficient authority within the organization, was directly responsible for overseeing the change process.

  2. 2.

    The role of Executive Sponsor was limited to passively monitoring local progress without becoming involved in day-to-day management of the change process, to avoid possible cross-contamination of experimental and control sites (both of which were under the authority of the Executive Sponsor). The Facility Sponsor more closely reflected the classic role of the Executive Sponsor in NIATx and was responsible for overseeing the specific problem to be addressed within the area of testing, prevention, or linkages (chosen by the Executive Sponsor).

  3. 3.

    NIATx tools (PDSA and rapid cycle testing) typically are framed as a “rapid” turnaround; however, in HIV-STIC, cycle testing more often occurred over longer periods, sometimes measured in months. In fact, low numbers of HIV seropositive inmates in some sites during the course of the study required months of data collection for testing strategies, to link individuals returning to the community with HIV service providers.

  4. 4.

    A key principle of NIATx is to “understand and involve the customer.” In the case of HIV-STIC, the customers (those expected to ultimately benefit the most from the changes) were the inmates living with or at risk for HIV. However, due to confidentiality issues and other logistical restrictions for involving inmates in the study, the “customer” was not involved in change team activities. [Inmates, regardless of HIV serostatus were anonymously surveyed as part of HIV-STIC at baseline and follow-up in order to illicit feedback on their perceptions and experiences with HIV services in the correctional facilities where they were incarcerated24 for results of the anonymous survey analysis.]

The variations described above point to the importance of measuring fidelity in research; perhaps especially relevant when testing a model that has undergone modification, as is the case in the HIV-STIC protocol. The significance of this thinking is further amplified by the complex nature of implementation research itself.25

HIV-STIC

Under HIV-STIC, change teams were charged with implementing improvements to HIV service delivery in one or more of the three areas of the CJ HIV care continuum: prevention education, testing, or linkages to community-based care following incarceration.26 Change teams selected specific outcomes to focus on, guided by clinical evidence and recommendations from the Centers for Disease Control and Prevention (CDC).3 For example, a change team working in the area of linkages might decide to reduce no-shows for aftercare treatment (the specific outcome), by formalizing communications between CJ staff and HIV treatment providers. In this example, the efforts to achieve more consist and documented communication reflect the improvement to service delivery.

HIV-STIC was implemented in prison and jail facilities, using a cluster randomized trial to determine experimental and control sites to test whether a modified NIATx strategy would improve organizational climate and systems linkages for HIV services, change staff and inmate attitudes and perceptions about HIV risk and HIV services delivery, and move HIV-related interventions and services toward full implementation and sustainability. Results of these outcomes of interest for HIV-STIC are published elsewhere.24 , 27 – 30

The goal of the current study was to assess the fidelity of change teams to the modified NIATx process improvement strategy in their efforts to improve HIV service delivery. Here, fidelity refers to the extent to which utilization of the process improvement strategy met structural (e.g., leadership) and process (e.g., PDSA and rapid cycle testing) components, determined to be measurable indicators of the strategy. The results provide important background knowledge about the appropriateness of the modified NIATx strategy for implementing improvements to service delivery in CJ settings.

Methods

Each of the nine RCs recruited a minimum of two matched prison or jail facilities (n = 28) to participate in the HIV-STIC cluster-randomized trial. Executive Sponsors determined the area for HIV service improvements (i.e., prevention, testing, or linkage to care) for each pair of sites. Following the informed consent process, staff at all sites received baseline training on best practices in HIV service delivery in correctional settings (prior to randomization). Sites in the control condition (n = 14) were instructed by the Executive Sponsor to use usual agency practices to work on HIV service improvements; in the experimental condition, sites (n = 14) formed change teams and received the modified NIATx strategy to work on the same HIV service improvements as the control sites. Change team members, identified by the Facility Sponsor and CTL, represented a range of positions (administrative, supervisory, and line positions from medical and correctional personnel). In cases where the focus for HIV service improvements was on linkage to care, community HIV service providers also were recruited as change team members.

Prior to the first change team meeting, each team participated in a 1-day training on NIATx and the HIV-STIC modifications from their external NIATx-trained coach and research representative from the RC. Additionally, change team members participated in site walk-through events to learn first-hand about the experience of the consumer (i.e., inmate) in receiving HIV services in a correctional setting. Throughout the remainder of the implementation period, the NIATx coach supported the team (members and team leaders) by attending monthly team meetings (in-person or by phone), guiding and advising the team on procedural areas including goal selection, PDSA techniques, and team facilitation.

The research representative from each of the nine RCs also supported the change team, working predominately through the team leaders (CTL and Facility Sponsor) and the coach in all aspects of the study (as a resource about the study, scheduling meetings, data collection, etc.). This behind-the-scenes perspective provided the researcher with a more in-depth understanding of the change team activities; thus, the research representative was determined to be the prime candidate to evaluate the team’s fidelity to the protocol strategy. Fidelity ratings by nine research representatives were completed on a quarterly basis during the 12-month research project. These ratings were submitted quarterly for review by participating RCs, under the direction of the study leads for the HIV-STIC protocol.

Measures

The HIV-STIC fidelity form consists of a scale tailored to evaluate the modified NIATx strategy of the protocol. Guidance from NIATx resource materials and consultation with a senior expert from NIATx informed the contents of the fidelity measure. The form incorporates critical components related to structure and process 31—indicators of the HIV-STIC process improvement strategy. Structural components reflect the emphasis on leadership in the NIATx core principles; process components include PDSA phases and rapid cycle testing activities—the former map to the NIATx action phases (previously described) and the latter, to the core principles. Thus, the fidelity scale for HIV-STIC includes two subscales: (1) structural components for rating change team roles and (2) process components for rating the ways in which change teams utilized PDSA and rapid cycle testing in making changes to service delivery. Items and rating scales for these two areas are shown in Table 1. Ratings apply to the timeframe within a specific study quarter representing a 3-month period. Additionally, a problem indicator subscale (see Fig. 3) with eight items was included on the form, to gauge the magnitude of the problems indicated by researchers in the structural and process subscales. Problem indicators were rated on a different scale from structural and process items. RCs were asked to expand on obstacles and solutions to reaching fidelity for structural and process components (where applicable) in an open-ended text space on a separate page of the rating form.

Table 1 HIV-STIC fidelity measure: structural and process components
Full size table

Analytic plan

Fidelity to the HIV-STIC process improvement strategy at 12 experimental sitesFootnote 1 was evaluated based on (1) to what degree experimental sites followed the HIV-STIC protocol and (2) what factors influenced the success or failure of implementing the protocol as designed. Because the quantitative ratings of structure, process, and problem indicators prompted the open-ended qualitative text, our mixed methods approach to analysis was expansion: the open-ended qualitative text was analyzed to explain the findings from the analysis of the quantitative ratings.32

Quantitative analysis primarily focused on ratings of the structural and process component items (Table 1 sections I and II). For each quarter of the protocol timeline, an average score of items in these two components was computed for each site in order to examine the general trend of fidelity ratings over time, as well as the inter-site variations of each component.

For the qualitative analysis, two researchers—trained in qualitative coding for the research cooperative—independently coded the researchers’ notes from the open-ended sections of the fidelity forms using thematic analysis.33 A team of researchers, including the two coders, then worked toward reaching consensus on that coding,34 resulting in the final codebook. The codebook contained main codes (e.g., communication, change team leadership, change team process) and secondary codes (e.g., change team process, shifting focus and change team leadership, competing job priorities). After reviewing the coded data, researchers identified four prominent themes in the open-ended text: leadership, communication, participation, and data utilization. Examination of the themes focused on looking for connections among the four themes, in order to increase our understanding of the study results.35 At each stage of this iterative process, team debriefings were held to promote reliability and validity of the findings and to control individual researcher bias.31 , 36 Consistent with the expansion approach to mixed methods, the research team evaluated whether the qualitative results were consistent with the degree of problem severity indicated on the rating form. This process was aimed at enhancing the convergent validity of the data.37

Results

Quantitative findings

Overall, fidelity to the structural components was substantial (M = 2.54; scale: 0 = never, 1 = sometimes, 2 = usually, 3 = always) across the 12 sites. In fact, across the entire study timeline, 87% of the mean fidelity scores for structural components were 2 (i.e., usually) or above (see Fig. 1). Fluctuation in fidelity (increase or decrease) was common early in the implementation phase of the study, demonstrated by changes in ratings from quarter 1 to quarter 2; however, 50% of the sites completed the implementation phase with quarter 4 ratings of 3, indicating high fidelity to the structural components. Additionally, another five sites (42%) finished the final quarter with averages of 2 or above. Still, even at quarter 4, there were indications that fidelity to the structural components was a struggle to maintain for a few sites: one site scored lower than a 2 suggesting that the structural components were only sometimes in place or appropriately functioning by the close of the implementation phase. The variation in fidelity to the structural components of HIV-STIC is illustrated in Fig. 1.

Figure 1
figure 1

Site variation in fidelity to the structural components, by quarter

Full size image

Overall, fidelity to HIV-STIC process components was also high (M = 2.29). Similar to the ratings for structural components, ratings of fidelity to process components increased over the course of the implementation phase, but again, there were variations across sites regarding how fidelity changed over time (see Fig. 2). Seven of 12 sites reported a decline in fidelity at quarter 2. Three of those sites continued to decline in their fidelity scores at quarter 3; however, the other sites increased or remained unchanged from quarter 2 to quarter 3—a pattern that also appears from quarter 3 to quarter 4 for the majority of sites. One site (site 9) reported substantial improvement in fidelity to the process components from quarter 3 to quarter 4.

Figure 2
figure 2

Site variation in fidelity to process components, by quarter

Full size image

On average, the ratings for problem indicators (Fig. 3) related to structure and process components, indicate a decrease in problem severity over time for approximately half of the sites. For most of the remaining sites, problem ratings showed relatively minor declines (indicating increases in problem severity) from the first to the fourth quarters. The degree of problem severity varied across sites, but leadership (questions 3 and 5) and data collection responsibilities (question 6) emerged as prominent major concerns. Frequent ratings for minor concerns indicated challenges in the areas of conducting team meetings, interruptions to team efforts, and lack of effective coaching.

Figure 3
figure 3

Problem indicators classified by question type

Full size image

The general trends and the amount of variation in the quantitative fidelity ratings across sites guided the qualitative analysis discussed in the next section.

Qualitative findings

Qualitative data consisted of comments and explanations that researchers provided for their fidelity ratings of their respective teams (n = 12) on the dimensions of structural components, process components, and problem indicators. Qualitative findings are organized under four main themes: leadership, communication, participation, and data utilization—representing data provided from all 12 experimental sites included in our analysis.

Leadership

Two site leadership roles were prominent in the HIV-STIC process improvement strategy: the change team leader (CTL) and the Facility Sponsor. The CTL position was designed to help team members understand the study protocol and improvement strategy, keep team members motivated and on track, and help the team overcome barriers either through strategizing or direct assistance. In many cases, fidelity to the CTL role was high and facilitated process improvement at those sites. Some sites reported having a CTL who acted as a champion; specifically, one who generated support for a new approach and came up with solutions to problems in implementing the change.38 At other sites, exceptional team members emerged to compensate for inadequate leadership from CTLs:

Changing leadership roles seemed to contribute to a lack of focus in the areas of establishing communication and setting meeting schedules. These concerns have been ameliorated during the 2nd quarter, particularly due [to] two highly motivated change team members (one became the new CTL).

Still, others reported on the improvement of CTL leadership over time:

Some change team members have expressed frustration with the lack of information on what they should be doing. In response, the [Change Team] Leader has begun to ask the coach for clarification on the PDSA process, thus improving the exchange of information (between coach and [CTL]).

A second leadership role in HIV-STIC was the Facility Sponsor (one of the modifications to the NIATx approach). The Facility Sponsor was primarily charged with allocating resources, supporting quality improvement efforts, and monitoring progress toward implementing changes. Collaboration between the CTL and the Facility Sponsor was also important for addressing challenges that teams faced in implementing the strategy. However, some Facility Sponsors lacked the institutional influence to accomplish changes recommended by teams. For example:

The Facility Sponsor has limited influence regarding the viability of processes that require approval/resources outside her unit. Due to her workload, the Facility Sponsor has progressively withdrawn from the project.

The impact of the Facility Sponsor’s decision to leave the project left the team without a major structural component of the modified NIATx strategy. At other sites, the Facility Sponsor had the requisite authority but did not always “buy-in” to the project and remained uninterested. For example, one site reported that:

[The] Facility Sponsor did not show much enthusiasm for the change team process once it got started, and the [CTL] reported some serious push back from the Facility Sponsor when she tried to implement one of the changes at that facility.

While tensions and a lack of fidelity to roles sometimes created barriers to process improvement, some teams responded to the lack of engaged leadership by adapting change strategies:

The rating for [“The Facility Sponsor has enough organizational influence to bring resources to the change team”] shifted from a major problem to a minor problem [during this quarter] because although the Facility Sponsor is still highly resistant to the project, the team has figured out ways to work around/without [the Sponsor].

The qualitative evidence in this section on leadership illustrates that structural and process components of the HIV-STIC process improvement strategy often are intertwined. In particular, challenges in maintaining fidelity to the structural components sometimes led to struggles in maintaining fidelity to the process components. For example, leadership (a structural component) often overlapped with communication issues (a process component)—another prominent theme discussed next.

Communication

Communication between CTLs and members of the team was vital in order for the group to identify service delivery needs, keep all members informed of the PDSA process, and to facilitate communication of the team’s progress to employees that were not members of the team. Teams reported that lack of consistent and meaningful communication impaired scheduling team meetings which resulted in delayed or overly lengthy change cycles, failure to complete change cycles, and a lack of clarity about the protocol and process improvement strategy among the team members. Furthermore, some sites described how a lack of communication between the CTL and the external NIATx coach resulted in confusion among the team members regarding the appropriate process for moving forward:

The change team could benefit from an extra push from the coach to initiate change cycles. The coach was always difficult to reach both to the research staff and the change team. He was not as much of a presence at the meeting as would be expected or needed. Because of infrequent communication between the coach and the team leader, the team is sometimes confused about what to do next.

After identifying a communication issue between CTL and the coach during the first quarter, one site reported on how the team focused on addressing this issue to facilitate their adherence to the protocol over time:

The frequency of [change team] meetings has improved and more consistent communication has been established [between] the coach and CT Leader during the 2nd quarter. The increased communication has helped to provide the CT Leader with a greater understanding of the PDSA process (enhances planning for change team activities).

Participation

Researchers commonly cited conflict between members’ involvement with the team and other job responsibilities as a challenge to fidelity. The simultaneous demands of these responsibilities required sufficient leadership support as well as consideration of some practical factors, such as the allotment of adequate time for both activities through a reduction in regular work duties or overtime opportunities. In the context of HIV-STIC (i.e., CJ facilities), such practical solutions were not always feasible, which created obstacles to maintaining fidelity to the modified NIATx strategy for many sites.

Some sites reported that competing job priorities made attending team meetings difficult, reducing time that could be devoted to the study and fidelity to the process. One site reported that the team activities were delayed because members needed to prioritize their regular job responsibilities:

The change team did not attend one meeting because of a crisis. When the change team did attend meetings, they were always 15 minutes late. After the RC asked for agency level information again in November, a member of the change team apologized, reported that the jail had been dealing with many crises, and remarked that the appropriate information would be provided after the jail calmed down.

Attrition due to turnover and transfers among team members, including leaders, also negatively impacted fidelity:

Changes are moving slowly due to change team turn-over [and] infrequent meetings. Several scheduled meetings have been cancelled. There has not been a consistent change leader or change team membership; this appears to impede the momentum of the change team. The change team membership has been rotating due to many staff/change team member transfers in and out of the facility.

Data utilization

Another challenge to maintaining fidelity to the process components of HIV-STIC was a lack of understanding of the value of data utilization in making decisions. Data are a key component in evaluating the impact of changes—an important aspect of PDSA. Many teams struggled in collecting data, utilizing data, and understanding the need for data in decision-making; instead, they relied on anecdotal experiences and practice knowledge. Even CTLs struggled with the validity of and need for data collection, evidenced by reports of lack of data utilization by leadership in guiding the team:

By the middle of the third cycle, the change team changed their goal to make the HIV prevention/education courses about health, wellness, and knowing your body. Based on experiences with the team, it appears that the change team acts on ideas without consulting the NIATx coach. There appeared to be little or no structure in the decision making process, and it appeared to be driven by the change team leader’s enthusiasm for the project.

For some teams, support from an engaged coach positively affected fidelity to the modified NIATx strategy by providing members with timely guidance in dealing with unanticipated data issues:

Change team meeting minutes indicate that changes might be happening simultaneously, and the group appears to be somewhat hampered by confusion about how to document the impact of changes (data collection), particularly with only a few HIV+ offenders releasing to the community. To address this, the coach has established a call schedule with change team leadership prior to change team meetings to discuss measurement options.

Discussion

At all HIV-STIC study sites, the change teams were able to implement one or more specific process improvements that resulted in a significant increase in HIV service provision for inmates in the experimental sites.27 Overall, structural and process fidelity ratings for the HIV-STIC process improvement strategy demonstrate the success of change teams at implementing the modified NIATx strategy in CJ settings.

Qualitative analysis revealed four prominent areas where the variation occurred: leadership, communication, participation, and data utilization. These themes identified in our study converge with the findings of recent theoretical/conceptual and empirical work in the field of implementation science.

Leadership

We found that one of the most important elements of undertaking inter-organizational change with the modified NIATx approach was having supportive leadership with a vision for change team activities. Similar experiences have been uncovered in implementation science studies. For example, in his study of leadership factors that are associated with attitudes toward adoption of EBPs among 303 public sector mental-health clinicians and case managers in 49 programs providing services to children and adolescents in San Diego,39 Aarons assessed provider perceptions of their immediate supervisors’ transformational leadership (that inspires and motivates followers) and transactional leadership (based on reinforcement and exchange) behaviors. Correlational and regression analyses indicated that providers who reported that their immediate supervisors enacted more transactional leadership were more willing to adopt EBPs, and providers who worked with supervisors with higher ratings on transformational leadership were more willing to adopt EBPs, if required to do so.40 – 42 Leadership is also critical in sustaining effective innovations.43 , 44

Communication

Effective communication is a key aspect in the transfer and implementation of EBPs. Our findings showed that undertaking inter-organizational change with the modified NIATx approach required consistent, meaningful communication between leaders and team members. Change team leadership communication issues were linked to a host of problems, such as scheduling team meetings, delayed change cycles, and even a lack of understanding among team members about the HIV-STIC process improvement strategy itself. Similar experiences have been identified in the implementation science literature. Aarons and colleagues45 worked with San Diego County Children’s Mental Health officials, public agency directors, and program managers to identify 32 individuals representing a diversity of organizational levels and a range of mental health agencies and programs—including inpatient and outpatient treatment. The participants generated 230 statements relating to facilitators or barriers to EBP implementation, statements which were distilled to 105 distinct declarations, then sorted and data analyzed. Results identified 14 factors perceived as helping or hindering EBP implementation. A key finding of this research was the importance of communication among stakeholders in facilitating a more thorough understanding of what factors influence EBP implementation—leading to more effective adoption efforts. Similar experiences have been found in other settings.46 , 47

Participation

A key component of successful implementation is participation, supported by popular models such as the EPIS model (exploration, adoption decision/preparation, active implementation, and sustainment),40 which includes engagement—a factor critical to the success of the implementation phase of the model. In HIV-STIC, competing job priorities and turnover in change team leadership and membership (often occurring because of job reassignment or extended leave), negatively impacted participation. These events were not unusual on teams, despite the addition of the Facility Sponsor role, conceived as a “power broker” with more senior administrators, to help maintain team membership by reducing job transfers (when possible), as well as providing support and resources for the work by change teams.

These logistical and practical change team membership issues are common in CJ settings48 and often cannot be predicted when conducting research or real-world implementation efforts. Yet, it is important to establish, whenever possible, contingency plans prior to establishing change teams, so that the potential disruptive effects of these issues can be reduced, if not eliminated. For example, organized rotation of staff on teams reduces demand on staff time, and assigning co-chair roles for team leadership provides flexibility for busy supervisors and administrators.

Data utilization

Our finding regarding teams’ difficulty with utilizing data for decision-making underscores the need for training teams on how to collect and use data to evaluate their work before making decisions to adopt, adapt, or abandon changes—a concept based on having measurable and reliable data available during rapid cycle testing. Training on data utilization should be considered necessary to reduce reliance on anecdotal knowledge or practical experience for making improvements in services delivery.

The collection and use of quality data is essential in evaluating EBP adoption, implementation, sustainment, and in assessing the fidelity of the adoption process.40 Indeed, fidelity to the HIV-STIC protocol varied over time within sites, suggesting that process steps should be routinely monitored for compliance to the HIV-STIC protocol. Ongoing monitoring can assist in the identification of time points or phases where fidelity lapses occur and result in efforts to recalibrate team performance. In HIV-STIC, fidelity monitoring served to bring problems to the attention of researchers, coaches, and teams, enabling corrective action during the course of the study. This outcome was particularly evident in facilitating effective communication between different parties for addressing data collection issues. Our fidelity measures, containing items relating to the structural and process components of the modified NIATx strategy, were tailored for use in our specific study. Further psychometric work would be needed to assess its usefulness in other settings.

Data are also important in monitoring implementation drift or misapplication or mistaken application of the practice or model, especially in circumstances where a dynamic adoption process of implementation is being evaluated.49 This is a critical area of agency infrastructure development, as main line staff in service agencies are often not experienced in collecting and using data for decision-making purposes.

Linkage between quantitative and qualitative findings

The quantitative data were intended for use in providing a background or contextual picture of HIV-STIC implementation structure and process fidelity over the implementation period. As we discussed earlier, these data indicated that overall fidelity to the HIV-STIC structural and process components was high, although there were variations in the fidelity of both components across sites over the four quarter implementation period. What these data did not provide, however, was insight into the factors and their dynamics that facilitated or hindered the implementation process. These elements were examined with qualitative data, drawing on mix-method design—an approach that is increasingly appreciated in implementation science studies.49

Limitations

A key lesson learned from this study is the benefit of a pilot phase in which a rigorous, small-scale implementation and evaluation of the protocol is conducted. Had the researchers conducted such a pilot, it is probable that it would have led to the development of strategies to reduce some of the barriers to fidelity encountered by HIV-STIC participants. For example, piloting the protocol might have highlighted additional aspects of CJ infrastructures likely to influence an inter-organizational change team (e.g., contracting practices with service providers), thus better informing the modifications to the NIATx approach for the HIV-STIC strategy.

The HIV-STIC protocol was designed so that the research representative was not directly involved in hands-on work, carried out by change teams and their coaches. However, researchers did communicate with coaches and change team leadership, increasing the chances of influencing the processes being rated for fidelity. Furthermore, we are unable to state with certainty that the behind-the-scenes involvement of the research representative did not bias the ratings, despite efforts to remain impartial.

Another lesson learned is limiting the source of qualitative data, relying only on written comments to the open-ended questions by the research representatives. Perhaps following up with researchers to complete a brief interview would have provided more nuanced information to enhance the interpretation of structural and process ratings data.

Future Directions

The HIV-STIC study demonstrated that multi-agency change teams can be successfully implemented to work on improving HIV services for inmates in correctional facilities. Future research is needed to gain a better understanding of change strategies in CJ settings and how fidelity to those approaches influences sustainability of new or improved health-related practices for inmate and other CJ populations. For example, is a change team approach the best fit in a CJ setting where safety and security often pre-determine staff schedules? What aspects of the change strategy are most effective in supporting sustainable practices in “real-world” service delivery settings? One such research effort with HIV-STIC experimental sites offers new insights into the relationship between the HIV-STIC process improvement strategy and the sustainability of improvements to HIV services28, but additional studies are needed to explore these issues.

Implications for Behavioral Health

Researchers recognize the complex nature of conducting implementation research as do behavioral health service providers with firsthand experience in implementation practice. Models that are designed to guide implementation efforts are numerous and diverse—in part reflecting the challenges with one model fits all thinking. The current study contributes important information about change team methodology, particularly in the areas of leadership and communication (two of four key findings—all of which influenced change team efforts in making improvements to services delivery for offenders with HIV in HIV-STIC sites). The four factors identified as influencing fidelity in our study (leadership, communication, participation, and data utilization) are relevant to collaboration efforts in all types of behavioral health settings where the goal is to improve care. Furthermore, these findings, while useful to researchers for developing and refining implementation models and tools, have a practical real-world application for change teams and other types of group collaboration, independent of research initiatives, namely, to aid in facilitating the often complex, yet essential dialog between key stakeholders involved in service delivery for individuals with health issues.