The dearth of knowledge about what accounts for evidence-based and promising health and mental health innovations (EBHI) to endure in the organizations that adopt them has been described as one of the most significant impediments to closing the gap between intervention development and program use.1–11 It prevents the nation from fully benefitting from the billions of US tax dollars spent on developing EBHIs and, in mental health, it unnecessarily prolongs the suffering of millions of Americans who live with mental disorders.3 While many barriers and complex challenges exist, long-term, longitudinal investigations of public organizations’ efforts to implement EBHIs under naturalistic conditions are needed to shed light on this important topic. This type of research is essential to ensuring that all populations have the opportunity to benefit from the nation’s investments in developing scientifically based healthcare innovations.9

A substantial body of theory and research from healthcare and organizational behavior identifies factors and processes that account for the success of organizations’ efforts to implement innovations. Much of this knowledge comes from studies which examine innovation implementation for relatively short time periods (e.g., 2 or 3 years), and which define success in a myriad of ways.12–15 In contrast, studies of long-term innovation-use are rare. Most of what is known about implementation beyond initial use and early outcomes originates from in-depth case studies16 which tend to define implementation success in simplistic ways (e.g., continued use: yes or no).13 Even when EBHI implementation has been carried out on a broader scale and examined in the context of large-scale field studies, the “primary focus has been on scaling up to insure that implementation is successful during the course of the study rather than on addressing issues that may facilitate continued use of innovations when formal investigations come to an end and/or when extramural funding” or other forms of support are discontinued.17 (p 135) Consequently, there is much to be learned about the extent to which findings from short-term investigations of innovation implementation explain enduring measures of implementation success, such as the extent to which EBHIs become embedded in organizations that initially adopt them either as part of research or demonstration projects18,19 or under naturalistic conditions.1,2,10,11

This study advances a management-based model of implementation success which defines success in terms of both fidelity and assimilation. The model is intended to apply to organizations engaged in implementing EBHIs or other “fixed” innovations (i.e., innovations for which fidelity is essential to effectiveness),15 which are first adopted under naturalistic conditions rather than as part of a research or demonstration project. Differences in the conditions under which innovations are first adopted by organizations warrant independent research attention due to their likely important implications for the course and outcomes of innovation implementation efforts. For example, adoption decisions made in research versus real-world contexts may be associated with differences in: adoption motives (e.g., relative interest in advancing science versus improving performance); adoption intentions (e.g., trial versus permanent use); and consequential organizational attributes (e.g., availability of slack resources—expertise, time, money).20,21 Thus, while it is vital to understand factors that prompt innovations to endure in organizations when research or demonstration projects end, that context is not the focus of this analysis.18,19

Building on the innovation literature, including recent long-term investigations of the sustained use of innovations by organizations, and on research regarding strategic cognition, the study explores the premise that members of organizations’ top management teams (TMTs) (e.g., Chief Executive Officer [CEO]/Agency Director) affect the success of efforts to implement EBHIs by virtue of what they think (e.g., views held about strategic fit of innovations) and what they do (e.g., championship behavior displayed).16,22,23 This idea is supported by management scholars who contend that the success or failure of organizational innovation efforts may have more to do with factors related to or under the control of top management than it has to do with exogenous variables.24 The proposed model and the methodological paradigm used to examine it reflect these ideas as well.

This investigation draws on data gathered from a 7-year observational field study of the adoption, implementation and de-adoption of two evidence-based and two promising mental health innovations (EBHI) within one state mental health system.25 The analysis is concerned with explaining the degree to which these EBHIs are assimilated into routine practice and the extent to which they are implemented with fidelity by the behavioral healthcare organizations (BHO) that adopted them roughly 6 years earlier. Following Yin’s work, assimilation is defined as an organizational process that (1) begins when organizational decision-makers first become aware of an EBHI, (2) can lead to the adoption of the EBHI, and (3) which may culminate in the EBHI’s routinization or institutionalization by adopter organizations.26,27 This analysis is concerned with the latter phase of this process and seeks to explain the degree to which innovations become embedded into organizational operations. And, as with other investigations of innovation implementation, the EBHI implementation effort rather than the organization is the unit of analysis for hypothesis testing.12,13,16,22,27,28

Conceptual and Methodological Issues

Implementation success: assimilation and fidelity

Business and healthcare scholars have identified numerous categories of outcome variables that may be useful for gauging the success of organizations’ efforts to implement innovations.14,15,29,30 The category of indicators labeled by O’Connor and colleagues “measures of implementation, integration, and institutionalization” is especially relevant to this study because it includes variables which are useful for understanding the extent to which innovations have taken hold in the organizations that adopt them.14 (p 74) For example, measures include: the degree of implementation (e.g., the extent of use among intended users such as targeted staff); the extent of assimilation or institutionalization (e.g., the degree to which the innovation is embedded in regular organizational operations); integration (e.g., the extent to which the innovation is interconnected with other core organizational processes and structures); fidelity of implementation (e.g., the extent to which the innovation as deployed mirrors the innovation as designed); and the sustained or continued use of innovations (e.g., duration of use; whether or not the innovation is still being used at a given point in time). Fidelity of implementation and assimilation are the success measures most relevant to this investigation.

The assimilation of EBHIs into routine practice by the organizations targeted to adopt them is essential to fully realizing the benefits these innovations have to offer. The idea that assimilation is important to the success of public-sector program and practice innovations is not new. Over three decades ago, Yin emphasized the many challenges involved with implementing innovative programs in public organizations and ensuring that innovations endure under real-world circumstances,26 concerns that continue to be echoed by scholars interested in program sustainability.8,17–19,31,32 Yin cautioned researchers and policymakers against making premature judgments about the success of implementation efforts. He explained “…the implementation problem extends well beyond the adoption and early use of innovations” and argued that assimilation is the ultimate indicator of implementation success because the incorporation of innovations into routine operations reflects their true value to the organizations that adopt them.26 (p 383)

Since that time, organizational behavior13,16,33 and healthcare1,17,32 experts have continued to view assimilation as essential to reaping the benefits of the investments made in developing EBHIs. Assimilation is an important behavioral indicator of an organization’s commitment to an innovation and of the likelihood that innovation implementation will endure for the foreseeable future. In addition, full incorporation of an innovation into organizational operations theoretically signals the “disappearance stage” of assimilation, the phase when innovations such as EBHIs presumably become part of the status quo and, therefore, are no longer prime targets for elimination.27 These claims are less likely to apply to some of the other success measures noted above. For example, “continued use” may not be a reliable predictor of the likelihood an innovation will endure because continued use may hinge on the availability of external resources,34,35 or be motivated by other concerns (e.g., organizational image management).36 As a result, the degree to which EBHIs are assimilated into routine operations is the key implementation success measure used, and the major dependent variable for this analysis.

An organization’s failure to achieve the intended benefits of an innovation may be a function of implementation failure or a failure of the innovation itself.12 The latter explanation, a failure of the innovation itself, is less likely to apply to EBHIs and to other fixed innovations which are supported by effectiveness research. In contrast, implementation failure is the more likely explanation and, for EBHIs, it may take the form of inadequate fidelity. This is especially true for service innovations such as EBHIs for which maintaining fidelity tends to be far more challenging than it is for product innovations.15 Consequently, fidelity is the second key measure of implementation success used in this analysis, and it is expected to be linked to assimilation in potentially interesting ways with regard to EBHIs. For example, fidelity may be a prime factor considered in assessing the extent to which EBHI assimilation has occurred.

Factors affecting implementation success

Three important themes are apparent in the large, multidisciplinary body of research regarding factors and processes that affect the success of organizations’ efforts to implement innovations. First, implementation success is contingent on a multitude of factors, which span multiple levels of analysis28,36,37 from the individual level (e.g., attitudes, championship behavior),33,38 to the program level (e.g., implementation climate),12,29,39 to the organization level (e.g., organization size, slack resources),34,36,40 and to the system-level (e.g., innovation policy).41–43 Second, implementation success begins with first events which often occur well before formal implementation efforts get underway.13,16,17,22 For example, success has been tied to the readiness of organizations to deliver innovations;44,45 to the ‘readiness’ of state systems40 and broader systems41 to support the uptake of innovations; to attributes of innovation decision-making, planning, and prioritization processes;16 and, also to the tactics and strategies used by organizational leaders to engage stakeholders and to build buy-in.13,42 Third, as with other strategic issues and decisions, top management makes a difference: the cognitions and actions of members of organizations’ TMTs affect implementation success in pivotal ways.12–14,16,22,27,33 This third theme is prominent in the strategic decision-making and planning literatures, as well.13,22,23

This investigation is mostly concerned with the third theme: the role of top management. It advances and provides a preliminary test of a management-based model of implementation success which defines success in terms of fidelity and assimilation. The proposed framework is intended to apply to organizations which adopt EBHIs or other fixed innovations under ordinary circumstances, rather than as part of a research or demonstration project.

Top management and implementation success

As stewards of organizations’ interests, CEOs and TMT members engage in ongoing processes of interpreting strategic developments, making strategic decisions, and taking action in response to a wide array of issues that have implications for organizational performance, such as issues concerning organizational innovation, in general, and specific innovation initiatives, as well.23,46–49 With regard to the latter, senior managers’ strategic cognitions (SC) about specific innovation initiatives are bound to affect the stances organizations take toward them and the actions, responses, and outcomes that follow.22,23

Top management cognition and implementation success

Decisions regarding innovation are strategic decisions which typically are made by senior organizational managers.12 They are organizationally important, complex, non-routine, and tend to be made in the face of ambiguous or incomplete information and in a context of risk and uncertainty.50 Examples of major decisions regarding innovation that warrant the attention of senior management include: decisions to adopt,51 disengage from,34,35 sustain, or embed innovations into routine practice.13,16,19,27,52

Theory and research from managerial and organizational cognition, and the innovation literature suggest that the SCs of TMT members regarding particular innovations are important for understanding innovation decisions such as those noted above.23,53 For example, the cognitions of senior management about the risks involved with adopting EBHIs explained significant variability in EBHI adoption decisions51 just as top managers’ perceptions about risk have been useful for understanding organizations’ responses to a host of other strategic issues.50,54,55 Similarly, the SCs of CEOs about the likelihood that particular innovations present opportunities versus threats to organizational interests explained significant variability in the success of innovation implementation efforts measured years later according to a recent longitudinal analysis.22 This finding is consistent with research grounded in the strategic issue diagnosis paradigm which examines linkages between how TMT members frame important developments and issues in terms of widely studied categories and dimensions (e.g., threat versus opportunity; gain versus loss)56,57 and organizations’ responses to those issues.

Strategic cognitions regarding the fit or compatibility of particular innovations to organizational priorities are likely to be important as well. In fact, an exploratory study of the de-adoption of EBHIs revealed that TMT member cognitions about the extent to which EBHIs fit organizational needs discriminated between organizations that sustained EBHIs and those that decided to de-adopt them.34 This finding is consistent with research which suggests compatible innovations are more likely to be adopted, and to be easier to implement with success by organizational and individual adopters.12,29,58,59

These findings support the idea that information about top manager cognitions is of value for understanding implementation success. In other words, if innovation implementation is a process that involves the use of strategies to adopt and integrate EBHIs and to change practice patterns within specific settings,60 the strategic cognitions of senior management are likely to affect this process in noteworthy ways.

Top manager cognitions about the strategic fit of EBHIs are of central interest to this investigation. Strategic fit is defined here as the extent to which the implementation of a particular EBHI is compatible with organizational goals, interests, and/or values.

Top management action and implementation success

The actions taken by or under the direction of senior management also are expected to affect the success of organizations’ effort to implement innovations in both the short29 and long term.16 TMT member actions have both symbolic and instrumental value and can speak volumes about the extent to which organizations prioritize and support specific innovations and efforts to implement them.29,41–45,52

However, innovation-specific support from top management must be consistent, strong and visible in order to send an unequivocal message that a specific innovation is highly valued by the organization, and in order to effectively facilitate efforts to implement particular innovations.12,13,16,30,33,34 Supportive actions that may be taken by TMT members, or by others with top management support or approval, include but are not limited to: dedicating adequate resources and setting policy to favor the implementation of specific innovations;29,44 setting goals and priorities that support the use of specific innovations;13,16 implementing performance monitoring systems, and establishing reward and reinforcement criteria that support effective use of particular innovations;52 intervening to reduce obstacles to effective implementation;39 serving as innovation champions;22,27,33 demonstrating a hands-on approach to innovation implementation efforts;30 demonstrating leadership behavior that fosters rather than dampens a spirit of innovation;12,13,38 and actively advocating for changes to be made in external systems in order to increase the feasibility of implementing particular innovations.52

Actions such as those listed above have been consistently linked to the success of organizations’ effort to implement innovations, including EBHIs.16,20,27,29,39 In theory, this is because these actions are likely to foster a “climate for innovation implementation”12,39,61 and lay the groundwork for the implementation of particular innovations to be successful. In fact, Klein and colleagues, who coined the term, define an organization’s climate for innovation implementation as the extent to which the use of a specific innovation is perceived to be rewarded, supported, and expected within a given organization.12 Empirical evidence supporting the idea that climate for implementation is key to implementation success is available from varied contexts such as education,62 healthcare,61 manufacturing,29 and mental health.39

However, views about climate strength may vary from employee to employee for a variety of reasons. For example, views may vary if TMT actions in support of particular innovations are not strong, consistent and visible. They may also vary as a result of the extent to which employees are directly involved in delivering service innovations such as EBHIs. For instance, many EBHIs are delivered by small teams of employees and climate perceptions of team members may be very different from employees not directly involved with EBHI delivery.

This investigation is concerned with climate perceptions of individuals most involved with or knowledgeable about the delivery of EBHIs, including top managers and EBHI implementation leaders. Their perceptions are expected to provide the most meaningful indicators of climate for innovation implementation in this research context because climate is expected to have a visible impact on the ease with which these individuals carry out their daily work. The majority of the EBHI implementation efforts targeted for this analysis involve EBHIs delivered by a small team of employees. The remaining efforts involve an EBHI which is delivered by individuals who are part of a select group. In light of these issues and in order to maintain uniformity in sources of information about climate across EBHI implementation effort, information to assess climate for innovation implementation will be sought from the TMT member most involved with the implementation effort and the EBHI implementation manager. Climate information also will be sought from external EBHI consultants, in cases in which consultants are both available and knowledgeable about EBHI implementation efforts.

Cognition, action, and implementation success: the assimilation framework

Senior managers in organizations are agents of implementation success by virtue of what they think and what they do. As with other strategic issues and developments, their cognitions pertaining to organizational innovation, such as organizations’ efforts to implement particular innovations, are important to consider because strategic cognitions are likely to influence organizational actions and responses to innovations in noteworthy ways.23,27 These ideas are buttressed by empirical findings in the innovation literature, including recent long-term investigations of implementation success,16,22 and are reflected in the model guiding this investigation which is shown in Figure 1.

Figure 1
figure 1

Implementation success: factors explaining the assimilation of evidence-based healthcare innovations (EBHI)

Full size image

The model in Figure 1 suggests cognitions about the strategic fit of an innovation to an organization’s goals and interests will affect the strength of the climate for innovation implementation that develops. Stronger fit is likely to yield stronger climates because innovations that are well-suited to organizations’ interests are likely to be valued and prioritized. Thus, Figure 1 suggests that the effects of strategic fit on implementation success will work through the climate for innovation implementation for a specific innovation. Climate is conceptualized to be a composite construct which consists of several dimensions (e.g., dedicated resources: the extent to which resources are earmarked to support a specific innovation implementation effort)39 that contribute to the strength of the climate in an additive way.12 Stronger climates are expected to provide greater support for organizations’ efforts to implement specific innovations and to result in more successful innovation implementation efforts. Thus, the framework also suggests that a positive association will exist between climate for innovation implementation and the two implementation success measures: fidelity and assimilation. Finally, because fidelity is expected to be a necessary but insufficient condition for achieving assimilation of EBHIs, fidelity is expected to mediate the link between climate and assimilation.

The following five hypotheses capture these ideas, represented by Figure 1:

  1. Hypothesis 1:

    Top management cognitions about the strategic fit of an EBHI will be positively related to the strength of the climate for innovation implementation.

  2. Hypothesis 2:

    Climate for innovation implementation will be positively related to fidelity of implementation.

  3. Hypothesis 3:

    Fidelity of implementation will be positively related to assimilation.

  4. Hypothesis 4:

    Climate for innovation implementation will mediate the effects of strategic fit on fidelity of implementation.

  5. Hypothesis 5:

    Implementation fidelity will mediate the effects of climate for innovation implementation on assimilation.

Methods

Research context and larger study

This investigation utilizes data from a mixed-methods, naturalistic field study of the adoption,29 implementation,39 and de-adoption34 of four evidence-based and promising mental health practices (EBHIs) conducted over a 7-year period from 2001 to 2008 in one state mental health system. The EBHIs that were the focus of implementation efforts included two evidence-based, team-delivered practices (i.e., multi-systemic therapy [MST])63 and integrated dual disorder treatment [IDDT]);64 one promising, psychiatrist-delivered practice (i.e., the Ohio medication algorithms (OMA) based on the Texas medication algorithm);65 and one promising practice delivered by qualified clinicians (e.g., Cluster-based Outcomes Management [CBOM]).66 Thus, the delivery of focal EBHIs directly involved only a limited number of employees within adopting organizations. Most often, EBHI implementation efforts were carried out under the direction of a TMT member and an EBHI implementation leader. In many cases, an external EBHI consultant assisted with some aspects of EBHI implementation (e.g., fidelity monitoring).

Targeted EBHI implementation efforts

Just before the larger study (of which this investigation is part) got underway, statewide technical assistance centers (i.e., Coordinating Centers of Excellence or CCOEs) received start-up funding from the state mental health authority to market and educate behavioral healthcare organizations (BHO) about these EBHIs and their benefits. However, decisions to contact CCOEs, contract for CCOE services (e.g., training), and/or adopt EBHIs were left entirely up to BHOs.

BHOs that contacted CCOEs subsequently were approached about participating in the larger observational field study of which this investigation is part. Participation was entirely voluntary but 84 of the 89 initial contacts resulted in decisions to participate in the broader study.25 Approximately 60% or 50 of the 84 EBHI adoption decision processes that took place resulted in a decision to adopt an EBHI.51 An earlier analysis suggests, on average, EBHI “adopters” saw the risks of adopting as lesser and more manageable than non-adopters.51 Those 50 EBHI adoption decision processes which resulted in a decision to adopt subsequently proceeded with efforts to implement them.39 These 50 cases were of potential interest to this study of EBHI assimilation.

Prior to the start of this investigation, data for the larger study had been gathered over the course of three rounds, separated from one another by approximately 2 years. Forty-four of the 50 EBHI implementation efforts were still underway as of the third round of data gathering. Those 44 efforts were targeted for participation in this investigation. Participation entailed taking part in a fourth round of data-gathering, conducted roughly 2 years after round three.

Targeted informants

For each of the 44 targeted EBHI implementation efforts, participation in telephone interviews and surveys was sought from the TMT member or members (e.g., CEO/Agency Director; Chief Clinical Office) most involved with the EBHI implementation effort and from the EBHI project implementation manager. In other words, consistent with other innovation implementation studies that are grounded in strategic cognition and/or planning paradigms,13,16,22 two upper-level management informants who were most knowledgeable and/or involved with a given EBHI implementation effort typically were approached to participate in a fourth round of data gathering. The three Coordinating Centers of Excellence (CCOE) associated with the 44 targeted EBHI implementation efforts also were contacted to identify staff members most knowledgeable about each EBHI implementation effort for potential participation in a brief, fourth-round survey. All CCOEs were receptive and provided names and contact information for these individuals. However, there were 8 EBHI implementation efforts (of the 44) for which no CCOE experts could be identified due to limited contact or no recent contact between the CCOE and those 8 EBHI implementation sites. As with the previous 3 rounds, participation in round 4 was voluntary and confidentiality was guaranteed to all participants.

Archival data set

Table 1 provides an overview of the archival data set for the larger study from which data for this analysis is drawn. Column headings indicate the round in which data were gathered and row headings indicate the method used and source of those data. As noted in the footnotes to Table 1, data gathered during rounds one, two, and three of the larger study have been used in other analyses. To date, data from round four has been used only for this investigation.

Table 1 Focal EBHI efforts, and data sources reflected in the archival data set, by contact period
Full size table

As shown under the column labeled “4th Contact” in Table 1, information about all 44 targeted EBHI implementation efforts was gathered during round four. Surveys were completed by 70 targeted BHO staff members and 34 targeted CCOE external consultants. Together, these surveys addressed all 44 of the targeted EBHI implementation efforts. Those 44 efforts included: 27 (61%) pertaining to IDDT, 7 (16%) pertaining to MST, and 10 (23 %) dealing with CPOM. No OMA implementation efforts were targeted for participation in round four because there were no OMA efforts still underway as of the third round of data-gathering.

Surveys and Measures

Multi-item measures included in round four BHO and CCOE surveys were adapted from prior analyses34,39,51 and were used to operationalize the concepts shown in Figure 1: strategic fit, climate for innovation implementation, fidelity, and assimilation. The focal entity for all relevant survey items was either the EBHI or the EBHI implementation effort.28 Operational measures for the four concepts made up a small portion of the items in the BHO versions of the survey but nearly all of the items included in the CCOE version of the survey.

Versions of the survey

The CCOE version of the survey included far fewer items than the BHO versions because some CCOE consultants were required to complete multiple surveys. This was because a single individual often was identified as the most knowledgeable external consultant for several EPHI implementation efforts. In an effort to minimize burden and increase the likelihood that external consultants would complete multiple surveys, CCOE surveys were kept as brief as possible. Information provided by CCOE staff was expected to provide important comparative data for the analysis.

The BHO surveys were designed to minimize burden as well, and to reduce concerns arising from common method variance. A decision-maker version of the survey was designed to be combined with an implementer version of the survey for each EBHI implementation effort. An abbreviated version of the combined surveys also was developed to accommodate BHOs for which only one knowledgeable informant was available to participate in the investigation.

Multi-item scales

Strategic fit was measured with eight items on the CCOE version of the survey and four items on the BHO version. Respondents were asked to indicate the extent to which they agreed, on a seven-point scale where 7 = “strongly agree” and 1 = “strongly disagree”, with statements such as: “EBHI Name fits well with the mission and overall goals of this organization” and “EBHI Name is compatible with the treatment philosophy at this organization”. A “Don’t Know” option also was available. Higher scale scores indicate better strategic fit.

Climate for innovation implementation was measured with nine items on the CCOE version of the survey and 25 items on the BHO version. Respondents were asked to indicate the extent to which they agreed, on a seven-point scale where 7 = “strongly agree” and 1 = “strongly disagree”, with statements such as: “Top administrators strongly support the implementation of EBHI Name at this organization” and “Adequate resources are available for implementing EBHI Name”. A “Don’t Know” option also was available. Higher scale scores indicate a stronger, more favorable climate.

Fidelity of implementation was measured with four items on both the CCOE and BHO versions of the survey. Respondents were asked to indicate the extent to which they agreed on a seven-point scale where 7 = “strongly agree” and 1 = “strongly disagree” with statements such as “EBHI Name is being implemented at this organization as prescribed by experts” and “EBHI Name is being implemented ‘to the letter’ as prescribed by its developers”. A “Don’t Know” option also was available. Higher scale score indicate better fidelity.

Assimilation was measured with five items on the CCOE survey and four items on the BHO survey. Respondents were asked to indicate the extent to which they agreed on a seven-point scale where 7 = “strongly agree” and 1 = “strongly disagree” with statements such as “EBHI Name is seen as a regular part of programming at this organization” and “EBHI Name will be part of the way this organization does business for the foreseeable future”. A ‘Don’t Know” option also was available. Higher scale scores indicate greater assimilation.

Results

Participants and EBHI implementation efforts

A total of 104 surveys were completed by BHO management (n = 70) and CCOE external consultants (n = 34). Combined, these surveys address all 44 implementation efforts. Seventy BHO management informants involved with 35 (of the 44) implementation efforts completed round-four surveys. The distribution of EBHIs represented by BHO surveys (i.e., 66% IDDT; 17% MST; 17% CBOM) parallels the distribution of the full group of 44 efforts reported above. On average, two BHO surveys were completed per effort, most typically by the CEO and the EBHI project manager. This rate of participation is typical of strategic decision-making studies, including studies of innovation adoption and implementation.13,16,22

CCOE consultants completed surveys pertaining to 34 of the 44 EBHI implementation efforts. As noted, knowledgeable external consultants were not available for 8 of the 44 efforts. The distribution of EBHI efforts addressed by CCOE surveys (i.e., 50% IDDT; 21% MST; 29% CBOM) is similar to that for the 44 efforts. Consultants were encouraged to respond “Don’t Know” to survey items for which they lacked current knowledge.

There were only 25 EBHI implementation efforts for which both BHO and CCOE survey data were available (i.e., ‘common EBHI efforts’). The EBHIs addressed by these common EBHI efforts is similar to the full distribution (i.e., 52% IDDT; 24% MST; 24% CBOM).

Multi-item scales

As suggested in Table 2, the internal consistency reliability of the four scales based on CCOE survey data as assessed with Cronbach’s alpha is quite strong, ranging from .81 to .90. Table 2 also suggests there is substantial variability on scale scores from the 34 CCOE surveys. Mean scale scores tend to be slightly above the seven-point scale midpoint (e.g., mean = 4.6/7.0) and the range of scores within scale is between 4.1 and 6.0 points.

Table 2 Strategic fit, climate, fidelity, and assimilation scales: Scale reliability and scale descriptive statistics
Full size table

As shown in Table 2, Cronbach alphas for scales based on BHO survey data are strong as well and range from .80 to .91. Table 2 also indicates there is reasonable variability in response on the four scales based on the 70 BHO surveys. Mean scale scores tend to be slightly above the seven-point scale midpoint (i.e., 5.0/7.0), and the range of responses on each of the four scales also is satisfactory (i.e., between 3.5 and 5.0 points).

Test of hypotheses

According to both CCOE and BHO informants, the vast majority of the 44 EBHI implementation efforts targeted for this investigation were described as actively implementing the focal EBHI as of round-four data gathering. Exceptions to this description were as follows: two efforts were described as recent de-adopters of EBHIs, and one was described as a recent re-adopter. Even so, all 44 efforts were included in hypothesis testing.

Simple bi-variate correlations were computed to test the first three study hypotheses which predict positive associations between major linkages shown in Figure 1: the association between strategic fit and climate for implementation (Hypothesis 1: H1); the association between climate for implementation and fidelity (Hypothesis: H2); and the link between fidelity and assimilation (Hypothesis 3: H3). Findings are displayed in Table 3.

Table 3 Inter-correlations among fit, climate, fidelity and assimilation scale scores derived from organization/BHO surveys and consultant/CCOE surveysa
Full size table

As noted in Table 3, scale inter-correlations pertaining to BHO surveys are displayed above the main diagonal; scale inter-correlations based on CCOE surveys are displayed below the main diagonal, and scale inter-correlations pertaining to the 25 “common EBHI efforts” for which both CCOE and BHO surveys were completed are shown on the main diagonal. In other words, coefficients displayed on the main diagonal represent correlations between CCOE and BHO scores on the same scale (e.g., r = .346, p <.05 is the correlation between strategic fit as assessed on the CCOE survey and fit as assessed on the BHO survey).

Findings in Table 3 lend support for Hypotheses 1, 2, and 3. They suggest a positive relationship exists between strategic fit and climate (H1); between climate and fidelity (H2); and between fidelity and assimilation (H3). All three hypotheses were supported by data pertaining to the 35 implementation efforts assessed by BHO surveys (i.e., Pearson r ranges in value from .62 to .79, p < .01), and by data pertaining to the 34 efforts assessed by CCOE surveys (i.e., Pearson r ranges in value from .54 to .74, p < .01).

The Pearson correlation coefficients shown on the main diagonal of Table 3 pertain to the 25 “common EBHI efforts.” They represent correlations between BHO informant ratings and CCOE informant ratings for a given scale (e.g., strategic fit). Only two of the four coefficients based on the 25 common EBHI efforts were significant: strategic fit (r = .346, p < .05, n = 25), and climate for innovation implementation (r = .375, p < .05, n = 25). Further, as shown in Table 4, the results of paired T tests comparing scale score ratings between the CCOE and BHO surveys revealed that BHO informants, on average, present a rosier view of strategic fit (paired T = 6.78 (24), two-tailed p < .01), climate (paired T = 2.65 (24), two-tailed p < .01) and assimilation (paired T = 2.65 (24), two-tailed p < .01) compared to CCOE external consultants. Although not significant, this trend holds for assessments about implementation fidelity as well.

Table 4 Paired-sample T tests between consultant/CCOE survey scale scores and organization/BHO survey scale scores for twenty-five (25) EBHI implementation efforts assessed on both surveys
Full size table

Hypothesis 4 (H4) predicts that climate for implementation mediates the effects of strategic fit on fidelity. Hypothesis 5 (H5) predicts that fidelity mediates the effects of climate for implementation on assimilation. Overall mediation is tested for each hypothesis using Baron and Kenny’s67 four-step approach in which several regression analyses are conducted and significance of the coefficients are examined in each step.

Tables 5 and 6 display results of the four-step regression approach used to test Hypothesis 4. Table 5 displays results using data from organization/BHO surveys. Table 6 displays results using data from CCOE surveys. Overall findings reported in these tables are quite similar and provide support for the notion that climate for implementation mediates the effects of strategic fit on fidelity. However, regression results based on BHO survey data suggest full mediation (i.e., beta for fit is no longer significant at step 4, Table 5) compared to results from CCOE survey data which suggest partial mediation (i.e., beta for fit remains significant at Step 4, Table 6).

Table 5 Organization1 survey mediated regression: Does climate mediate the effect of strategic fit on fidelity?
Full size table
Table 6 Consultant1 survey mediated regression: Does climate mediate the effect of strategic fit on fidelity?
Full size table

Tables 7 and 8 display results of Baron and Kenny’s four step regression approach used to test Hypothesis 5. Table 7 displays results using data from organization/BHO surveys. Table 8 displays results using data from CCOE surveys. Overall findings reported in these tables are quite similar and provide support for the notion that fidelity mediates the effects of climate for implementation on assimilation. Regression results for both analyses suggest that mediation is partial (i.e., beta for climate remains significant at Step 4, Tables 7 and 8).

Table 7 Organization1 survey mediated regression: Does fidelity mediate the effect of climate on assimilation?
Full size table
Table 8 Consultant1 survey mediated regression: Does fidelity mediate the effect of climate on assimilation?
Full size table

Discussion

Scales used in the analysis were internally consistent and demonstrated reasonable variability based on data from the BHO and CCOE surveys. For the BHO survey, average scale scores for the strategic fit, climate, fidelity and assimilation scales tended to be slightly above the midpoint on a seven-point scale (e.g., 5 = “somewhat agree” resources are adequate to support implementation). In all cases, higher scale scores indicate more favorable ratings.

Scale scores based on the CCOE survey were a bit less rosy compared to scores from the BHO survey. In fact, when considering data pertaining to the 25 common EBHI implementation efforts, BHO scale scores were significantly higher for three of the four scales. Although differences were not significant, the trend also held for the fidelity scale.

Study findings also provide preliminary support for major hypotheses and the underlying management-based model of implementation success. Data from both the CCOE and BHO surveys revealed significant positive associations between strategic fit and climate (Hypothesis 1); between climate and fidelity (Hypothesis); and between fidelity and assimilation (Hypothesis 3). In other words, survey data from BHO top managers and EBHI program leaders, and survey data from knowledgeable and involved CCOE external consultants suggest these concepts co-vary as predicted. In addition, the similarity in the magnitude and patterns of bi-variate associations between the two surveys is noteworthy.

Support was found for Hypotheses 4 and 5 which predicted mediation effects. More specifically, Hypothesis 4, which predicted that climate for innovation implementation would mediate the effects of strategic fit on fidelity, was supported by both BHO and CCOE survey data. Hypothesis 5, which predicted that fidelity of implementation would mediate the effects of climate for implementation on assimilation, was supported by data from both surveys as well.

Limitations

The investigation has many noteworthy and apparent weaknesses. The design is observational and cross-sectional rather than experimental and longitudinal. Findings are correlational and little can be said about causality. Findings may be more useful for generating questions than for testing theory. Statistical power and variability is limited, small sample sizes raise questions about the validity of findings pertaining to mediational analyses,68 and operational measures of concepts are not identical for the BHO and CCOE surveys. Common method variance is a concern, and attributes of the context (e.g., a system with CCOEs) and of EBHIs (e.g., team-based) may limit the generalizability of findings to other EBHIs and settings.

The investigation also has noteworthy strengths. The framework is theory-based and focuses on understanding the assimilation of fixed innovations, such as EBHIs adopted under naturalistic conditions. Assimilation is examined several years post-adoption, which provided time for EBHIs to become embedded in organizational operations. Scale reliability is strong, methods are grounded in the managerial and organization cognition literature, and perspectives of internal and external subject matter experts are compared.

Implications for Behavioral Health

The issues of assimilation and sustainability as they pertain to evidence-based and promising healthcare innovations (EBHI) will continue to gain momentum and currency in the public healthcare sector as fiscal constraints drive policymakers to demand evidence about the likelihood EBHIs will be adopted under naturalistic conditions by the organizations they are intended for, and the likelihood EBHIs will become embedded in regular operations among organizations that choose to adopt them. Healthcare experts contend that we don’t really know what becomes of organizations’ efforts to implement healthcare innovations,1 but there seems to be general recognition that an “assimilation gap”69 exists between rates of initial adoption and rates of continued use. Clearly, a lot needs to be learned about the assimilation of EBHIs, so, it’s important to seek-out real-world opportunities to gain insight about this important phenomenon.

This naturalistic case study was conducted with that spirit in mind. It focuses on key management-related factors that account for variability in assimilation. At the outset of the larger project of which this investigation is part, there was no intention to explore questions about factors that explain the extent to which EBHIs get embedded into organizational operations among a group of long-term implementers of EBHIs. In fact, there was no expectation that there would be long-term implementers of EBHIs to study. The realization that a sizeable group of organizations was still implementing EBHIs roughly 4-years post-adoption presented an unexpected opportunity to begin to explore this question by implementing a fourth-round of data-gathering roughly 2 years later (i.e., 6 years post-adoption). This investigation was designed to take advantage of that opportunity.

An important premise guiding the investigation is that members of organizations’ TMTs are agents of EBHI assimilation by virtue of what they think and what they do. Yet, theory and research from managerial and organizational cognition suggest it may be particularly important to pay attention to what TMT members think because their cognitions are likely to significantly affect the stances taken by organizations toward innovations and the actions and responses that follow. Moreover, because innovation decisions (e.g., decisions to adopt, sustain, de-adopt and embed innovations) are strategic decisions typically made by senior management, it’s imperative to understand top managers views about the extent to which innovations fit with changing organizational needs and priorities, and the extent to which efforts to implement particular innovations are seen as viable and valuable to the organization. This information is likely to provide important insights about the likelihood of assimilation.

Yet, findings also raise questions about whose point of view matters most when it comes to understanding the assimilation of fixed innovations. As noted, this investigation assumes TMT perceptions are pivotal because they are likely to drive organizational response and action. Even so, it is important to examine the extent to which TMTs cognitions mesh with or are influenced by the perceptions of others, such as external consultants and front-line EBHI implementers. In other words, if TMT perceptions matter most, it’s essential to get a better understanding of the extent to which strategic cognitions are shaped by social information.

Patterns of findings based on those 25 common EBHI efforts assessed on both the CCOE and BHO surveys suggest knowledgeable informants may not see eye-to-eye on important matters related to innovation assimilation, such as the extent to which climate for innovation implementation is strong. Differences in perspectives may simply be a function of differences in exposure and access to information. On the other hand, differences in perspectives may be a function of differences in the frames of reference considered in arriving at ratings. For example, for those CCOE external consultants who completed surveys about multiple EBHI implementation efforts, ratings provided about a particular EBHI effort may have been made by drawing comparisons with the other EBHI efforts to be rated. It’s important to understand the extent to which differences in frames of reference or other factors account for why internal and external expert informants may not see eye-to-eye on the progress and outcomes of particular EBHI implementation efforts.

The highly similar pattern of co-variation observed among major model variables (e.g., climate and fidelity) based on CCOE and BHO survey data raises a very intriguing question. Do these findings represent true co-variation among study variables or the implicit models held by top managers and external consultants about how these variables should be inter-related in a rational world? The answer to this question has significant implications for the management of innovation in real-world settings, and for the paradigms and methods used to study implementation as well.

Finally, the study raises important questions about the nomological network surrounding the concept of assimilation. While major model concepts are grounded in theory and research, more thought needs to be given to the meaning of these concepts as they pertain to assimilation. Are they antecedents, aspects or indicators of assimilation? For example, is a strong climate for implementation an indication that an EBHI has become embedded in organizational operations? Or, is a strong climate an antecedent to assimilation? Is fidelity a prerequisite for assimilation to occur when it comes to EBHIs and other fixed innovations? Does assimilation mean something different for fixed versus adaptive innovations?15 These are important conceptual questions with practical and scientific implications for policymakers, practitioners and researchers to consider.