Introduction

Significant deficits remain in the development of implementation strategies that increase the use of evidence-based practices (EBPs) in routine mental health care (Collins et al. 2011; Insel 2009; NIMH 2008). Nearly half of all implementation strategies tested in mental health services fail to influence the targeted implementation, services, or clinical outcomes (Powell et al. 2014) and the most promising strategies are the least economically feasible (Novins et al. 2013). We argue that developing more effective and efficient implementation strategies requires a better understanding of the mechanisms that explain how these strategies influence implementation outcomes (Williams and Glisson 2014b). However, almost no studies test the mechanisms that link implementation strategies to outcomes and no studies have supported a hypothesized linking mechanism in mental health settings (Williams 2015).

The present study tests the mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians’ EBP adoption and use in a 4-year randomized controlled trial. ARC is an organizational intervention that facilitates the adoption and integration of empirically-supported clinical interventions (i.e., EBPs) and other innovations into routine practice by changing organizational cultures and reducing job-related service barriers (Glisson et al. 2010; Powell et al. 2012). ARC has demonstrated positive main effects on clinician turnover, organizational culture, clinicians’ work attitudes, and clinical outcomes for youth (Glisson et al. 2006, 2013, 2010). However, no studies have tested ARC’s effects on clinicians’ EBP adoption or the cross-level change mechanisms that explain how ARC influences clinicians’ practice behaviors. In the present RCT, we examine the cross-level contextual, motivational, and barrier-related mechanisms that explain ARC’s effects on clinicians’ EBP adoption and use. Our outcome criteria distinguish between clinicians including specific EBPs in their treatment repertoires (i.e., adoption) and the extent to which they incorporate EBPs in their work with clients (i.e., use).

Mechanisms of Change in ARC

The ARC organizational intervention is designed to improve an organization’s effectiveness and support clinicians’ EBP adoption by addressing two theoretically- and empirically-supported predictors of clinicians’ practice behaviors: (a) organizational culture, and (b) organizational capacity to reduce job-related service barriers (Williams and Glisson 2014b). According to organizational culture theory, organizations that deliver mental health services are characterized by norms and behavioral expectations (i.e., cultures) that explain organizational differences in service quality and outcomes (Cooke and Rousseau 1988; Hartnell et al. 2011; Hofstede et al. 1990; Olin et al. 2014). The norms and expectations that characterize an organization’s culture influence clinicians’ practice behaviors by providing cues regarding the behaviors that are most valued and likely to be rewarded and by providing social and material contingencies that direct clinicians’ work priorities (Schein 2004; Trice and Beyer 1993).

According to behavioral science theories such as the Theory of Planned Behavior (Ajzen 1991), culture-based social processes create shared outcome expectancies, social norms, and beliefs that influence individuals’ intentions to engage in select behaviors (Jaccard et al. 1999). Intentions represent self-instructions to perform a behavior or to obtain a specific outcome and describe a person’s motivation to perform a targeted behavior (Sheeran 2002). The Theory of Planned Behavior construes intentions as the primary cross-level mechanism through which culturally-defined norms and expectations in a work environment influence individual practice behaviors (Fig. 1). The ARC strategy builds on this logic to argue that changes in an organization’s culture lead to changes in clinicians’ intentions to engage in innovative behavior such as the adoption and use of EBPs (Williams and Glisson 2014b).

Fig. 1
figure 1

Hypothesized cross-level mechanisms of the ARC organizational implementation strategy

Full size image

A large body of correlational research, summarized by a recent meta-analysis of 84 studies (Hartnell et al. 2011) supports the association between organizational culture and innovation behavior in the workplace. Hartnell and colleagues (2011) documented a population average correlation of ρ = .59 between organizational culture and work-related innovation behavior across a range of business, industrial, and healthcare settings. In mental health settings, several studies have demonstrated significant variability in mental health agencies’ cultural norms and expectations (Glisson et al. 2008a, 2012a) and strong associations between organizational culture and clinicians’ practice attitudes, behaviors, and clinical outcomes (Aarons et al. 2012; Glisson et al. 2008b; Olin et al. 2014; Williams and Glisson 2013, 2014a).

Building on this research, the ARC strategy is designed to create proficient organizational cultures as a primary mechanism for supporting mental health service innovations such as the adoption and use of EBPs. Clinicians who work in agencies characterized by proficient cultures report they are expected to prioritize improvement in client well-being and to be competent in the use of up-to-date treatment practices such as EBPs (Glisson et al. 2008a). Proficient organizational cultures have been linked to more positive clinician attitudes toward EBPs, higher quality mental health services, and superior client outcomes in mental health and social service settings (Aarons et al. 2012; Olin et al. 2014; Glisson et al. 2013; Williams and Glisson 2013, 2014a). In the present study, organizational norms and expectations that emphasize proficiency are expected to increase clinicians’ intentions to adopt EBPs and to explain ARC’s effects on clinicians’ EBP intentions, adoption, and use. The study provides an initial test of ARC’s cross-level effects on clinicians’ EBP adoption and use through this hypothesized contextual–motivational mechanism (Fig. 1).

A second mechanism targeted by ARC involves the development of an organization’s capacity to identify and reduce job-related service barriers. Numerous studies of EBP adoption in mental health settings indicate job-related barriers are among the most widely cited impediments to clinicians’ EBP adoption even when EBP use is mandated by governmental authorities and clinicians have ready access to EBP training and consultation (Aarons et al. 2009; Baer et al. 2009; Gioia and Dziadosz 2008; Jensen-Doss et al. 2009). Examples of job-related barriers to EBP adoption and use include: an agency’s refusal to modify policies or workflow to accommodate EBPs, failure to provide materials necessary for EBPs, and failure to provide clinical supervision that supports EBPs (Bartholomew et al. 2007; Raghavan et al. 2007). These data are consistent with implementation theories such as the Capacity, Opportunity, and Motivation model (Michie et al. 2011) which predict that job-related barriers limit individuals’ enactment of EBP behaviors. The ARC strategy addresses this hypothesized determinant through a structured process that develops the organizational infrastructure and capacity to identify and remove job-related barriers to service innovation and effectiveness. The present study provides an initial test of this second ARC mechanism which is referred to as a barrier reduction mechanism.

The change mechanisms described above are depicted in the conceptual model shown in Fig. 1. As shown in the model, ARC is expected to increase clinicians’ EBP adoption and use through two experimentally-manipulated mechanisms: (a) a contextually-induced increase in clinicians’ EBP intentions (i.e., contextual–motivational mechanism), and (b) the reduction of job-related EBP barriers (i.e., barrier reduction mechanism). Based on the model, we tested the following hypotheses: (1) ARC will increase clinicians’ EBP adoption and use, (2) ARC’s effects on EBP adoption and use will be mediated by improved proficiency culture leading to increased EBP intentions, (3) ARC’s effects on EBP adoption and use will be mediated by a reduction in job-related EBP barriers, and (4) the contextual–motivational mechanism and the barrier reduction mechanism will explain unique variance in clinicians’ EBP adoption and use. The study tests these hypotheses in a randomized controlled trial using longitudinal data sequenced over a 4 year period.

Method

Participants

The study included fourteen non-profit specialty mental health agencies purposefully selected from a large mid-Western metropolitan area in the United States. Agencies were recruited by inviting their chief executive officers (CEOs) to attend a conference at a local university in order to learn about the project. CEOs who were interested in the study met individually with the study investigators to obtain additional information about the study and to provide information necessary to assess their agency’s eligibility. Agencies were chosen to reflect the structural and workforce characteristics typical of children’s mental health service organizations in the United States (Schoenwald et al. 2008). Each agency employed 10 or more clinicians that served youth referred for emotional or behavioral disorders. In order to avoid selection of “early adopters” (Rogers 2003), agencies were excluded from the study if they had implemented strategic initiatives related to EBP adoption in the 12 months prior to recruitment or if they were affiliated with a mental health services research network. A total of 16 agencies that met criteria were selected for the study based on this process; however, two agencies declined, resulting in a sample size of 14. Following approval by the CEO, clinicians were recruited for participation during regularly scheduled staff meetings at which they provided written informed consent. Similar to national samples (Schoenwald et al. 2008), clinicians working in the agencies delivered a variety of mental health services to youth (e.g., individual psychotherapy, family therapy, skill-training, groups) with the specific theoretical orientation and approach for each youth determined by individual clinicians (e.g., cognitive behavioral, integrative-eclectic, family systems, behavioral).

A total of 475 clinicians participated in the study (n = 259 clinicians in ARC agencies, n = 216 clinicians in control agencies), representing 86 % of clinicians in agency programs that provided clinical services to youth and their families. On average, 34 clinicians per agency participated in the study (SD = 23.88, min = 8, max = 96). Participating clinicians exhibited a variety of levels of training and educational backgrounds with the most common terminal degrees at the bachelor’s (19.2 %, n = 91) and master’s levels (73.7 %, n = 350) and the most common majors in social work (38.3 %, n = 182) or an allied health field such as counseling (29.9 %, n = 142). The majority of clinicians were highly experienced, working in mental health settings for an average of 9.14 years (SD = 8.87), female (82.1 %, n = 390), and white (82.5 %, n = 392) with an average age of 36.42 years (SD = 11.65). Clinicians in the ARC and control conditions did not differ on gender, age, race, education, years of experience, or turnover (all p’s > .05).

Procedure

The study was designed as a randomized controlled trial in which agencies were matched on size (i.e., number of clinicians and budget) and one agency from each matched pair was randomly assigned to participate in the ARC intervention for 36 months. Agencies in the control condition received no intervention, although clinicians in these agencies were contacted at equal intervals using identical protocols to collect identical information regarding organizational culture, EBP intentions, job-related EBP barriers, and EBP adoption and use. Following completion of data collection for the study, CEOs in the control group were offered a 3-day workshop in which they received feedback on their organizations’ culture profiles along with information on the use of ARC to improve services in their organizations.

Clinicians in the ARC and control groups completed study questionnaires on-site at their agencies at baseline (Time 1 or 6–8 weeks prior to the intervention phase), two-thirds of the way through the ARC intervention period (Time 2 or 24-months post-baseline), immediately upon completion of the ARC intervention period (Time 3 or 36-months post-baseline), and at 12-month follow-up (Time 4 or 48-months post-baseline). Questionnaires were sequenced and timed based on theory-derived expectations regarding the hypothesized mechanisms. Questionnaires for both groups included scales assessing organizational culture (Time 1 and 2), intentions to adopt EBPs (Time 3), job-related EBP barriers (Time 4), and EBP adoption and use (Time 4).

Participating clinicians provided written informed consent and completed questionnaires in face-to-face meetings scheduled during regular work hours without supervisors or managers present. To ensure confidentiality, research staff unaffiliated with the agencies administered the questionnaires and collected them from clinicians immediately upon completion. Clinicians in all agencies were notified at the beginning of the project that participation in the study was voluntary and were able to opt out by avoiding scheduled meetings or by returning unfilled questionnaires with no penalty or knowledge of their participation by supervisors. Clinicians were not compensated for participation in the study. Average clinician turnover was 28 percent annually at baseline and because of turnover and new hires, sample sizes varied from a high of 207 clinicians at baseline to a low of 191 clinicians at 12-month follow-up. The overall response rate among participating clinicians across four waves of data collection was 86 %.

ARC Intervention

The procedures used to train the ARC specialists and the ARC manuals used to guide the implementation strategy were validated in three prior randomized controlled trials (Glisson et al. 2006, 2010, 2013). Detailed specification of the ARC intervention and a description of the ARC procedures used in the present study are available in manuals published by the University of Tennessee Children’s Mental Health Services Research Center (cmhsrc.utk.edu). Briefly, ARC employs external facilitators (i.e., ARC Specialists) who work with agency CEOs and administrators, an internal ARC liaison, and front-line clinical teams to: (a) embed five principles of service effectiveness into the agency’s operating procedures, (b) develop an organizational infrastructure (e.g., vertically integrated organizational action team and clinical level treatment teams) and tools (e.g., continuous quality improvement) necessary to identify and remove barriers to service effectiveness, and (c) generate staff and leadership mental models that support innovation and service improvement efforts. These strategies are enacted during a three-year process and are designed to create proficient organizational cultures that support the use of effective practices (e.g., EBPs) and an organization’s capacity to eliminate barriers that hinder service effectiveness.

In the present study, ARC was facilitated by external ARC specialists during a 36-month period from 2010 to 2013. ARC specialists had advanced graduate degrees and 5 or more years of experience working with mental health and social service agencies. Training for specialists included reading ARC manuals and facilitation guides, didactic sessions, and weekly consultation and supervision for the duration of the intervention period. Specialists facilitated the ARC intervention using separate manuals for team leaders and team members.

Measures

ARC Fidelity

Fidelity to the ARC implementation strategy was assessed as a manipulation check at the conclusion of the ARC intervention period using seven items from the ARC principles questionnaire (APQ). The APQ was validated in two prior RCTs of ARC in which it demonstrated reliability, sensitivity to change, and discriminant validity with respect to agencies in ARC and control conditions (Glisson et al. 2010, 2013). Clinicians complete scale items that refer to the unit-level enactment of ARC principles and to the completion of planned ARC activities within the previous month. Sample items include “Our program makes changes to be more effective in serving clients” and “All service team members participate in decisions to improve services.” Items were accompanied by a 5-point rating scale ranging from 1 (never) to 5 (always). Alpha reliability of this scale was 0.75.

Organizational Proficiency Culture

Organizational proficiency culture was assessed using the 15-item proficiency scale from the Organizational Social Context (OSC) measure (Glisson et al. 2012a, b, 2008a). The factor validity of the OSC (including the proficiency scale) was confirmed in two national studies of children’s mental health clinics and child welfare agencies, respectively, and subsequent studies confirmed its predictive validity for clinicians’ EBP attitudes (Aarons et al. 2012), program sustainability (Glisson et al. 2008b), clinician turnover (Glisson et al. 2008b), service quality (Olin et al. 2014), and youth mental health outcomes (Glisson et al. 2013; Williams and Glisson 2013, 2014a). Items assessing proficiency culture refer to shared norms and behavioral expectations that clinicians will place the well-being of each client first and will be competent and have up-to-date knowledge of effective treatments. Items were accompanied by a 5-point rating scale ranging from 1 (never) to 5 (always). Alpha reliability for the proficiency scale was .89.

Clinicians’ responses to the proficiency culture items were aggregated to the agency level based on a referent-shift consensus composition model (Chan 1998) and evidence of adequate within-agency agreement using the r wg(j) index (James et al. 1993). A cutoff of r wg(j) = .7 is recommended to support aggregation to the unit level (LeBreton and Senter 2008). Examination of the r wg(j) values for all 14 agencies indicated an average r wg(j) of .97 with a range from .91 to .99. Given this evidence of adequate within-agency agreement, aggregate agency-level scores were used to characterize the organizational cultures of participating agencies. Agency scores were converted to T-scores with a μ = 50 and σ = 10 using national norms from the children’s mental health agency sample (Glisson et al. 2008a).

EBP Intentions

Clinicians’ intentions to adopt EBPs (hereafter EBP intentions) were measured using five items developed following well-established procedures for constructing behavioral intention scales based on the theory of planned behavior (Fishbein et al. 2001). Items referred to clinicians’ intentions to adopt EBPs in their work with clients and to behaviors indicative of intentions to adopt EBPs. Following prior research on EBP adoption in children’s service systems (Aarons et al. 2009), the instructions defined an “evidence-based practice” as “a specific treatment protocol that has been developed through research and is supported by the results of controlled treatment studies.” Sample items include “I intend to use an EBP in each treatment session” and “Out of the next 10 new clients you see, how many would you expect to treat using an EBP?” Items were accompanied by a 7-point scale ranging from 1 (strongly disagree) to 7 (strongly agree) or an 11-point scale ranging from 0 to 10, as appropriate. Alpha reliability of this scale was 0.80.

Job-Related EBP Barriers

Job-related EBP barriers were assessed with five items rated on a scale from 1 (strongly disagree) to 7 (strongly agree). Items referred to job-related EBP barriers reported in the literature on EBP adoption in mental health settings with an emphasis on policies and procedures that support or inhibit EBP adoption (Aarons et al. 2009; Bartholomew et al. 2007; Gioia and Dziadosz 2008; Raghavan et al. 2007). Clinicians’ indicated the extent to which they experienced EBP barriers within the last 6 months. Sample items include “My agency does not provide materials—such as workbooks or technology—that are necessary to use EBPs,” and “My agency would not change paperwork or forms to accommodate the use of an EBP.” Alpha reliability for this scale was 0.76.

EBP Adoption and Use

Guided by the taxonomy of implementation outcomes proposed by Proctor et al. (2010) EBP adoption was conceptualized in this study as any action to try or employ an EBP including a clinician’s initial implementation of an EBP. This definition supports two related but distinct components of assessing EBP adoption—an intervention-centered approach that assesses whether a clinician has adopted a specific EBP and a client-centered approach that assesses the extent to which a clinician uses or tries to use EBPs with clients on her or his caseload. Both facets of adoption were assessed in the present study.

First, clinicians’ EBP adoption was measured with a condensed version of the EBP checklist developed by Walrath et al. (2006) and validated in prior research on EBP adoption in children’s mental health settings (Aarons et al. 2009). Clinicians completed nine items selected a priori based on the client populations served by clinicians in this study and the most up-to-date evidence regarding the treatment of emotional and behavioral disorders of youth. Selected EBPs included those addressing both externalizing (e.g., parent–child interaction therapy) and internalizing (e.g., exposure/systematic desensitization) problems. Items referred to whether or not clinicians had used each specific EBP with any client in the last 6 months. Clinicians’ responses were dichotomized to indicate whether or not clinicians had adopted any one or more of the specific EBPs within the last 6 months (no = 0, yes = 1).

Second, clinicians’ EBP use was assessed as the percentage of clients currently treated using EBPs. After reading a definition of EBP as the use of specific protocols supported by research, clinicians completed a broadband measure indicating the percentage of clients they currently treat using an EBP (0-100 %). To contextualize clinicians’ responses and to ensure continuity with the EBP intentions measure, the same definition of EBP was provided. A broadband item was selected to complement the EBP intentions measure and capture clinicians’ EBP use with their entire range of clients (Fishbein et al. 2001). The point biserial correlation between EBP adoption and EBP use was r = .36 (p < .001) suggesting these measures captured related but distinct facets of clinicians’ EBP adoption behavior.

Control Variables

In order to optimize statistical power and adjust for differences in baseline organizational social context between the treatment and control agencies, baseline OSC profile scores were included as covariates in all analyses. OSC profile scores represent a configural approach to characterizing organizational social context (Schulte et al. 2009) that incorporates all six dimensions of culture and climate assessed by the OSC (i.e., proficiency, rigidity, resistance, engagement, functionality, stress). The predictive validity of OSC profile scores was confirmed in two separate samples of children’s mental health agencies on three separate outcome indicators including work attitudes, service quality, and clinical outcomes (Glisson et al. 2013, 2014; Olin et al. 2014). OSC profile scores range from 1.00 to 3.00 with higher values indicating more positive organizational social contexts that are associated with more positive clinician attitudes toward EBPs, higher service quality, and better client outcomes. In the present study, agencies’ OSC profile scores at baseline ranged from 1.00 to 3.00 with a mean of 1.70 (SD = .72).

Prior research has produced mixed results regarding the association between clinician characteristics and EBP adoption (Aarons et al. 2012). In the present study, clinicians’ characteristics were unrelated to any of the mediators or EBP outcomes with the exception of clinicians’ years of experience in mental health settings (r = .16 with EBP adoption, p = .035). Inclusion of this variable as a covariate did not change the results of the analyses and as a result the more parsimonious models are presented.

Data Analysis

Given the hierarchical structure of the data (i.e., clinicians nested within agencies), analyses incorporated two-level mixed-effects regression models with random agency intercepts, also known as hierarchical linear models (Raudenbush and Bryk 2002). Analyses were implemented using the TWOLEVEL procedure in Mplus software (Version 7). Models with a dichotomous outcome (i.e., EBP adoption) incorporated a logit link function (i.e., two-level mixed effects logistic regression). Cases with missing values were excluded listwise for each analysis based on the missing at random assumption. This approach maximized statistical power and used all available information. Results of Little’s missing completely at random (MCAR) test supported the MCAR assumption for the study variables, χ2 = 24.08, df = 27, p = .626, suggesting that missing responses were not systematic and therefore were unlikely to bias the study results. Preliminary analyses confirmed the presence of significant between-agency variance on each of the clinician-level criterion variables (i.e., EBP adoption and use) and clinician-level mediator variables (i.e., EBP intentions, job-related EBP barriers) with ICC(1)’s ranging from .10 to .30.

Tests of mediation were conducted using the product of coefficients approach for multilevel mediation analysis described by Zhang et al. (2009). Under this approach, simultaneous equations are fit to the data which parse the total effect of the independent variable on the dependent variable into direct and indirect effects based on the hypothesized mediation model. Given that all of the indirect effects in this study crossed from the organizational level to the clinician level (e.g., ARC → proficiency culture → EBP intentions → EBP adoption), the analyses incorporated the centered within context with means reintroduced (CWCM) procedures described by Zhang et al. (2009) in order to obtain unbiased estimates of the mediated effects and correct statistical tests in the lower-level portion of the models. The models test the extent to which ARC’s effects on EBP adoption and use are mediated through two mechanisms: (a) a contextual–motivational mechanism in which ARC creates proficient cultures that increase clinicians’ EBP intentions, and (b) a barrier-reduction mechanism in which ARC reduces job-related EBP barriers.

The statistical significance of the mediated effects was tested using two complementary approaches (Hayes and Scharkow 2013; MacKinnon et al. 2002; MacKinnon et al. 2004). First, the joint significance test provided a null hypothesis significance testing approach (Cohen and Cohen 1983, p. 366; MacKinnon et al. 2002). This approach tests the null hypothesis for each leg of the mediated effect (i.e., H0: a = 0 and b = 0), and rejects H0 in the event that the null hypotheses for both the a and b coefficients are rejected in their respective models. In the case of serial mediation (e.g., ARC → proficiency culture → EBP intentions → EBP adoption), this logic is extended to the a, b, and c coefficients (see Fig. 1). MacKinnon et al. (2002) demonstrated the superiority of the joint significance test over other null hypothesis significance tests for simple mediation models and Taylor and colleagues (2008) extended this finding to three-path serial mediation models such as those tested here.

Second, asymmetric 95 % confidence limits were developed for each mediated effect based on computationally intensive Monte Carlo methods using 100,000 replications (Hayes and Scharkow 2013; Preacher and Selig 2012). Under this approach, the mediated effect is statistically significant if the 95 % confidence intervals do not span zero. Monte Carlo methods have been recommended by methodologists because they provide more accurate confidence limits and exhibit higher statistical power (while maintaining Type I error rates) than normal theory approaches such as the Sobel test (Hayes and Scharkow 2013; MacKinnon et al. 2004). Monte Carlo CIs were constructed using an R utility developed by Preacher and Selig (2012).

Results

Table 1 presents descriptive statistics and bivariate correlations for the agency- and clinician-level variables in the study. The small and non-significant correlation between ARC and baseline OSC profile (r = .03, p = .912) suggests the randomization procedure was successful in balancing the treatment and control groups on organizational social context at baseline. The pattern of correlations among the study variables provides discriminant and convergent validity evidence and suggests the indicators are not simply capturing common method variance. For example, the absolute value of the correlations ranged from .00 to .69 and variables such as EBP use at Time 4 related to other constructs such as OSC at Time 1 (r = .00), Proficiency culture at Time 2 (r = .25) and EBP intentions at Time 3 (r = .60) in differential and theoretically expected ways. Significant variation was observed on all of the clinician- and agency-level variables indicating they captured meaningful variation across clinicians and sites. For example, clinicians’ EBP use at 12-month follow-up ranged from 0 to 100 %, with 14 % of the sample reporting they used EBPs with 25 % or less of their clients and agency’s proficiency scores ranged from .76 SDs below the national mean (22nd percentile) to 1.8 SDs above the national mean (96th percentile).

Table 1 Descriptive statistics and correlations for study variables
Full size table

Manipulation Check

Examination of clinicians’ responses to the ARC fidelity measure at the conclusion of the ARC intervention period confirmed (a) clinicians in the ARC condition reported significantly higher fidelity to the ARC principles than clinicians in control agencies (γ = .26, p = .007, d = .48), and (b) agencies in the ARC condition did not differ in their fidelity to the ARC principles (τ = .02, SE = .02, p = .422). The average APQ score in the ARC group was 4.07 out of 5, corresponding to a rating of “almost always” and representing 81 % fidelity as a percentage of the total possible score. These results support the experimental manipulation and indicate the ARC agencies successfully executed the ARC implementation strategy.

ARC Main Effects on EBP Adoption and Use

Hypothesis 1 stated ARC would increase clinicians’ self-reported EBP adoption and use. Results of the two-level mixed effects regression analyses supported this hypothesis (see Table 2). At 12-month follow-up, clinicians in ARC agencies exhibited significantly higher odds of adopting a well-established EBP (γ = 1.16, SE = .39, p = .003, OR = 3.19) representing a 219 % increase in their odds of EBP adoption relative to control. This translates into an increase of 21 % in the probability of EBP adoption for clinicians in ARC agencies.

Similarly, at 12-month follow-up, clinicians in ARC agencies reported significantly greater use of EBPs with clients (γ = 25.13, SE = 8.40, p = .003) representing a standardized mean difference of d = .79 and a 45 % increase in EBP use relative to control. These results support ARC’s main effects on EBP adoption and use.

Table 2 Main effects of ARC on clinician EBP intentions, job-related EBP barriers, EBP adoption and use
Full size table

Contextual–Motivational Mechanism

Hypothesis 2 indicated ARC’s effects on clinicians’ EBP adoption and use would be mediated by improved proficiency culture followed by increased clinician EBP intentions. Three steps were required to test this cross-level, serial mediation hypothesis. First, ARC improved proficiency culture at Time 2 relative to control. As shown in Table 3, results of an ordinary least squares regression analysis indicated ARC increased proficiency culture at Time 2 (B = 6.91, SE = 2.49, p = .020) accounting for 28 % of the variance in proficiency culture.

Table 3 ARC effects on clinician EBP adoption and use through proficiency culture and EBP intentions
Full size table

Second, a mixed effects regression analysis indicated ARC increased clinicians’ EBP intentions at Time 3 (γ = .64, SE = .30, p = .030), representing a standardized mean difference of d = .44 and accounting for 39 % of the between-agency variance (see Table 2). Results from a second mixed effects regression analysis supported mediation by showing that ARC’s effect on EBP intentions was reduced when proficiency culture was added to the model (γ = .06, SE = .63, p = .928) and higher proficiency culture predicted higher clinician EBP intentions (γ = .08, SE = .04, p = .045), controlling for ARC and baseline OSC profile (see Table 3). The joint significance test and the 95 % Monte Carlo CIs (ab = .56, 95 % CI [.01–1.38]) confirmed ARC’s effect on clinicians’ EBP intentions was significantly mediated by improved proficiency culture (see Table 5). The ratio of ARC’s indirect effect to its total effect was P M  = .88, suggesting improved proficiency culture accounted for 88 % of ARC’s total effect on clinicians’ EBP intentions (Preacher and Kelley 2011).

Third, ARC’s effects on EBP adoption and use were serially transmitted through improved proficiency culture and subsequently increased EBP intentions (see Fig. 1). ARC’s effect on EBP adoption at Time 4 was reduced once proficiency culture and EBP intentions were added to the model (γ = .92, SE = .61, p = .127) and increased clinician EBP intentions was the only significant predictor of EBP adoption (γ = 1.75, SE = .60, p = .003) after controlling for ARC, proficiency culture, and baseline OSC profile (see Table 3). As is shown in Table 5, results of the joint significance test of the a, b, and c coefficients for the hypothesized serial mediation effect (see Fig. 1) as well as the 95 % Monte Carlo CIs supported the hypothesized model (a × b × c = 1.12, 95 % CI [.17–2.74]). Results from the analysis indicate ARC exerted a significant indirect effect through the serial influence of improved proficiency culture and increased EBP intentions accounting for 96 % of ARC’s total effect on clinicians’ EBP adoption (i.e., P M  = .96).

Results of the serial mediation analysis for clinicians’ EBP use mirrored the results for EBP adoption. ARC’s effects on EBP use were decreased once proficiency culture and EBP intentions were added to the model (γ = 13.90, SE = 8.73, p = .111) and increased clinician EBP intentions was the only significant predictor of EBP use (γ = 24.32, SE = 5.36, p < .001) after controlling for ARC, proficiency culture, and baseline OSC profile (see Table 3). Results of the joint significance test of the a, b, and d coefficients for this serial mediation effect (see Fig. 1), as well as the 95 % Monte Carlo CIs (abd = 15.36, 95 % CI [3.71–33.31]), indicated ARC’s effect on EBP use was serially mediated by improved proficiency culture and increased EBP intentions (see Table 5). This two-step, contextual–motivational mechanism accounted for 61 % of ARC’s total effect on EBP use.

Barrier Reduction Mechanism

Hypothesis 3 tested an alternative cross-level mechanism, namely that reduced job-related EBP barriers would mediate ARC’s effects on clinicians’ EBP adoption and use. This hypothesis required two tests. First, ARC reduced job-related EBP barriers (see Table 2). Clinicians in ARC agencies reported significantly fewer job-related EBP barriers at 12-month follow-up relative to control agencies (γ = −.70, SE = .31, p = .026) representing a standardized mean difference of d = −.59 and accounting for 52 % of the between-agency variance.

Second, decreased job-related EBP barriers predicted EBP adoption and use controlling for ARC and baseline OSC profile (see Table 4). ARC’s effect on clinicians’ EBP adoption was diminished when job-related EBP barriers was added to the model (γ = .63, SE = .36, p = .075) and decreased job-related EBP barriers predicted increased EBP adoption (γ = −.90, SE = .30, p = .003), controlling for ARC and OSC profile. The joint significance test and the Monte Carlo 95 % CIs for this mediated effect confirmed ARC’s effect was mediated by reduced job-related EBP barriers (ij = .69, 95 % CI [.06–1.60]), accounting for 59 % of ARC’s total effect on clinicians’ EBP adoption (Table 5).

Table 4 ARC effects on clinician EBP adoption and use through job-related EBP barriers
Full size table
Table 5 Indirect effects of ARC on clinicians’ EBP adoption and use
Full size table

Results from the analysis of EBP use were substantively similar to the results for EBP adoption (see Table 5). ARC’s effect on EBP use was diminished once job-related EBP barriers was added to the model (γ = 12.54, SE = 6.95, p = .071) and decreased job-related EBP barriers predicted increased EBP use (γ = −17.46, SE = 5.35, p = .001) in the same model (see Table 4). Results from the joint significance test and the Monte Carlo 95 % CIs (ik = 12.47, 95 % CI [1.52–27.67]) confirmed ARC’s effect was mediated by reduced job-related EBP barriers accounting for 49 % of ARC’s total effect on clinicians’ EBP use.

Unique Contributions of the Contextual-Motivational and Barrier Reduction Mechanisms

The analyses reported above indicate ARC’s effects on clinicians’ EBP adoption and use were explained in part by improved proficiency culture which increased clinicians’ EBP intentions and by reduced job-related EBP barriers. Hypothesis 4 tested these two mechanisms simultaneously to assess the unique variance explained by each. This hypothesis was tested by estimating two-level mixed effects regression models in which proficiency culture, EBP intentions, and job-related EBP barriers simultaneously predicted clinicians’ EBP adoption (or use), controlling for ARC and OSC profile. Results from these analyses were substantively identical. In both analyses, EBP intention was the only significant predictor of EBP adoption (γ = 1.68, SE = .79, p = .032) and EBP use (γ = 21.91, SE = 7.10, p = .002) once all covariates were included in the model. These findings indicate clinicians’ increased EBP intentions rather than reduced job-related EBP barriers played the larger unique role in explaining ARC’s effects on clinicians’ EBP adoption and use.

Discussion

Results from this 4 year randomized controlled trial suggest mental health service systems can increase clinicians’ EBP adoption and use by creating proficient organizational cultures that prioritize client well-being and expect clinician competence in up-to-date treatment practices and by reducing job-related EBP barriers. To our knowledge, this is the first experimental study to support the mediational role of planned organizational culture change in shaping mental health clinicians’ practice behaviors (Parmelli et al. 2011). These results are important given that changes in clinicians’ self-reported behaviors were confirmed 12 months after the ARC intervention was concluded and were explained by sequentially measured effects on organizational proficiency culture and clinicians’ EBP intentions.

Results from this study support organizational culture theory in showing that purposeful change of an agency’s culture can contribute to targeted changes in mental health clinicians’ practice behaviors (Schein 2004). A growing body of intervention studies in a variety of settings have demonstrated the effects of context-focused organizational interventions on organizational social contexts (Glisson et al. 2012b), employees’ work behaviors (Larson et al. 2000; Zohar and Polachek 2014), and clinical outcomes (Glisson et al. 2010, 2013). The present study extends this research by demonstrating that the effects of the ARC organizational strategy on EBP adoption and use are mediated by improvements in organizational culture.

Findings from this study show that organizational proficiency culture and clinicians’ EBP intentions are sequential components of an effective mechanism for increasing clinicians’ EBP adoption. Moreover, results suggest these components may be more important determinants of clinicians’ EBP adoption behavior than job-related EBP barriers. Although prior correlational and qualitative studies have associated these variables with clinicians’ EBP adoption (Chaudoir et al. 2013), this is the first study to experimentally test the variance explained by each of these hypothesized mechanisms. The results support a social contextual perspective on EBP adoption in which prevailing cultural norms and subsequent socially-influenced intentions play a larger role than job-related barriers in shaping clinicians’ innovation behavior. This finding is consistent with diffusion of innovations theory (Rogers 2003) which has argued for many decades that social processes are more important to innovation adoption and implementation than the technical requirements of the innovations themselves. The implication is that more attention should be devoted to developing efficient strategies for creating organizational social contexts in community-based mental health services that support targeted innovations such as EBPs. The role of social context in these findings provide a counter-balance and complement to the predominant focus on simply increasing technical competence in implementation studies, suggesting that both the technical and social aspects of implementation must be addressed to improve implementation outcomes (Novins et al. 2013).

The range of outcomes that occur during a phased implementation process provides an important lens for interpreting these results (Proctor et al. 2010). For example, implementation models such as the Exploration, Preparation/Adoption, Implementation, and Sustainment model proposed by Aarons et al. (2011) suggest adoption occurs relatively early in the implementation process following an exploration phase and preceding active implementation and sustainment phases. Our findings indicate organizational proficiency culture and clinicians’ EBP intentions are more important than job-related barriers for increasing EBP adoption. However, different mechanisms could be more important for achieving other implementation outcomes in other phases. Although there is evidence from other studies that the advantages of a proficient organizational culture include positive effects on other implementation outcomes (e.g., Aarons et al. 2012), further research is needed to test these hypotheses and link the phases empirically.

An important challenge for future studies is addressing the different ways in which implementation strategies focus on organizational social context in shaping implementation outcomes. Whereas the present study addresses proficiency culture, other models of EBP implementation focus on different contextual constructs (Aarons et al. 2014; Ehrhart et al. 2014). For example, the EBP implementation climate construct focuses more narrowly on the implementation of EBPs (Ehrhart et al. 2014). Important questions for the field include (a) the extent to which proficiency culture and other contextual constructs such as EBP implementation climate have differential effects on the range of implementation outcomes (e.g., acceptability, adoption, fidelity, sustainment) with different EBPs (e.g., modular cognitive behavioral therapy, measurement-based care), and (b) which dimensions of organizational social context should be targeted to optimize ongoing EBP exploration, adoption, implementation, and sustainment as new EBPs are developed? The norms and expectations that characterize a proficient organizational culture may have advantages over other constructs in terms of generalizing positive effects across EBPs, time, and the range of implementation outcomes representing each phase.

Limitations

Four caveats must be considered in interpreting these results. First, agencies were randomly assigned to ARC and control conditions after matching on size so that clinicians were cluster randomized by the agencies that employed them. Cluster randomization is a well-accepted design strategy, particularly in implementation trials where randomization of organizations often results in relatively small intervention sample sizes. However, unmeasured clinician variables could have influenced the results. This concern is mitigated by the finding that there were no differences between the two conditions in clinician education, years of experience, gender, age, race, or turnover during the study period. Moreover, there was no significant difference in organizational social context between ARC and control agencies at baseline but differences emerged over the course of the study in organizational culture, clinicians’ EBP intentions, job-related EBP barriers, and EBP adoption and use. In addition, the effect of ARC on EBP adoption and use was fully mediated by the sequential improvement in proficiency culture and increased EBP intentions during the study period. Collectively, these findings suggest the threat of unmeasured variables confounding the results is minimized.

A second caveat is clinicians’ self-reported EBP adoption and use may not fully capture outcomes associated with each of the phases of EBP exploration, preparation, adoption, implementation, and sustainment (Aarons et al. 2011). Implementation research focuses on a variety of implementation outcomes and the variables incorporated into this study readily map onto the implementation outcome of adoption as described by Proctor and colleagues (Proctor et al. 2010). EBP adoption and use with clients represent necessary but not sufficient conditions for effective integration of EBPs into routine care. Clinicians are uniquely aware of the practice behaviors they attempt to use in sessions with clients and their reports of efforts to use EBPs represent but one important dimension in the effective integration of EBPs into community-based mental health services.

A third caveat concerns the reliability of the self-report measures of EBP adoption and use given evidence that observational and self-report measures of clinicians’ EBP adoption do not always correspond (Hogue et al. 2014; Hurlburt et al. 2010). There were several findings that reduce but do not eliminate these concerns. First, the wide range of values observed on the EBP outcome measures and the high percentage of clinicians’ who reported low EBP use increases confidence in the reliability of the findings by reducing concerns about social desirability biases. Also, the fact that ARC’s effects were observed across multiple indicators over 4 years provides convergent evidence that supports the reliability of the observed changes and indicates they were sustained over time. The 12 month intervals between measurement occasions reduces concerns about common method error variance because the long time interval reduced the salience of clinicians’ earlier responses and impeded the retrieval of prior responses from memory (Podsakoff et al. 2003). Finally, the statistically and substantively large effects of ARC on the EBP outcomes (e.g., 50 % use vs. 81 % use) suggests ARC’s effects were practically meaningful.

A fourth caveat pertains to additional mediating variables that may help explain ARC’s effects on clinicians’ EBP adoption and use. Although this study focused on contextual and motivational mechanisms because of their theoretical and empirical support, other intervening variables could be introduced into the model to help explain ARC’s effects. For example, increased leader receptivity to providing agency-sponsored EBP training may mediate the relationship between proficient organizational culture and increased clinician EBP adoption. Elaborating these linkages is an important area for further research.

Conclusion

Findings from this randomized controlled trial provide evidence that the ARC organizational intervention can increase clinicians’ EBP adoption and use through the transformation of organizational culture and the activation of clinicians’ EBP intentions. These findings highlight directions for future research on the role of organizational social context in shaping a variety of implementation outcomes in mental health services. Results also suggest organizational social context and clinicians’ behavioral intentions represent salient targets for increasing EBP adoption and effectiveness in mental health systems. Additional studies that extend our knowledge of how to efficiently marshal these and other mechanisms are essential to advancing implementation science and practice in the complex arena of mental health services.