Introduction

In recent years, there have been numerous large-scale efforts to implement evidence-based practices (EBPs) in public mental health service systems with the aim of bolstering the quality of care and client outcomes (Hoagwood et al. 2014; McHugh and Barlow 2010; Nakamura et al. 2014; Starin et al. 2014; Trupin and Kerns 2015). Several frameworks have been developed to describe the multiple levels of influence (system, community, organization, individual) on the outcomes of EBP implementation efforts (Tabak et al. 2012). This study is framed in the Exploration Preparation Implementation Sustainment (EPIS) framework developed by Aarons et al. (2011) that delineates outer (i.e., system) and inner (i.e., organizations) context variables key to EBP implementation in public mental health service settings. The outer context encompasses factors that impact the capacity of a service system or organizations within the system to implement and sustain EBPs such as regulations and procedures, funding sources, and legislation. The inner context encompasses the individual organizations within a system and levels of influence and action within the organizations (e.g., top management, workgroups, direct service providers). Within this inner context, there may be important variability in the perceptions of EBP implementation as a function of individuals’ roles and related perspectives. The EPIS framework was selected because it emphasizes the multi-level context of EBP implementation, identifies key domains and constructs within outer and inner contexts, identifies outer and inner context domains and constructs within and across relatively distinct phases of the implementation process, and it was developed with a specific focus on EBP implementation and sustainment in publicly-funded settings serving children and families, which is the services context of this study. Agency leaders and front-line mental health therapists occupy distinctive roles and carry unique responsibilities in community implementation of multiple EBPs. Their experiences and perceptions of implementation may be shaped by different organizational vantage points and pressures (Beidas et al. 2016b; Gibson et al. 2009).

Accumulating research highlights the importance of perceptions of EBPs as an inherent driver of key implementation outcomes such as adoption (Borntrager et al. 2009; Proctor et al. 2011). Direct service provider perceptions of EBPs are linked to uptake and delivery of EBPs (Aarons et al. 2010; Aarons and Palinkas 2007; Henggeler et al. 2008). Research suggests that provider attitudes toward EBPs in general differ from their perceptions towards specific EBPs (Gray et al. 2007; Reding et al. 2014). These findings indicate that EBPs may not be uniformly evaluated by therapists and this variability may systematically predict rates of use of EBPs. Thus, attitudes towards EBPs have been considered a target mechanism to facilitate adoption (Borntrager et al. 2009). Greater understanding of EBP-specific perceptions has particular relevance for implementation contexts involving the adoption of multiple EBPs, as providers readily differentiate among EBPs in terms of perceived appeal and limitations (Reding et al. 2014). Barnett et al. (2017) found that both therapist background characteristics and characteristics of interventions predict EBP-specific perceptions, such that therapists prefer practices with more structure and ongoing consultation.

The preponderance of the implementation literature on perceptions of EBPs has focused on direct service providers (i.e., therapists), as the end-users of EBPs, and less on other agents involved in community implementation. It is well established that leadership is a key component of successful adoption, implementation, and sustainment of EBPs (e.g., Aarons et al. 2016, 2011). Leadership can act as a critical driver of EBP implementation through fostering an organizational environment receptive to EBP use through policies and procedures that prioritize EBP training and fidelity monitoring. In addition, leaders are instrumental in facilitating adoption and institutionalizing EBPs in the service system by serving as EBP champions, particularly when interfacing with system-level decision-makers or stakeholders, and allocating resources strategically to ensure continuity of implementation (Aarons et al. 2016). Leaders can have a profound influence on the organizational climate (i.e., perception of the psychological impact of the work setting) and culture (i.e., workplace norms and priorities) that shapes the perceptions and performance of front-line staff (Glisson et al. 2006; Peterson et al. 2014) as well as strategic climates for a particular organizational imperative (Aarons et al. 2017). Such organizational characteristics are, in turn, associated with therapist perceptions of EBPs (Aarons et al. 2012; Aarons and Sawitzky 2006; Brimhall et al. 2015; Williams et al. 2017), therapist implementation outcomes (Beidas et al. 2016a), service quality (Olin et al. 2014), client outcomes (Glisson and Green 2011; Williams and Glisson 2014), and the sustainability of new practices or procedures (Glisson et al. 2008).

The managerial and organizational research literature has reported that discrepancies (also framed as “perceptual distance” or incongruence) between self- and other-ratings of leader behavior (Ostroff et al. 2004) and organizational culture are common (Martin et al. 2006; Zyphur et al. 2016), and also found in health care settings (Hasson et al. 2016) and leader self-ratings tend to be more positive and uniform than staff ratings (Thornton 1980; Wolf et al. 2014). Within mental health services research, there is emerging literature similarly documenting discrepancies between leader and provider perceptions and these discrepancies are associated with organizational climate and culture (Aarons et al. 2017; Beidas et al. 2016b). Specifically, these studies reported that leaders, compared to providers, reported more positive organizational climate and culture (Beidas et al. 2016b) and the magnitude of discrepancy between leader and provider ratings was associated with more negative organizational culture (Aarons et al. 2017). For implementation leadership, discrepancies characterized by lower leader self-ratings and higher follower ratings of that leader—commonly referred to as “humble leadership”—are associated with more positive climate for performance feedback and program involvement, both important aspects of a context for EBP implementation and sustainment (Aarons et al. 2017). This research showcases not only a lack of concordance between leader and provider perceptions but highlights the importance of organizational functioning characteristics, which can be important determinants of EBP implementation in mental health services (e.g., Aarons and Sawitzky 2006; Glisson 2002; Rogers 1962). A natural extension of this work would be to examine the differences between leader and provider perceptions of specific EBP innovations and gain and understanding of how organizational climate may contribute to these distinct viewpoints.

Thus, it is important to consider the perceptions of both leaders who may be instrumental in selecting EBPs for adoption and in setting conditions for their implementation as well as those of providers. It is certainly plausible that perceptions towards EBPs may differ based on the stakeholder role in relation to both the mental health organization and within a given EBP implementation initiative. In particular, leaders and therapists may have different service-related priorities and vantage points on EBP implementation that impact their perceptions of EBPs. This notion of “Innovation-values fit” (Klein and Sorra 1996) involves a strong match between the adopted EBP and the mission, values, and job tasks of the employees and organization, and is thought to facilitate successful EBP implementation. This concept has been extended in the EPIS framework to consider EBP fit at system, organization, and provider levels. Yet, perceptions of “fit” may vary across different levels and agents (Aarons et al. 2011). The unique priorities and values of frontline providers versus leaders may similarly or dissimilarly impact important implementation outcomes (Proctor et al. 2011) based on the staff member’s proximity to the outcome. For example, leaders must consider organizational investments and returns in EBP implementation (e.g., resources needed for training and fit with agency service mission and structure), while therapists are likely most concerned with their own end-user experiences with delivering the EBP to clients (e.g., ease of use and clinical utility; Aarons et al. 2011; Bonham et al. 2014). Further, key implementation considerations for leaders may include funding and costs of EBP training, staffing resources and competing demands such as contractual requirements (Aarons et al. 2009; Proctor et al. 2007; Willenbring et al. 2004). Examining potential differences and similarities in perceptions of specific EBPs across leaders and front-line therapists is an important next step in understanding the social context of community EBP implementation initiatives.

The current study addressed several key gaps in the literature. First, research examining perceptions towards EBPs within mental health services has largely focused on direct service providers. Yet, providers often have a limited role in the selection of EBPs and ongoing oversight of EBP implementation, highlighting the importance of gathering perspectives of both leaders and providers because of their unique contributions to EBP implementation efforts (Hasson et al. 2016). Further, although research supports the association between organizational context and general perceptions about EBPs, there is need to examine these associations with perceptions towards specific EBPs. Perceptions may vary not only as a function of an individual’s role in EBP implementation but also because of the unique practice about which is reported. Finally, there is a need to understand how perceived elements of organizational functioning influence the perceptions of leaders and front-line providers.

Current Study

We examined leader and therapist perceptions towards EBPs within the context of a system-level reform mandating the implementation of multiple EBPs through reimbursement policies in children’s mental health services. The study context is the Los Angeles County Department of Mental Health (LACDMH) Prevention and Early Intervention (PEI) Transformation of Children’s Mental Health Services. The LACDMH is the largest public sector mental health system in the United States. The PEI Transformation initiative was borne out of a reorganization of mental health services reimbursement within publicly-funded mental health agencies operating in LACDMH in which agencies were reimbursed for the delivery of specific EBPs. The current study focuses on six evidence-based practices [Cognitive Behavioral Interventions for Trauma in Schools (CBITS; Jaycox 2003), Child-Parent Psychotherapy (CPP; Lieberman and Van Horn 2005), Managing and Adapting Practice (MAP; Chorpita et al. 2014), Seeking Safety (SS; Najavits 2002), Trauma Focused Cognitive Behavior Therapy (TF-CBT; Cohen et al. 2006), Triple P Positive Parenting Program (Triple P; Turner et al. 2002)] that received implementation support via training and consultation coordinated by LACDMH in the initial PEI rollout. These practices vary in the primary mental health target and training requirements but were selected to address a range of common child mental health problems.

Capitalizing on the system-driven multiple EBP implementation context of the LACDMH PEI Transformation, the current study has two primary aims. The first aim was to describe and compare the perceptions of specific practices between leaders and therapists within a unique multiple EBP implementation context. Given the research indicating that therapists vary in their attitudes towards specific EBPs and implementation theory suggesting that attitudes towards EBPs may differ as a function of staff role, we hypothesized that leaders and therapists would differ in their preferences for specific practices, particularly within this multiple EBP implementation setting. The second aim of this study was to identify leader and therapist characteristics (sociodemographic and professional background), organizational context factors (climate, culture), and therapist emotional exhaustion associated with perceptions of EBPs within a multiple EBP implementation context within which leaders had oversight over the implementation and sustainment of multiple EBPs that therapists had been trained to deliver concurrently. This was an exploratory aim given the limited research that has examined the perspectives of both leaders and therapists and the role of organizational context.

Method

Participants and Procedures

Participants included leaders and front-line therapists in community mental health clinics contracted to deliver at least one of the six practices described above within the LACDMH PEI initiative. The research team requested contact information for all clinical staff from the management at eligible agencies. For this study, leaders were defined as employees who provided administrative or clinical oversight for at least one of the six EBPs being implemented at their mental health agency. Therapists were defined as employees who provided direct clinical services using at least one of the EBPs with children and families at their agency.

Based on claims data, eligible LACDMH agencies were selected if at least one provider billed for at least one of the six original practices during fiscal year 2013–2014 or quarter one of fiscal year 2014–2015. A total of 98 agencies were identified. The contact information for all clinical staff (leaders and therapists) was obtained from agency leadership or through a staff opt-in process at each eligible agency. Contact information for staff from 69 agencies (70.4%) was obtained for recruitment into the survey. Of those 69 agencies, 62 provided email contacts for staff and seven agencies elected to forward an email to staff that would allow them to provide their contact information to the research team to opt-in to receive the survey.

Two versions of the survey were developed: (1) leader survey for staff who provide administrative oversight of the agencies and (2) therapist survey for direct service providers. Leaders who also provide direct clinical service were asked to complete both surveys; but only their responses from one of the survey versions (their leader survey response was prioritized) were included in this study. Respondents received a $20 online gift card for survey completion. A total of 162 leaders and 777 therapists completed the “Knowledge Exchange on Evidence-Based Practice Sustainment” (4KEEPS) Online Survey between March 2015 and July 2015 for response rates of 60.7 and 41.5% for the direct email campaign, for leaders and therapists respectively. Our response rates are similar to those reported in similar types of studies in similar community mental health service settings, which have ranged from 25 to 51% (e.g., Hawley et al. 2009; Cashel 2002; Piotrowski and Keller 1989; Rosenberg and Beck 1986). Because some participants enacted both roles, a total of 160 leaders from 58 agencies and 720 therapists from 63 agencies were included in this study. The average number of participating leaders from each agency was 2.76 (SD = 2.54; range = 1–11; median = 2) and the average number of participating therapists from each agency was 11.43 (SD = 13.74; range = 1–74; median = 6). See Table 1 for demographic and employment details of participants.

Table 1 Leader and therapist characteristics
Full size table

Measures

Perceived Characteristics of Intervention Scale (PCIS; Cook et al. 2015)

The PCIS was designed to assess perceived characteristics of innovation (Greenhalgh et al. 2004; Rogers 1962) that may influence healthcare provider uptake of a particular EBP. The original measure includes 20 items that capture aspects of innovations. In the current study, we administered eight items related to the following: Relative Advantage, Compatibility, Complexity, and Potential for Reinvention. Respondents are asked to rate the extent to which they agreed with each item on a 5-point Likert scale from (1) not at all to (5) to a great extent. Example of PCIS items are: “[Specific practice] is more effective than other therapies I have used” and “[Specific practice] can be adapted to fit my treatment setting.” Both leaders and therapists completed the PCIS although the criterion for PCIS completion was slightly different. Therapists were asked to complete the PCIS about each practice that they “had ever used with a client”. Therapists reported on an average of 2.56 practices (SD = 1.08). Leaders were asked to complete the PCIS about each practice that “had ever been used by therapists in their agency”. Leaders reported on an average of 4.03 practices (SD = 1.16). In summary, a leader or therapist respondent completed a PCIS about each practice that either they had used with a client (therapist report) or had been used by therapists in their agency (leader report). For example, a therapist who reported that they had ever used MAP and Triple P would be asked to complete one PCIS about MAP and a separate PCIS about Triple P. A mean of the eight items was used as a total composite score (possible range: 1–5) for each completed PCIS with higher scores representing more favorable perceptions towards a practice. In the example where a therapist completed a PCIS about MAP and a separate PCIS about Triple P, this therapist would receive a mean composite score for their PCIS about MAP and another mean composite score for their PCIS about Triple P. This scoring approach was applied because psychometric properties reported from the original PCIS measure indicated a unidimensional factor structure (Cook et al. 2015). The total 8-item scale demonstrated strong internal consistency across practices for both leaders (mean α = 0.92; range = 0.89–0.97) and therapists (mean α = 0.94; range = 0.92–0.96). This was the outcome measure of interest in the study.

Leader and Therapist Characteristics

Items related to agency leader and therapist background characteristics were adapted from survey content used in an effectiveness and implementation trial of a treatment for children with autism spectrum disorder (Brookman-Frazee et al. 2010). Leaders and therapists responded to questions about the following: age, gender, race/ethnicity, licensure status, highest degree obtained, direct service hours (number of hours per week for therapists; binary endorsement of any versus no direct service hours per week for leaders), frequency of interaction with therapists (leaders only; ranged from rarely/never to daily), caseload (therapists only), years at their current agency, and the number of practices (out of the six of interest in this study) in which therapist respondents had been trained to deliver (even if they had not used the practice with a client).

Organizational Climate Measure (OCM; Patterson et al. 2005)

The OCM measures employee perceptions of their organization’s policies, practices, and procedures. The full OCM consists of 17 subscales and 82 items and has demonstrated acceptable reliability and validity properties (Patterson et al. 2005). Leaders and therapists both completed the OCM Autonomy and Performance Feedback subscales; therapists additionally completed the Involvement subscale because it was more conceptually relevant to therapists than leaders. The Autonomy subscale assesses perceptions of independence in job decision-making and performance (5 items, e.g., “People at the top tightly control the work of those below them.”—reverse scored), the Involvement subscale assesses perceptions that employees are involved in their organization and can influence organizational decision-making (6 items, e.g., “Program managers and/or agency leaders involve people when decisions are made that affect them.”), and the Performance Feedback subscale assesses perceptions that clear feedback on job performance is provided (5 items, e.g., “The way people do their job is rarely assessed” – reverse scored). Each item is rated on a four-point Likert scale ranging from (1) definitely false to (4) definitely true. The mean of each subscale was calculated, each with a possible range of 1–4. In our leader sample, internal consistency was commensurate with the therapist sample: α = 0.78 (Autonomy), and α = 0.83 (Performance Feedback). In our therapist sample the subscales retained good to adequate internal consistency in this study: α = 0.85 (Involvement), α = 0.72 (Autonomy), and α = 0.80 (Performance Feedback).

Therapist Emotional Exhaustion

Therapist perceptions of workplace fatigue and stress in were assessed using 5 items from the Emotional Exhaustion subscale of the Organizational Social Context Measure (OSC; Glisson et al. 2012, 2008). Example items included, “I feel fatigued when I get up in the morning and have to face another day on the job.” The OSC has established psychometric properties, particularly among healthcare professionals (Glisson et al. 2012, 2008). Participants were asked to rate their agreement with each item on a 7-point Likert scale from (0) strongly disagree to (6) strongly agree. A mean composite score (possible range: 0–6) was calculated with higher scores representing stronger feelings of emotional exhaustion. In our therapist sample, the measure had strong internal consistency, α = 0.89.

Organizational Readiness for Change (Lehman et al. 2002)

The Organizational Readiness for Change scale (ORC) was used to assess leader perceptions of their organization’s functioning related to the implementation of EBPs. In this study, 16 items from three subscales (Staffing, Cohesion, and Stress) of the original ORC were used to efficiently measure organizational functioning from a management perspective. The Staffing subscale assesses perceptions of workforce capacity and quality (6 items, e.g., “Frequent staff turnover is a problem for your program.”); the Cohesion subscale assesses perceptions of staff trust and cooperation (6 items, e.g., “Mutual trust and cooperation among staff in your program are strong.”); the Stress subscale assesses perceptions of staff strain, stress, and workplace burden (4 items, e.g., “The heavy staff workload reduces the effectiveness of your program.”) Strong support for the psychometric functioning of the ORC has been documented (Lehman et al. 2002). Leaders were asked to rate each item on a 5-point Likert scale ranging from (1) disagree strongly to (5) agree strongly. A mean score was calculated for each subscale each with a possible range from 1 to 5. In our leader sample, subscales demonstrated adequate internal consistency: α = 0.66 (Staffing), α = 0.76 (Cohesion), and α = 0.85 (Stress).

Analytic Plan

Three-level mixed effects modeling with random intercepts were specified using SPSS v. 20 MIXED commands. This approach was used to account for the nested (i.e., non-independent) nature of the data because leader and therapists (level-2) reported their perceptions of up to six practices (level-1) and multiple leaders and therapists were employed at the same agency (level-3). These models were specified as follows: level-1 (practice) variables were entered as five effect-coded variables (i.e., k-1 with k = 6 practices) to allow for practice-specific perceptions to be compared to the grand mean of the PCIS across practices. Level-2 (respondent) variables were leader and therapist sociodemographic and professional characteristics and individual perceptions of emotional exhaustion (therapists only). Level-3 was the primary agency in which respondents were employed. Perceptions of organizational climate (leaders and therapists) were aggregated to the level-3 (agency-level). In all models, the outcome was the PCIS mean composite score for each practice (up to six) being implemented in their agency (leaders) or in which they had received training (therapists). This modeling approach is similar to implementation research studies that have examined differences in ratings between leaders and therapists within mental health settings (e.g., Beidas et al. 2016b; Reding et al. 2014). Further, this modeling approach, specifically related to the practice-level variables, best represents how the PCIS data were collected, the multiple EBP implementation context of this study, and allows for the variance to be modeled across practices in a parsimonious manner.

Results

Aim 1: Describe and Compare the Perceptions of Specific Practices Between Leaders and Therapists Within a Unique Multiple EBP Implementation Context

Three-level nested models with cross-level interactions between practice and respondent type were specified to identify differences in staff type (therapist versus leader) preferences. The omnibus test of fixed effects indicated that there were significant main effects for all practices, no significant main effect for respondent type (EMM for therapists = 3.41; SE = 0.03; EMM for leaders = 3.47; SE = 0.05), F(1, 1050.19) = 0.92, p = .34 and significant cross-level interactions between respondent type (level 2) and practice for MAP, F(1, 1867.61) = 39.28, p < .001, CPP, F(1, 2046.46) = 4.4,7 p < .05, Triple P, F(1, 1962.15) = 15.83, p < .001, and SS, F(1, 1924.52) = 4.12, p < .05. Specifically, compared to the grand PCIS mean (M = 3.26, SE = 0.06), therapists reported significantly more favorable perceptions towards CPP and Triple P compared to leaders. In contrast, leaders reported more favorable perceptions towards MAP and SS compared to therapists. Figure 1 displays the estimated marginal means (EMM) of the PCIS for leaders and therapists. See Table 2 for full model details and Fig. 2 for a visual display of the model estimates for the cross-level interactions between respondent type and practice.

Fig. 1
figure 1

Estimated marginal means of the PCIS for leaders and therapists from unconditional model. PCIS mean scores ranged from 1 to 5 with higher scores indicating more favorable attitudes

Full size image
Table 2 Differences in leader and therapist perceptions by practice
Full size table
Fig. 2
figure 2

Beta coefficients of respondent type by practice interaction estimates. Estimates are relative to the grand mean of the PCIS. Leaders reported more favorable perceptions to MAP and SS compared to therapists. Therapists reported more favorable perceptions towards Triple P and CPP compared to leaders. *p < .05; **p < .001

Full size image

Aim 2. Examine the Associations Between Leader and Therapist Characteristics (Sociodemographic and Professional Background) and Perceptions of Their Organization’s Context on Their Practice-Specific Perceptions

A separate three-level model was specified for leaders and therapists. Practice (level 1) was again entered as five effect-coded variables. Predictors in the leader and therapist models included sociodemographic characteristics (e.g., gender, race/ethnicity), professional characteristics (e.g., licensure status, length of tenure at their agency), and therapist emotional exhaustion. Perceptions about organizational context as measured by the OCM and the ORC, were aggregated to the agency-level (level 3). See Table 3 for full model details. Findings are described separately by leader and therapist models.

Table 3 Leader and therapist perceptions of organizational context on practice perceptions
Full size table

Leader Characteristics and Perceptions of Organizational Context

Level 1: Practice

Leader PCIS scores were significantly higher for MAP F(1, 501.19) = 46.15, p < .001, b = 0.42, CPP (1, 524.20) = 3.96, p < .05, b = 0.16, and TF-CBT F(1, 503.32) = 63.67, p < .001, b = 0.48 compared to the leader grand mean of the PCIS (EMM = 1.68). There were no significant differences between PCIS scores for CPP, Triple P and SS relative to the leader grand mean. These findings indicate that leaders reported significantly varied preferences only for MAP and TF-CBT compared to their average rating across practices.

Level 2: Leader-Level Predictors

Leader Characteristics: The leader characteristics that significantly predicted more favorable practice-specific perceptions were education, F(1, 135.70) = 5.03, p < .05 and the number of practices adopted by their agency, F(1, 138.85) = 9.14, p < .01. Specifically, leaders who had doctoral degrees had less favorable perceptions compared to leaders with master’s degrees, b = − 0.28, p < .01; and leaders who reported a higher number of practices that were adopted by their agency indicated more favorable perceptions towards practices, b = 0.07, p < .001. Leader race/ethnicity, gender, licensure status, whether they delivered any direct service hours on a weekly basis, and frequency of interaction with line therapists were not significantly associated with leaders’ perceptions towards practices.

Level 3: Agency-Level Predictors

Perceptions of Organizational Context: Perceptions of staffing resources were significantly associated with more favorable practice-specific perceptions for leaders, F(1, 358.89) = 5.16, p < .05. Specifically, more positive views about workforce capacity and quality at the agency-level was associated with more favorable leader perceptions towards practices, b = 0.32, p < .05. Ratings on the OCM (Autonomy and Performance Feedback) subscales and on the ORC (Cohesion and Stress) subscales were not significantly associated with leaders’ perceptions towards practices.

Therapist Characteristics and Perceptions of Organizational Context

Level 1: Practice

Therapist PCIS scores were significantly higher for CPP F(1, 1539.34) = 31.55, p < .001, b = 0.39; Triple P, F(1, 1458.11) = 22.65, p < .001, b = 0.29; and TF-CBT F(1, 1406.25) = 115.70, p < .001, b = 0.44, compared to the therapist grand mean of the PCIS in this model (EMM = 3.63). Therapist PCIS scores for SS were significantly lower than the grand therapist mean, F(1, 1448.92) = 16.12, p < .001, b = − 0.17. Therapist PCIS scores of MAP were not significantly different from the grand therapist mean. These findings indicate that therapists reported significantly varied preferences for all practices except MAP compared to their average rating across practices.

Level 2: Therapist-Level Predictors

Therapist Characteristics and Emotional Exhaustion: The therapist characteristics that were significantly associated with more favorable practice-specific perceptions were: race/ethnicity, F(2, 655.35) = 5.18, p < .01; tenure at their agency, F(1, 767.87) = 4.35, p < .05; and licensure status, F(1, 642.43) = 19.07, p < .001. Specifically, therapists who were Hispanic/Latino reported more favorable perceptions compared to non-Hispanic White therapists, b = 0.19, p < .01; therapists who were unlicensed reported more favorable perceptions compared to licensed therapists, b = 0.27, p < .001; and therapists who had longer tenure at their agency, b = 0.01, p < .05 reported more favorable perceptions. Therapist gender, education, number of practices in which they were trained, caseload, and number of direct service hours per week were not significantly associated with their perceptions towards practices. Ratings of emotional exhaustion F(1, 690.63) = 36.94, p < .001, b = − 0.10 were significantly associated with therapist perceptions towards practices. Specifically, those who reported lower levels of emotional exhaustion had more favorable perceptions towards practices.

Level 3: Agency-Level Predictors

Perceptions of Organizational Context: Aggregated therapist ratings of organizational context (OCM Involvement, Autonomy, and Performance Feedback subscales) were not significantly associated with individual therapists’ perceptions of practices.

Discussion

This study examined the perceptions of leaders and therapists towards six specific child mental health practices delivered within a system-driven multiple EBP implementation effort in LACDMH. Overall, findings indicated that perceptions towards the six practices were moderately favorable among both leaders and therapists and aligned with the average ratings reported in Cook et al. (2015) that reported on the psychometric properties of the PCIS, our outcome of interest. However, leaders and therapists reported different preferences for specific practices and their preferences were associated with distinct sociodemographic and professional background characteristics and their perceptions of organizational context.

Compared to the average rating of practice-specific perceptions (grand mean across reporters and practices), therapists reported more favorable perceptions towards Triple P and CPP compared to leaders, whereas leaders reported more favorable perceptions towards MAP and SS compared to therapists. These findings are consistent with the literature reporting differences in general versus practice-specific perceptions (Gray et al. 2007; Reding et al. 2014) and with our hypothesis that practice-specific perceptions would differ based on staff role due to unique priorities and values attributed to EBPs. Although our ability to report the explicit reasons for practice-specific preferences was beyond the scope of the current study, we offer several possible explanations for these differences in preferences. First, the therapist preference towards Triple P and CPP may be explained by the relative importance of the clinical content or training costs associated with delivery. Specifically, both Triple P and CPP are caregiver-mediated interventions and thus necessitate a specialized skill set. Community mental health therapists have expressed their receptivity to targeted training in working with caregivers to promote therapeutic engagement and clinical care (e.g., Baker-Ericzen et al. 2013; Brookman-Frazee et al. 2012). However, these practices are among the more expensive of the six practices of our interest in which to train therapists in LA County, which may explain the less favorable ratings of leaders towards CPP and Triple P (Los Angeles County Department of Mental Health 2016). It should be noted that when examining perceptions across leaders and therapists and across practices, CBITS was perceived as the least favored. We caution against over-interpreting this finding because, within the PEI Transformation, CBITS had very limited uptake (see Brookman-Frazee et al. 2016). A next step within this line of research is to qualitatively analyze interview data from leaders and therapists to contextualize challenges associated with implementation of these initial practices, including CBITS.

Second, leader preference towards MAP and SS supports the literature that has identified factors related to EBP funding sources and costs, training resources, and staffing considerations as important in EBP implementation according to leadership (Aarons et al. 2009; Proctor et al. 2007; Willenbring et al. 2004). Specifically, MAP has several characteristics that may be appealing to leaders. To start, there were systematic efforts to scale up implementation of MAP in Los Angeles County through partnership with key system and agency leadership to support the fit within the mental health service context (Southam-Gerow et al. 2014). Further, MAP was designed as a transdiagnostic therapist decision support system to facilitate selection and delivery of EBP strategies, thus allowing for therapists to address a wide variety of presenting problems (Chorpita et al. 2011; Rotheram-Borus et al. 2014; Southam-Gerow et al. 2014). The breadth of clients who are appropriate for MAP may contribute to potential fiscal savings of training therapists to deliver a broadly applicable intervention. Further, when used effectively, the Clinical Dashboard offers comparatively greater clinical oversight via visual display of session-to-session data relative to usual care or other practices that we examined. These clinical data may facilitate the efficiency of case selection in supervision, which could be another appealing feature of MAP from leaders’ perspective. While some of these characteristics may also be attractive to therapists (e.g., broad clinical application), MAP requires administrative time outside of session to use the decision and practice support tools (e.g., Clinical Dashboard) that typically necessitate session-by-session attention. In addition, the oversight afforded by the wealth of progress monitoring may be perceived by therapists as limiting their autonomy. Qualitative data from therapists delivering EBPs in CMH settings indicates that preparation time is often considered a challenge (e.g., Drahota et al. 2014; Brookman-Frazee et al. 2012). Thus, time required of therapists and the potential for increased oversight may lessen the appeal of MAP for therapists compared to leaders. Related to leaders’ preference for SS, this may be related to the availability and relative swiftness of training (1 day workshop), thus facilitating workforce capacity for SS clinical delivery. Given these leader and therapist differences in preferences for specific practices, further research is needed to test explanatory models of whether and how these preferences impact direct practice delivery or implementation support provided to therapists (e.g., consultation or supervision procedures).

Findings related to our second study aim indicated that the unique preferences of leaders and therapists were associated with distinct sociodemographic and professional characteristics as well as perceptions of their current organizational context. For leaders, those who had master’s degrees (compared to those with doctoral degrees) and those who reported a higher number of practices adopted at their agency had more favorable perceptions towards practices. Given the limited quantitative data examining leader characteristics associated with perceptions of specific EBPs, these findings are novel and clearly highlight the need for additional research. However, some interpretation can be offered based on EBP implementation theory (Aarons et al. 2011; Damschroder et al. 2009) that has described key inner context drivers of leaders related to EBP implementation. It was unexpected that leaders with doctoral degrees reported less favorable views towards practices given that implementation research focused on therapists has revealed a positive association between education and EBP attitudes (e.g., Aarons 2004; Aarons et al. 2012; Reding et al. 2014). Follow-up bivariate analyses between education and related seniority variables (agency tenure and licensure status) indicated small (r values ranged from − .003 to .124) and non-significant correlations. However, training in EBPs and integration of science and practice is highly variable in mental health doctoral programs (Health Service Psychology Education Collaborative 2013), such as PsyD versus PhD programs. Data on the type of doctoral degree were not collected from leaders so it is not known the extent of research or EBP training that our doctoral-level leaders received. Finally, leaders in agencies that adopted more practices reported more positive perceptions of EBPs. It is plausible that these organizations were generally more receptive to EBP implementation overall, a cultural factor hypothesized in multiple implementation frameworks to support the implementation process (Aarons et al. 2011; Damschroder et al. 2009).

For therapists, those who reported as Hispanic/Latino, unlicensed, and who had been working at their agency for a longer tenure, reported more favorable perceptions towards practices. These findings are highly consistent with the therapist background and training characteristics identified in Reding et al. (2014) that were related to EBP-specific attitudes. Further, these therapist characteristics are conceptually consistent with theory on EBP implementation (Aarons et al. 2011) and the empirical literature on general attitudes towards EBPs. For example, we found that unlicensed therapists reported more favorable attitudes towards practices. This finding aligns with both qualitative work (Proctor et al. 2007) that therapists who “are closer to their training” were described as more receptive to EBP training and delivery compared to more senior staff therapists, and quantitative work (Aarons 2004) documenting that intern therapists had more positive general EBP attitudes relative to full-time staff. Although consistent with previous work, our therapist sample, in particular, was unique because the average agency tenure was approximately 4 years and nearly half of these therapists were unlicensed. We conducted follow-up univariate analyses to probe the association between licensure status and agency tenure. There was a significant mean difference in the agency tenure between licensed therapists (M years = 6.12; SD = 5.17) and unlicensed therapists (M years = 2.74; SD = 2.80). This offers greater support for interpreting this finding as being related to unlicensed therapists may be “closer to their training” and thus hold more favorable attitudes towards EBPs.

Over and above individual characteristics, findings indicated that a specific facet of organizational context was associated with leader but not therapist practice-specific preferences. For leaders, the positive association between leader assessment of workforce capacity and quality and practice perceptions expands the implementation literature that has reported on the role of leadership in promoting adoption and continued use of EBPs over time (Aarons et al. 2015, 2014, 2016). For example, the literature on “primary climate embedding mechanisms”—strategies that leaders can use to create and support an organizational climate receptive to EBP implementation—includes two strategies that specifically relate to leader decisions about resource allocation and provider staffing decisions germane to EBP implementation and oversight (Aarons et al. 2014; Schein 2010).

For therapists, lower levels of emotional exhaustion were associated with more favorable ratings towards practices. The negative association between therapist evaluation of emotional exhaustion and practice ratings supports a similar finding reported in Reding et al. (2014) that shared the same implementation context as the current study. It may be that more favorable perceptions towards using EBPs mitigates the impact of therapist burnout, which is consistent with prior research that has documented the potential buffering impact of EBP implementation on provider emotional exhaustion (Aarons et al. 2009). Agency-level therapist perceptions of organizational context were not related to their views on practices. This was somewhat unexpected given the growing theoretical and empirical literature highlighting the influential role of organizational climate on the EBP implementation process (e.g., Aarons et al. 2011, 2017; Damschroder et al. 2009; Glisson et al. 2006; Peterson et al. 2014). We conducted follow-up univariate analyses to determine whether there were differences in therapist reports of organizational climate on the OCM by agency. There were significant differences (p < .001) in the average ratings of organizational climate per therapist-report between agencies in our sample. Further, between 10 and 20% of the variance in therapist-reported views on the OCM subscales was at the agency-level (Autonomy ICC = 0.10; Performance Feedback ICC = 0.15; Involvement ICC = 0.20). These follow-up analyses suggest that there is a good amount of variability among therapists in their assessment of organizational climate within an agency but there is also some amount of systematic variation in therapist perceptions of climate accounted for by agencies. Our findings also suggest that therapists’ professional (agency tenure, licensure status) and background (race/ethnicity) characteristics and their individual perceptions of workplace strain (emotional exhaustion) are stronger predictors of their perception of practices than agency-wide average assessments of organizational context.

This study contributes to the implementation literature by specifically examining and comparing leader and front-line provider perceptions of EBPs and highlighting the role of views on organizational context in preferences towards EBPs (Glisson et al. 2006; Lehman et al. 2002). This study is also particularly unique because of the service system backdrop of our study in which a key outer context factor (i.e., funding for EBP implementation) was fixed through a fiscal mandate that had been established several years prior to our data collection, placing the timing of our data collection within the sustainment phase of the EPIS framework (Aarons et al. 2011) and the implementation process involved multiple practices rather than a single practice as has historically been the case in extant studies (Aarons et al. 2011). Further, our findings may inform the efforts of current and ongoing EBP implementation initiatives, particularly related to selection of EBPs to implement and their corresponding implementation strategies. For example, our findings suggest that organizational climate supportive of employee autonomy and performance along with workforce capacity may be helpful to establish at the outset of implementation efforts. Ongoing monitoring of these organizational conditions along with therapist burnout is recommended to support sustained implementation. Although beyond the scope of this paper, a natural next step will be to examine our qualitative interview data with leaders and therapists for the purposes of identifying potential convergence and/or expansion (Palinkas et al. 2011) of our quantitative findings. This will be particularly informative to refine our understanding of why leaders and therapists prefer different practices, and the influence of a broader range of organizational context variables on their views of specific EBPs.

Balanced with these strengths and unique contributions, this study has some limitations that are important to consider. A primary limitation is the cross-sectional nature of our data, precluding the ability to definitively conclude directionality of our effects. For example, it may be that leader or therapist preferences towards practices are the driver of perceptions of organizational context rather than the latter shaping attitudes towards the practices. Another limitation is the self-report nature of the data sources used in this study. While reliance on self-report methodology is a limitation, it was the best available method to assess for our constructs of interest (i.e., practice-specific perceptions; organizational functioning) in an efficient and meaningful manner. Relatedly, our survey response rates, while comparable to those reported in similar community mental health samples, (e.g., Hawley et al. 2009; Cashel 2002; Piotrowski and Keller 1989; Rosenberg and Beck 1986) may still under-represent the broadest spectrum of perspectives from leaders and therapists. Common method bias may also be a concern as some of the predictor and outcome variables were self-reported by the same participants at the same time (Podsakoff et al. 2003). This may be somewhat mitigated as there were both leader and therapist reports and these represent different organizational perspectives. A related limitation is the self-selected nature of the sample. Leaders and therapists were invited to participate in the online study survey so it is possible that those who participated may have stronger opinions about the LACDMH implementation effort compared to those who chose not to participate. Finally, a methodological limitation is that we did not administer the same battery of measures to both leaders and therapists to evaluate perceptions of organizational context. Specifically, ratings of emotional exhaustion were only collected from therapists while evaluation of staffing characteristics from the ORC were only collected from leaders in our sample. Therefore, we are unable to directly compare the specific facets of organizational functioning that are most influential to leaders and therapists as they relate to their perceptions of specific EBPs.

Taken together, findings highlight the value of gathering perspectives from multiple staff stakeholders involved in EBP implementation efforts. Our study findings support and extend the empirical and theoretical implementation literature positing that frontline providers and leaders have unique values and priorities that influence perceptions of EBPs. The unique perspectives of various stakeholder groups showcase the complexity of the implementation process and the importance of considering multiple stakeholder perspectives in initial EBP selection and ongoing refinement to implementation procedures.