Introduction

The dissemination and implementation of evidence-based practices (EBPs) to improve the quality of services and outcomes for families and youth may help to improve the quality of care in real-world human service settings. Considerable resources are being used to increase the implementation of EBPs into community care. For example, the California Mental Health Services Act has set aside funding to support implementation of EBPs and the State of Ohio has developed “Coordinating Centers of Excellence” to promote use of best-practices and EBPs. However, actual implementation requires consideration of the multiple stakeholders who are impacted and, importantly, mental health service providers in community settings. It is important to consider service provider attitudes toward adopting EBPs in order to better tailor implementation efforts to meet the needs of providers in community agencies. Measures to assess provider attitudes should be stable across different service settings, geographic locations, and data collection methods. This study examines the factor structure of the Evidence-Based Practice Attitude Scale (EPBAS) in a different and more geographically diverse sample and employs a different data collection method than that of the original scale development study (Aarons 2004).

Multiple factors at different system and organizational levels influence implementation of innovation in mental health settings. These include the social, economic, and political context, characteristics of the innovation itself, characteristics of the organization attempting to implement the innovation, and characteristics of both the providers and clients (Aarons 2004, 2005; Glisson and Schoenwald 2005; Greenhalgh et al. 2004; Grol and Wensing 2004). Mental health service providers’ attitudes toward change and innovation may influence the implementation of EBPs at several stages. First, the attitudes of providers toward innovation in general can be a precursor to the decision of whether or not to try a new practice. Second, if providers do decide to try a new practice, the affective or emotional component of attitudes can impact decision processes regarding the actual implementation and use of the innovation (Candel and Pennings 1999; Frambach and Schillewaert 2002; Rogers 1995). However, measurement of provider attitudes toward EBPs has only recently been undertaken and consistency in the structure of such measures across provider groups and geographic variation has yet to be demonstrated.

The Evidence-Based Practice Attitudes Scale (EBPAS; Aarons 2004) was developed to assess mental health provider attitudes toward adoption of innovation in mental health services. The EBPAS assesses four dimensions of attitudes toward adoption of EBPs including: (a) the intuitive Appeal of EBP, (b) the likelihood of adopting an EBP given Requirements to do so, (c) Openness to new practices, and (d) perceived Divergence between research-based/academically developed interventions and current practice. The measure fills a void in that it allows for quantitative assessment of provider attitudes that can then be used in models of innovation implementation and to assess provider readiness to adopt new practices. However, other than the original scale development study, there have been no published tests of the EBPAS’ factor structure. The present study examines the factor structure of the EBPAS in a sample of mental health professionals providing services to children and adolescents with severe emotional disabilities (SED) and their families. It was hypothesized that the four-factor solution derived by Aarons (2004) would provide a good fitting model of provider attitudes towards EBPs. This study differs from previous work in three important ways. First, it uses a new and independent sample. Second, the catchment area is much more geographically diverse (17 states) compared to that of the original scale development that took place in one large California County. Finally, it uses a web-based survey in contrast to the in-person, paper–pencil method used in the original scale development study.

Methods

Sample Identification

A list of mental health agency providers affiliated with communities funded under the federal Comprehensive Community Mental Health Services for Children and Their Families (CCMHS) Program was compiled. The CCMHS Program is funded by the Substance Abuse and Mental Health Services Administraton Center for Mental Health Services. Further details on this initiative can be found elsewhere (Holden et al. 2001). Snowball sampling was used to generate a provider list: structured community-contact telephone calls were made to 22 currently funded CCMHS communities to identify all of the local mental health agencies providing services to children with SED. Twenty-one of the 22 communities identified affiliated mental health agencies. Next, a telephone contact was made with each identified agency to obtain a list of their mental health providers. Each mental health agency contact was also asked to identify other local mental health agencies that provided similar services to children with SED and those not previously identified were also contacted, resulting in the identification of 703 potential individual respondents.

Procedure

A multi-stage emailing process (Dillman 2000) was used to recruit selected potential respondents (N = 703) for the survey: (1) a pre-survey email, (2) survey invitation email with web link, username, and password, (3) reminder email to the full sample, (4) reminder follow-up email to those who had not yet responded, and (5) targeted follow-up phone calls to non-responders. Stages 1 through 4 were completed at one-week intervals and stage 5 was completed over the month subsequent to stage 4. Data collection was conducted August through October 2005. The institutional review boards at the organizations conducting the study approved all data collection procedures. Respondents were informed that completion of the survey indicated their consent. Survey responses were received from 288 mental health providers representing a response rate of 41%, a rate higher than those found in studies of strategies for web-based survey response rate improvement (e.g., 20.7–31.5%; Kaplowitz et al. 2004). Of the 21 CCMHS communities who provided lists of agencies in their area, 19 provided data for the overall study, resulting in a community level response rate of 90.5%. One of the responding communities had missing data for EBPAS items resulting in service provider data from 18 of the CCMHS communities from 17 states.

Participants

The current study includes only those respondents (N = 221) who provided direct mental health services, and had no more than one missing value for the EBPAS. Sixty (27.1%) respondents worked for public mental health agencies, 26 (11.8%) for private-for-profit agencies, 130 (58.8%) for private-not-for-profit agencies, and the remaining 5 (2.3%) for “other” types of agencies. Respondents had worked as mental health providers for a mean of 11.77 years (SD = 9.00). Their mean age was 40.83 years (SD = 11.23; Range: 23–72), and 57.5% (n = 127; data missing for 47) were female. Just under 60% (n = 128) were licensed mental health providers. Twenty-eight respondents (12.7%) had a doctoral degree, 157 (71%) a masters degree, 32 (14.5%) a bachelors degree, and 3 (1.4%) had attended some college but had no degree (data missing for one respondent). Primary disciplines included psychology or counseling (n = 100; 45.2%), social work (n = 75; 33.9%), marriage and family therapy (n = 13; 5.9%), and “other” (e.g., nursing, education; n = 29; 13.1%). Data on primary discipline were missing for four (1.8%) providers. Of those who provided information on race, the majority self-identified as Caucasian (n = 154; 84.1%), followed by African American (n = 17; 9.3%), Asian (n = 3; 1.6%), American Indian (n = 2; 1.1%), and “Other” (n = 7; 3.8%).

Measures

The present study focuses on the EBPAS (Aarons 2004), which was embedded within a larger survey assessing providers’ knowledge, training, and use of evidence-based treatments (Evidence-Based Treatment Survey; Walrath et al. 2006). The EBPAS consists of 15 items measured on a 5-point scale ranging from 0 (Not at all) to 4 (To a very great extent). The EBPAS is comprised of four subscales and a total scale score, which represents respondents’ global attitude toward adoption of EBPs. For the subscales, the Appeal subscale assesses the extent to which the provider would adopt an EBP if it were intuitively appealing, could be used correctly, or was being used by colleagues who were happy with it. The Requirements subscale assesses the extent to which the provider would adopt an EBP if it were required by an agency, supervisor, or state. The Openness subscale assesses the extent to which the provider is generally open to trying new interventions and would be willing to try or use more structured or manualized interventions. The Divergence subscale assesses the extent to which the provider perceives EBPs as not clinically useful and less important than clinical experience. The EBPAS Total score is computed by first reverse scoring Divergence scale item scores and then computing the overall mean and reliability. Cronbach’s alpha reliability for the EBPAS is good (alpha = 0.77), with subscale alphas ranging from 0.59 to 0.90 (Aarons 2004), and the measure’s validity is supported by studies of EBPAS score associations with mental health clinic structure and policies (Aarons 2004), culture and climate (Aarons and Sawitzky 2006) and leadership (Aarons 2006).

Analyses

A confirmatory factor analysis (CFA) was conducted specifying the factor structure identified in the original EBPAS scale development study (Aarons 2004). CFA is a method for testing the structure of items, scales, and measures (Long 1983). Rules of thumb suggest having at least four observations for each measured variable but a higher observation to case ratio is preferable (Rummel 1970) and this study had 14.7 observations for each measured variable. In CFA one can specify whether or not individual items load on only one factor or on multiple factors (i.e., cross-loadings). CFA was used in this study because previous EBPAS exploratory and confirmatory factor analyses indicated support for the factor structure applied here. We also wanted to determine if the factor structure was generalizable and could be replicated in a different sample and with a different data collection method. In this study EBPAS items were constrained to load only on their respective subscale with no cross-loadings, thus providing a highly stringent test of the factor structure. Maximum likelihood estimation with robust standard errors (MLR) was used and, as in the original study, factor intercorrelations were allowed. When CFA is conducted with data from a complex or nested design (e.g., clinicians nested in communities), it is generally advisable to adjust for potential dependency or intercorrelations within clusters. Our CFA model was estimated using Mplus analytic software (Muthén and Muthén 1998–2004) and because providers were nested within communities (k = 18), analyses controlled for the nested data structure by adjusting standard errors and the chi-square fit statistic. Examination of modification indices suggested that the association between two highly correlated items within the Appeal scale be freely estimated. In addition, three cases had missing data, which resulted in computational problems (matrices not positive-definite). The expectation maximization method of imputation was used to estimate values for missing data (Little and Rubin 2002; Schafer 1997) and these modifications solved computational issues.

Model fit was assessed using a combination of fit indicators including the χ 2 and χ 2/df ratio, the comparative fit index (CFI), the Tucker–Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). Use of these indices in combination provides a more comprehensive evaluation of model fit where a χ 2/df ratio less than 3 indicates good fit (Carmines and McIver 1981), and CFI and TLI values greater than .90, RMSEA values less than .10, and SRMR values less than .08 indicate excellent model fit (Dunn et al. 1993; Hu and Bentler 1999; Kelloway 1998).

Results

The CFA results indicated that the model demonstrated very good fit, supporting the EBPAS factor structure (χ 2(83) = 183.51; CFI = .92; TLI = .90; RMSEA = .07; SRMR = .07; χ 2 /df = 2.21). As shown in Table 1, all factor loadings were statistically significant and the model supported the a prior factor structure of the EBPAS. Factor intercorrelations ranged from r = .15 to r = .48. The Appeal subscale had a large positive correlation with the Requirements (r = .48, p < .01) and Openness subscales (r = .42, p < .01). The Requirements and Openness scales were significantly positively correlated (r = .21, p < .05). The Divergence subscale had a moderate negative correlation with the Openness subscale (r = −.36, p < .01), but was not significantly correlated with the Appeal (r = .20) or Openness (r = .21) subscales (p’s > .05). Table 1 shows the means, standard deviations, item–total correlations, and Cronbach’s alphas of the EBPAS subscales and EBPAS Total scale. Subscale reliabilities ranged from .66 to .93 with an EBPAS Total scale alpha of .794.

Table 1 EBPAS subscale and item means, standard deviations, item–total correlations, and Cronbach’s alpha
Full size table

Discussion

The results of this study provide strong support for the EBPAS factor structure of attitudes toward EBPs including the four subscales of Appeal, Requirements, Openness, and Divergence. Psychometric analyses indicated that the EBPAS subscales and EBPAS Total scale demonstrate fair to excellent internal consistency reliability, which were improved in the current study compared to the original scale development study. The reproduced factor structure supports the utility of the EBPAS using varied settings, samples, and methods, given the geographically diverse nature of the present sample and the different (i.e., web-based) method of data collection that was used relative to the original scale development study.

Consistent with the previous scale development, the Appeal, Requirements, and Openness subscales were all positively correlated, suggesting that while these three dimensions of provider attitudes toward EBPs are distinct, they also address a common theme of perspectives on EBPs. Providers who endorse positive attitudes toward changes in practice that are part of work requirements may have more open attitudes toward adoption of EBPs in general and also endorse more positive attitudes given the intuitive appeal of EBPs. The negative correlation of the Divergence and Openness subscales suggests that providers who perceive that their preferred or usual practice differs from research-based interventions such as EBPs are less likely to be open to trying such interventions that are new to them.

Some limitations should be noted. Although the response rate (41%) is higher than other published web-based response rates (Kaplowitz et al. 2004), it is low when compared with the original scale development study that used in-person on-site surveys (i.e., 96%). However, the web-based data collection method was deemed most practical given the geographic dispersion of the participants. In the present study we could not compare respondents with non-respondents to examine potential bias. In light of research suggesting that both provider characteristics (e.g., education) and organizational context (e.g., level of organizational bureaucratic structure; leadership) play a role in the implementation of EBPs in real-world settings (Aarons 2005; Glisson 2002), future research would benefit from examining the relationship between these types of variables and provider attitudes toward EBP. The purpose of this study, however, was to examine the EBPAS factor structure and internal consistency in a new and different sample.

Additional research is needed to further establish the concurrent and predictive validity of the EBPAS. For example, provider attitudes toward adopting EBPs may predict uptake and satisfaction with new practices. Most critical is establishing a link between attitudes and the fidelity with which EBPs are implemented. However, where local adaptation of EBP is needed to meet the constraints of implementation context or the needs of particular clients or populations (Aarons and Palinkas 2007), fidelity assessment may require reformulation.

It is likely that there are a number of other dimensions of provider attitudes toward EBP and change in practice that have yet to be identified and explored. For example, the impact of learning new clinical skills on perceived professional accomplishment might impact attitudes toward adopting new practices. Input from clinicians and supervisors could provide insight into potentially important questions that might impact staff attitudes and implementation of EBPs. Future studies should address these concerns in an effort to tailor practice change efforts to the attitudes and preferences of mental health service providers.