Introduction

In 2003, The President’s New Freedom Commission on Mental Health urged the public sector to provide financial incentives in support of evidence-based practices and endorsed treatments based on research for people with serious mental illness. The National Implementing Evidence Based Practices Project examined the implementation of five psychosocial practices in routine mental health care settings in eight states. The practices {and number of sites} were assertive community treatment {13}, family psycho-education {4}, illness management and recovery {12}, integrated dual disorders treatment {11}, and supported employment {9}.

The Project used a common implementation model that was developed by experienced mental health services researchers and grounded in a literature review. The implementation model was developed under the auspices of Dartmouth Psychiatric Research Center, which subsequently became the Project’s Coordinating Center with responsibility for monitoring practice implementation over 2 years from their start date. The Dartmouth Psychiatric Research Center and its research collaborators in eight states conducted a mixed-methods evaluation of the evidence-based practice implementation in each site. This included semi-annual fidelity reviews, in-depth interviews with key informants, staff surveys, and on-site monitoring. Mental health authorities identified sites in each state, and most sites received incentive funding for their participation. Evidence-based practices were deployed on a staggered schedule, starting in summer 2002 and ending the 2-year implementation phase in late 2005.

Evidence-based practice-specific implementation resource kits (“toolkits”) and consultant/trainers were provided to each site in order to facilitate implementation. Toolkits supplied educational, training, and reference materials for use by program leaders, administrators, consumers, and clinical staff members. Consultant/trainers were first-line resources for evidence-based practice program leaders, offering practice-specific training, clinical supervision, and on-going consultation. The consultant-trainer also advised agency leadership on organizational changes considered optimal for successful implementation. Additional details concerning the National Implementing Evidence-Based Practices Project are available in Drake et al. (2001), Torrey et al. (2001), Dixon et al. (2001), Carpinello et al. (2002), Goldman et al. (2001), and Mueser et al. (2003).

The Project yielded a wealth of data that has been analyzed to assess various facets of the implementation process. These included, for example, implementation strategies (Rapp et al. 2007), client outcome monitoring (Marty et al. 2007), fidelity (McHugo et al. 2007; Bond et al. 2008), and staffing issues (Woltmann et al. 2008). Others have looked at the influence of important externalities such as state policy (Magnabosco 2006; Isett et al. 2007) and state supported technical assistance centers (Salyers et al. 2007). Groups of researchers have examined implementation of individual evidence-based practices across sites: integrated dual disorders treatment (Wieder and Kruszynski 2007; Brunette et al. 2008), supported employment (Marshall et al. 2008), illness management and recovery (Whitley et al. 2009), and assertive community treatment (Mancini et al. 2009). The project enhanced current interests in dissemination of evidence-based practices. For example, supported employment is being implemented at other sites within the US (Drake et al. 2006) and internationally (van Erp et al. 2007).

Practice sustainability, that is, continuation at its original site, is rarely the subject of research studies on innovative health services delivery (Greenhalgh et al. 2004). Two recent studies of innovative depression models for primary care (Meredith et al. 2006; Blasinski et al. 2006) included post-implementation research on model sustainability, with follow-up periods of 18 and 12 months, respectively. Both studies used semi-structured telephone interviews to assess sustainability. Meredith and colleagues found variability in model component sustainability at the 17 sites. The clinical information systems were most often maintained (96% of sites), followed by delivery system redesign (59%). Blasinsky et al., reporting on Project IMPACT, found that five of seven sites sustained their practices, and they cited organizational support, staff training, funding, and successful clinical outcomes as important factors. The samples in these studies were small, and follow-up periods were short. Moreover, their populations were less exigent than the seriously mentally ill, who are most in need of improved services and least likely to receive them in routine mental health settings (US Department of Health and Human Services 2003).

The research presented here reports on sustainability of the practices within the National Implementing Evidence Based Practices Project for people with serious mental illness. Forty-nine sites completed the initial 2-year implementation phase and were the focus of our study. Our aims were to discern (1) the number of sustainers overall and by evidence-based practice, (2) the reasons sites sustained or did not sustain their practices, (3) the characteristics of sustainers and non-sustainers, and (4) the nature of, and reasons for, any adaptations of the sustained practices.

Methods

The primary outcome was evidence of practice sustainability 2 years after the initial implementation phase. We used a mixed-methods approach, based on a telephone survey that gathered qualitative and quantitative data from site representatives and others familiar with the sites and practices during the follow-up period.

Participants

The unit of analysis was site. The inclusion criteria were implementation of the evidence-based practice and participation in regular fidelity reviews at the site during the initial 2-year period. At the conclusion of the initial implementation phase, we conducted endpoint interviews with program leaders and administrators at the sites and with consultant/trainers, all of whom informed our decision as to whether or not a site had continued implementation of the practice. Two family psycho-education and two integrated dual disorder treatment sites did not complete the 2-year initial phase, which restricted for follow-up pool to 49 out of the original 53 sites. In order to insure at least one follow-up interview per site, we included site representatives and external sources in our interview respondent pool. Our preference for a site representative was the current program leader, because these individuals had the most intimate knowledge of the current practice. We also interviewed clinical practice directors and team leaders. Our primary external source was the consultant/trainers, who had served during the initial implementation phase, provided they had current knowledge of the practice. When consultant/trainers were no longer involved with the sites, we interviewed mental health authority staff members who were responsible for regulating the practices for their states. This method of triangulation of data sources strongly enhances validity, provides a more complex and balanced perspective of the situation, and serves to counteract biases intrinsic to the participants’ roles (Miles and Huberman 1994). We believed that program leaders would provide the most detailed information and consultant/trainers, because they supported multiple sites, would add critical objectivity.

Respondents were contacted by phone or email and asked to participate in a brief telephone survey, between 30 and 40 min long. They received a copy of the interview protocol before it was administered by telephone. Surveys were administered between March 2007 and September 2008. Questions were anchored in time to the 2 years following the end of the initial 24-month implementation period. At the beginning and frequently during the survey, respondents were reminded of the time period in question, which was “the last 2 years.”

Measures

The telephone survey contained 47 questions including both quantitative and qualitative items, divided into three sections (available upon request from the authors). Section A ascertained whether or not the site was continuing the practice, and elicited reasons for sustaining or not sustaining. Respondents that affirmed sustainability were asked questions about practice evaluation, penetration, training, supervision and consultation. Some of these questions were open-ended, and responses were recorded verbatim. Other questions were close-ended (e.g., “Have you conducted fidelity assessments in the past 2 years?”) or asked about frequency or intensity (e.g., “How often have you conducted fidelity assessments?”).

In Section B, respondents from sustaining sites were asked whether or not the practice had been modified to suit local conditions, including socio-cultural milieu, local regulations or policies, client characteristics, practitioner skills or experience, or recent research findings. Respondents were asked to describe the adaptations to the practice model and to rate the degree of adaptation on a Likert-type scale (1 = little, 5 = considerable).

Section C investigated the factors affecting sustainability for both sustainers and non-sustainers. Respondents were asked to rate the impact of 15 factors on a five-point scale (very negative, negative, neutral or no effect, positive, very positive). Non-sustaining sites were asked to answer questions only if they were able to do so adequately in the context of the post-implementation study timeframe. A summary question asked if there was anything else the informant would like to add that would help in understanding the sustainability (or not) of the practice at their agency.

The procedures were approved by the institutional review board at Dartmouth College. All authors certify responsibility for the study. There are no known conflicts of interest.

Data Collection and Analysis

Data were transcribed from the questionnaire and entered into ATLAS.ti for qualitative data analysis. Individual codes were combined into categories, and linkages between categories generated development of themes. For example, on the question of reasons for sustainability, we selected quotes that conveyed this concept and tagged them with the code, “reasons for sustainability”. Descriptive codes such as “financing” might be linked to the category “reasons for sustainability,” as well as to other descriptive codes such as “mental health authority,” to develop themes such as, “State mental health authority financial support of a practice was a reason for sustaining.” On the topic of practice adaptation, we uncovered numerous distinct reasons, and each of these was coded separately (e.g., adaptation/client population or adaptation/staffing).

Quantitative data were analyzed using descriptive statistics. For site-level dichotomous data, we examined frequency and percent within various strata, (e.g., evidence-based practice, sustainers/non-sustainers). Continuous data were analyzed as means within strata. Because we expected program leaders to have more detailed practice information, we planned to resolve disparities on pivotal issues, such as judgment of sustainability, in their favor. In practice, major disparities rarely occurred and never in judging sustainability. We also found that consultant/trainers were more reliable reporters of fidelity assessments, probably because they often administered them. Sources of qualitative data (e.g., program leaders, consultant/trainers) are cited with their responses. Whenever possible, we verified assertions attributed to the actions of third parties such as state governments and other organizations external to the agency by personal contacts with the entities or through their organizational websites. Commonalities and disparities between qualitative and quantitative data were identified. This triangulation acted as a cross-check on regularities in the research data, increasing the validity and credibility of the results (Miles and Huberman 1994).

Results

There was at least one respondent for each of the 49 sites. Of these sites, 39 (79.6%) sustained the practice for 2 years after initial implementation, and 10 (20.4%) failed to sustain. Table 1 shows the number/percent of sustaining and non-sustaining sites stratified by evidence-based practice.

Table 1 Status of evidence-based practices at 2-year follow-up
Full size table

Reasons for Sustaining

Reasons given for sustaining were compiled from the direct question or from quotes in other portions of the transcript that used reference phrases such as: “The reason we’re still doing this is…”; “Training [funding, staff] is key to sustaining”; and “We need to have [funding, agency support].” The given reasons for sustaining or not were grouped into conceptually linked (within the survey instrument) and thematic categories (emerging from qualitative data as described above): (1) state support for the practice, (2) practice proficiency, (3) practice evaluation, and (4) agency leadership and staff support.

(1) State support for the practice included direct financing, technical assistance, and practice evaluations. Direct financing came in the form of block grants and favorable Medicaid reimbursement, including federal government support, for evidence-based practices. Some states dipped into their general funds to support evidence-based practices and services of external organizations linked to evidence-based practices (Maryland Department of Health and Mental Hygiene 2006). Respondents described the impact of funding variously. “Money is necessary; otherwise there is drift from the model (assertive community treatment).” “Funding got us going and kept us interested in assertive community treatment over the years.” “Per diem rate is very important. Assertive community treatment is expensive.” “We got a grant from the mental health authority to fund non-reimbursable aspects in joining [family psycho-education].”

Respondents also described contingencies for continued funding and practice licensing. “Evidence-based practices get Medicaid reimbursement at a higher rate. We get paid the higher rate for producing competent work [supported employment].” “The mental health authority tied the assertive community treatment licensing structure and Medicaid incentive financing to high fidelity.” Sanctions for persistently low fidelity scores included suspension of practice licenses and withdrawal of favored Medicaid billable rates. The financial models themselves differed among states. Some respondents reported having per diems, whereas others had capitated case rates and reimbursement for specific aspects of evidence-based practices. State funding provided critical support to sites that sustained. Most respondents were aware of this, and some voiced concern about potential withdrawal of this support, either in favor of other evidence-based practices or more general cost consolidation, “If funding disappeared, it would be difficult to continue functioning at our current level.”

The quantitative results in Table 2 show the mean (SD) of factors rated by participants as affecting sustainability. The results support respondents’ perceptions of the positive impact of financing for sustainers (3.3 ± 1.2) and negative impact for non-sustainers (2.0 ± 1.3). Although the mean financing rating for sustainers is only slightly positive, the State Mental Health Authority factor, which is a proxy for financing, had a higher positive value of 3.9 (±1.0) for sustainers and, on average, a neutral value (2.8 ± 1.2) for non-sustainers.

Table 2 Factors affecting sustainability of evidence-based practices
Full size table

States supported technical assistance was provided through Technical Assistance Centers or university-based training and consultation centers. Most site-level respondents said that they had neither resources nor time for in-house training and depended heavily on training centers for new hire and booster training. Consultation, except when combined with fidelity reviews, was described as occurring sporadically, by e-mail or phone conversations by both external and internal sources. The consultative relationship between consultant/trainers and program leaders became more peer-to-peer over time and was viewed as psychologically supportive in comments from program leaders such as, “Talking to the [assertive community treatment] Center, catching the vision and passing it onto staff was very motivating.”

At sustaining sites that had significant state financing, training, consultation and other supports, program leaders often emphasized the necessity of such support for continuation of their practices. Table 3 illustrates differences between those with significant state supports and those without. Sites not reporting significant state support also said they had less training, fewer fidelity reviews, and more practice adaptations. Sites that reported state support for training, consultation and fidelity, but no direct on-going practice financing, reported somewhat more local practice adaptation.

Table 3 Characteristics of sustaining sites with or without supports from state mental health authorities
Full size table

(2) Practice Proficiency included training and supervision. Training was reported regularly as key to sustaining an evidence-based practice. External training predominated, largely through the auspices of the state. Supported employment sites participating in the Johnson and Johnson-Dartmouth Community Mental Health Program (Drake et al. 2006) received program-sponsored training. Many program leaders reported continuing to provide some internal training, and agency leadership provided non-billable time and resources for their staff to attend training sessions. For time efficiency respondents said that they consolidated training to focus on core aspects of the practice. Innovations such as home-grown videos, internet training, and power point presentations were produced for group and individual use. Sustaining sites spawned “agency trainers” and “peer consultants” to act as resources to new implementation sites and novice program leaders. Respondents said that on-going training mitigated staff turnover. A typical response was, “Staff turnover is inevitable. Continuous training offsets the problem.”

Table 4 presents the rates of supportive activities at the sustaining sites. Sustainers had a high rate of training for each evidence-based practice, notably for supported employment and assertive community treatment sites, where virtually all sites provided training. Mean training hours for the 2 years varied by practice, ranging from 13 (±1.4) for family psycho-education to 81 (±62.1) for integrated dual disorders treatment. The reported duration of training for integrated dual disorders treatment is consistent with the model, which requires extensive clinical knowledge and skills (Brunette et al. 2008).

Table 4 Supportive activities by sustaining sites during the past 2 years
Full size table

Supervision was viewed as integral to sustaining a practice, but site-level respondents reported that supervision time decreased after the initial implementation period due to time constraints. At some sites where multiple evidence-based practices were implemented, group supervision was integrated, with designated times for individual practices on a rotating basis. Even at sites where competition among practices was not a factor, program leaders reported that supervision time decreased during the sustaining period. Despite site-level respondents’ belief that supervision was inadequate, high rates of supervision were reported. Mean supervision hours for the 2 years ranged from 39 (±14) for family psychoeducation to 162 (±100) for integrated dual disorders. The variability in supervision time across practices may be attributable to model differences, whereas within-practice variation may reflect the level of resources.

(3) Practice evaluation, whether positive or negative, was often cited by sustaining respondents as reinforcing. Program leaders viewed the fidelity reviews as validating and as an opportunity for consultation: “Fidelity helps programs focus and see room for growth”, “Fidelity is very important because there is tendency to backslide.” Program leaders were well aware of the requirement for model adherence and threshold scores with contingencies: “The state is very strict about agencies being consistent with the model [assertive community treatment]. Fidelity is tied to certification.” Where technical assistance centers and consultant/trainers were available, both program leaders and consultant/trainers reported externally conducted fidelity reviews. Although many states continued to use the specified fidelity scale, others developed modified versions of “core practice components.” External and internal sources reported that states struggled with the feasibility of setting identical thresholds scores across evidence-based practices and were in the process of reevaluating fidelity criteria by practice.

The quantitative data showed high rates of fidelity reviews in sustaining sites. Only illness management and recovery sites had a low rate of fidelity reviews (25%) during the follow-up period. Since illness management and recovery was implemented in states that neither consistently provided outside consultants to rate fidelity nor mandated fidelity reviews as a requirement for favorable funding, there was little incentive to continue. Table 4 shows the rates of sites, by practice, conducting fidelity reviews during the sustaining period, and the average number of reviews. Both program leaders and consultant/trainers reported annual fidelity reviews as compared to semi-annual reviews during the implementation period.

Outcome reporting was required by most states (for e.g., rates of hospitalization, incarceration, homelessness, and substance abuse). Outdated management information systems and a shortage of technologically capable staff often prevented site-level outcomes analysis, leaving technical assistance centers or other state-sponsored groups to provide data analysis services. Delays of 1–3 months in receiving analyzed outcomes were common. Still, program leaders offered perceptions of client outcomes, usually positive, which they attributed to the evidence-based practice: “We love the model because it works”; “Consumers do better because they are working [supported employment]”; and “Clients benefiting is a motivating factor for practitioners struggling with conflicting demands.”

Most sites reported outcomes at least twice a year or as often as required by the state mental health authority (see Table 4). Supported employment Johnson and Johnson-Dartmouth sites reported outcomes quarterly. All sustaining assertive community treatment and supported employment sites (12 and 7, respectively) said that they reported outcomes. These practices have relatively unambiguous and easily quantifiable outcomes (e.g., employment rate and institutionalization rate, respectively). Integrated dual disorders treatment, illness management and recovery, and family psycho-education sites reported outcome monitoring at lower rates.

Penetration, defined as the proportion of eligible people enrolled in the practice, was rated in comparison to the initial implementation phase (see Table 4). Most program leaders at sustaining sites reported somewhat more people served, with assertive community treatment sites reporting the greatest gains (2.5 ± 0.5). Only illness management and recovery sites reported lower penetration during the sustaining period (1.7 ± 0.6).

(4) Agency leadership and staff support. Total agency buy-in, from top leadership to front-line staff, was important in sustaining sites, more so for sites that reported less state mental health authority involvement. Leadership commitment was articulated as provision of space for the practice, time for training, financial support, and a vocal mandate for practice continuation. Agency leadership was rated among the highest factors by sustaining sites in Table 3 (mean = 4.2 ± 1.0), whereas its mean was 2.5 (±1.6) among non-sustaining sites.

The presence of a “champion,” usually the program leader, was important during the initial implementation phase, and it was also important in the sustaining phase. Hiring and maintaining a well-trained staff to counteract the negative effects of turnover was viewed as key. Program leaders cited pride in being part of something that helped clients, and seeing them improve was important to staff morale: “The model works. We love to see clients thrive [supported employment].” “I got support from my superiors and the team itself—they are dedicated.” “We have a champion-mentor and peer-to-peer success stories of effectiveness [illness management and recovery].” “We had a champion vested in consumers moving forward. Without [a champion] there is a tendency to fall back to other practices [family psychoeducation].” “We had buy-in from frontline staff, top management and mid-level people. They believe in it [integrated dual disorders treatment].”

Reasons for Not Sustaining

Ten of the sites failed to sustain their evidence-based practices. Some stopped soon after the final (2-year) fidelity review of the initial implementation period. Others continued longer and were able to provide some insight into the reasons for discontinuing, which included inadequate funding and staff turnover (see Table 2).

Inadequate funding was a pervasive and forcefully stated reason for not sustaining, “When the implementation was over and the money dried up, we stopped doing it.” Some respondents reported financial problems at the state level, “Budget problems hurt the state during [the period].” The lack of direct funding and supported training from the state, and staff shortages and turnover, were linked in responses from internal and external sources, “We had staff turnover and adding staff was difficult with no financial incentive.” It was doubly damaging without concomitant state support: “There was lack of state resources and within the agency, lack of administration support.” and “Agency leadership was not interested and [they] were reorganizing. Reimbursement was inadequate.” Some states selected one evidence-based practice to promote and finance, leaving other practices without funded consultation/training, “[other practices] got state financing. We did not.”

Staff turnover was a problem, particularly for small or chronically understaffed agencies: “Staff turnover prevented sustaining”, “Had to start all over again with new staff”, and “We had a small team covering a large area and lacked resources.” Agency culture presented a barrier to sustaining in sites that had implemented program models that their staff resisted. Once the national project ended, these practices failed to survive, and sites drifted back to former models with which they felt more comfortable and capable, “Staff liked a group format, not individual, so we went back to [the practice they did before].” “Skills and attitudes of practitioners are for [other practice].” “Our clients are low functioning. Many could not read”; and “Consumers liked [previous practice].”

Site respondents also mentioned lack of transportation, need for bilingual staff, lack of office staff, adversarial relationships with local mental health authorities, and the absence of fidelity assessments as reasons for not sustaining, “When the study ended, resources disappeared and fidelity stopped.”

Sites that failed to sustain evidence-based practices encountered a multitude of barriers, either absent from sustaining sites or perceived by program leaders at sustaining sites as more tractable. Table 2 shows average ratings on factors affecting sustaining, stratified by sites that sustained and sites that did not. The mean ratings within sustainers or non-sustainers were fairly similar, except that all sites viewed staff turnover as a barrier. The major differences were between sustainers and non-sustainers. Sustainers had consistently higher mean scores than non-sustainers. The item with the largest mean difference was training (2.0), followed by supervision (1.9), agency leadership (1.7), practitioner turnover (1.5), consultation (1.4), financing (1.3) and state/local mental health authority (1.1).

Practice Adaptations Among Sustaining Sites

Of the sustaining sites, 88% said that they had adapted the evidence-based practices to meet local needs. The extent of practice adaptation was measured on a scale from 1 to 5, with answers ranging from “a little” to “considerable” (see Table 4). Mean adaptation scores across the five practices ranged from 1.5 (±1.0) for assertive community treatment to 3.9 (±1.3) for illness management and recovery.

The more important reasons of practice adaptation were: competing evidence-based practices, client population, agency culture, staffing, and financing. As evidence-based practices proliferated in study sites after the initial implementation period, partial or complete operational integration became a practical approach for agencies with limited resources that, nevertheless, “wanted to be on the cutting edge.” One respondent, reflecting on making concessions to economic necessity said, “Integrated treatment planning, training, and supervision… is a reasonable approach to operating multiple evidence-based practices.” Sites that sustained assertive community treatment, which has structural components integral to its philosophy and functioning, assimilated in whole or part, other practices with commonalities appropriate to their target population. For example, when integrated with assertive community treatment, illness management and recovery, admission criteria were, as one respondent put it, “tightened up because we were treating sicker clients.” At another site, integrated dual disorders treatment eligibility requirements were changed to allow for clients “non-dually diagnosed, because the need was there for intensity of service.” Assertive community treatment accommodated other evidence-based practices by adopting a “modified trans-disciplinary approach, a primary practitioner model with specialists”, or from another program leader, “Components of integrated dual disorders treatment were used in assertive community treatment.”

As a practical matter, some adaptations were driven by client demography. Transitional age groups, youth to adult, were targeted by family psycho-education practices, because the clients’ connections with families were still intact. Supported employment sites added supported education to appeal to young adults, 18–20, who wanted to go to school as well as to work. Under-staffing prompted modification of staff standards, resulted in specialized staff doing general case management work, and replacing individual-level with group-level treatments. Sites responded to mixed caseloads by training all case managers in each practice. Lack of financial assistance and staff support systems forced sites to discontinue fidelity assessment, reduce training, limit the use of psychiatrists’ time, and decrease group supervision hours.

Most adaptations to sustaining evidence-based practices were reported as necessary and many as desirable innovations. Evidence-based practice integration was most often viewed as a positive or neutral accommodation to limited resources. Model changes, such as group instead of individual therapy, were seen as optimizing agency staff time and skills as well as increasing penetration, because more clients could be served with fewer staff. Client and staff preferences frequently drove model changes when the implemented evidence-based practice superseded a similar practice with deeper roots in agency culture. For example, at sites sustaining illness management and recovery, program leaders said they “developed group protocols” and allowed staff to “bring in added materials.” Conversely, illness management and recovery components were added to other recovery group models that were favored by staff and clients. Relaxing or changing eligibility rules was viewed as optimizing penetration and, in some cases, as necessary for sustaining an evidence-based practice in a client environment not precisely suited to it. Adaptations involving decrease in services to clients, because of lack of funding or staffing shortage, were regularly cited as negative.

Discussion

The National Implementing Evidence-Based Practices Project had a high sustainability rate, as nearly four out of five sites were still offering the practice 2 years after the formal implementation phase. Sustaining and non-sustaining sites differed on several key issues. Lack of state funding or supported training, lack of strong commitment from agency leadership, adverse staff or client culture, competing evidence-based practices, and staff turnover were cited as contributing factors, often in combination, for not sustaining a practice. These findings are largely consistent with findings from a study of indicators of innovative mental health practice discontinuation in Ohio (Massati et al. 2007).

State involvement in the form of direct financing and technical assistance was a driving force for sustaining evidence-based practices, but our results also showed some ambivalence about the role of policy makers. State mental health authorities, who hold the purse strings for many evidence-based practices and provide much needed training support, have a vested and legitimate interest in regulating financing, policy adherence, and the effectiveness of practices. Nevertheless, these policies were sometimes viewed as arbitrary and coercive by program leaders.

Many sites reported monitoring client outcomes, the ultimate indicator of success, during the sustainability phase. The ability to document outcomes was cited by Blasinski et al. (2006) as one of the most important factors in sustainability, evident both in client outcomes data from the trial (Unutzer et al. 2002) and the direct experience of the primary care physicians (Levine et al. 2005). Evidence-based practices such as supported employment have reliable and valid outcome measures, which are consistently applied and show good results (Drake et al. 2005). Other practices, having complex outcomes not readily reduced to scales, were less frequently monitored. Notwithstanding the lack of outcome data, program leaders said that they were aware of clients’ progress and attributed good outcomes to the evidence-based practices.

Successful sustainers adapted the evidence-based practices to some extent to suit local conditions. Adaptations varied, were driven by a variety of issues, and were more extensive in sites without state supports. The adaptations were generally viewed as positive innovations that helped to sustain evidence-based practices that were under-financed and under-staffed. As programs matured and site personnel took ownership of their practices, the distinction between model erosion and innovative adaptation became fungible, at least in the views of practice personnel.

However, desirable they may be, the need for local adaptations in order to sustain effective practice models is not generally supported by empirical evidence. Youth violence and substance abuse prevention services research is equivocal about the value of adaptations (Elliot and Mihalic 2004). Even the most justifiable modifications, those sensitive to gender, race and ethnicity, do not consistently alter or sustain practice effectiveness (Gottfredson and Koper 1996; Botvin et al. 1995, 2001). Conversely, mental health services research shows a positive relationship between model fidelity and implementation effectiveness (e.g., Jerrel and Ridgely 1999; McDonnell et al. 1989; McHugo et al. 1999). Prevention research also indicates that effectiveness decreases with erosion of the model over time, but not with adaptations in the form of additions (Battistich et al. 1996; CSAP 2002). As a general principle, modifying core components of a practice is unwise, but adding elements to respond to cultural diversity or other local conditions may be helpful. Because the bottom line is client outcomes, adaptations should be made only while closely monitoring outcomes.

A limitation of our study is the small number of sites that implemented each of the evidence-based practices, which challenges the validity of both within- and cross-practice comparisons. In addition, the small number of non-sustainers, although gratifying, makes conclusions about reasons for failure, or decisions to drop the practice, provisional. Because we expected that there would be many more sustainers, the survey questions were designed mostly for them. Although a summary question was asked to uncover unsuspected factors, the telephone survey was not comprehensive and left open questions for further investigation. Most importantly, the retrospective accounts that were obtained in this study must be viewed with caution due to problems of self-report, such as post hoc justification and rationalization. Our use of multiple respondents per site and triangulation of the results reduces the likelihood of such systematic biases.

Several areas for future research were raised by the current study. First, state policies for supporting evidence-based practices should be explored further. Although leveraging federal Medicaid dollars is an important consideration, these funds do not offset the entire cost of implementing a new practice. Understanding the various means used by states to support evidence-based practices would provide a compendium of strategies for future reference (for e.g., see Isett et al. 2007). Second, some sites sustained the evidence-based practice despite financial and personnel constraints. To what extent do agency-level strategies for implementing and sustaining a practice mitigate negative factors? Third, the effect of adaptations on practice fidelity and client outcomes should be investigated. What are acceptable model adaptations? Should core components be identified? When should program leaders focus on model fidelity versus client outcomes in making adaptations?

The National Implementing Evidence-Bases Practices Project showed that five psychosocial practices could be implemented with high fidelity in routine mental health settings over a 2-year period (McHugo et al. 2007). The current study showed that nearly 80% of the sites sustained the practices for an additional 2 years. This study provided some of the first evidence concerning the factors that help or hinder sustainability. The findings should enable states and sites more effectively to plan for and to support the implementation and continued growth of evidence-based practices. If mental health services are to attain the standards of care for people with serious mental illness that have been enumerated in recent reports (Institute of Medicine of the National Academies 2005; US Department of Health and Human Services 2003), policy makers, administrators, and practitioners need practical advice on implementation strategies and barriers in order to succeed. Evidence of practice effectiveness and rhetoric from leadership are not sufficient, for it is the everyday realities of financing, staffing, and leadership that determine implementation outcomes. The current study contributes to the growth of implementation research in mental health and helps to lead the way to a future where effective services are accessible to those in need.