Introduction

During the last decade, multiple sources have documented that people with psychiatric disabilities have difficulty accessing mental health treatment (Wang et al. 2002; Kessler et al. 1996) and once they do, they infrequently receive services with effectiveness demonstrated through research (US Department of Health and Human Services (DHHHS) 1999; President’s New Freedom Commission on Mental Health 2003). The schizophrenia patient outcome research team (PORT) revealed that people with schizophrenia were unlikely to receive effective services (Lehman et al. 1998). They found, for example, that only 22% of consumers in outpatient programs received any vocational services. A recent study found that of unemployed patients, none received any form of vocational rehabilitation (West et al. 2005). The situation is parallel with other evidence-based practices: medication service, family psychoeducation, integrated dual diagnosis treatment, etc.

A lack of knowledge about implementation processes acts as one barrier to faithful dissemination (Torrey and Gorman 2005). The National Evidence-Based Practice (EBP) Implementation Project was mounted to explore whether EBP’s can be implemented in routine mental health service settings and to discover the facilitating conditions, barriers, and strategies that affected implementation. The project involved 49 sites in eight states. All but one state sought to implement two of the five targeted practices: supported employment, integrated dual diagnosis treatment, family psychoeducation, illness self-management, and assertive community treatment. The project’s active stage intervention lasted 2 years with the first year being devoted to implementation and the second year on sustaining the practice.

This article reports on a qualitative study of barriers to EBP implementation in one Midwestern state that sought to implement supported employment (SE) and integrated dual diagnosis treatment (IDDT). Supported employment EBP is based on six principles: (1) eligibility in based on consumer choice; (2) employment services are integrated with treatment; (3) competitive employment is the goal; (4) rapid job search; (5) follow-along supports are continuous; (6) consumer preferences direct the work. Integrated Dual Diagnosis Treatment simultaneously treats the mental health and substance abuse disorders using a stage-wise approach with motivational interviewing, substance abuse counseling, self-help and other support services (Substance Abuse and Mental Health Services Administration, SAMHSA 2004).

Each practice was implemented at three community mental health centers as part of their Community Support Services. Although all six sites in the state reached a high level of fidelity, it was not an easy task. This paper describes the major challenges experienced by the agencies involved. Subsequent reports will explore facilitating conditions and strategies that benefited implementation.

Methods

The evaluation of the EBP Implementation Project was carried out in Fall 2005 through Spring 2006. In contrast to the evaluation of the National EBP Implementation Project, this evaluation focused solely on the six sites in one state implementing IDDT or SE EBP’s. The research was based on the naturalistic paradigm of Lincoln and Guba (1985).

Sample

After making a commitment to participate in the national project, the state mental health authority issued a request for and gathered applications from community mental health centers (CMHC) who were interested in either implementing Integrated Dual Diagnosis Treatment or Supported Employment. The Commissioner of Mental Health selected five mental health centers from those applying to participate based on the guidelines presented by the National Project oversight committee (one site offered both EBPs). Guidelines for selection included a mix of rural and urban sites and commitment of agency leadership. Each CMHC designated a team to implement each EBP. The teams consisted of a program leader (in some case, two supervisors shared the role) and 3–6 direct service staff. Each site created a Leadership Team comprised of the CMHC executive director, Community Support Services director, program leader, consumers, families, and a state representative. These meetings were initially facilitated by the consultant and trainer (CAT) assigned to the site. The CAT was the principle support for implementation. The Leadership Team had overall oversight of the project at each site and was the central decision-maker.

Data Collection

Implementation data were collected over 2 years by implementation monitors and trainers during site visits, trainings, leadership meetings, team meetings, shadowing workers, and through interviews with consumers, direct service workers, supervisors and administrators. Notes reflecting implementation efforts came from monthly Leadership Team meeting minutes, semi-monthly trainer contact notes during the implementation phase (bi-monthly during the sustaining phase), and implementation monitor site visit summaries which occurred monthly through the implementation phase. Implementation monitors and trainers met regularly to share observations, notes and impressions to help ensure consistency with data recording and accuracy. Formal fidelity reviews were conducted every 6 months by the implementation monitor and CAT during the implementation and sustaining phase of the project. These reviews assessed the agencies’ degree of adherence to the particular practice’s standards (Bond et al. 2000). The supported employment scale had been previously validated (Bond et al. 2001) and the integrated dual diagnosis treatment scale was developed by the national project investigators and the developers of the practice. The fidelity scales can be obtained at http://mentalhealth.samhsa.gov/cmhs/communitysupport/toolkits/illness/. A full report on project fidelity has recently been published (McHugo et al. 2007). Ratings were established in partnership between the implementation monitor and trainer.

Data Analysis

The first step was to organize the raw narrative data for each of the six sites into three categories: Facilitating conditions, strategies, and barriers. Facilitators identified evidence of factors that helped EBP implementation but were not intentionally developed as a result of implementation. Strategies identified evidence of intentional actions seeking to help EBP implementation. Barriers identified evidence of actions seeking to hinder or the intentional failure to act in support of EBP implementation. The second step was to identify the major themes within this data that helped or hindered implementation. The national project researchers defined theme as a thread of activity or condition that was salient, prominent, conspicuous, or non-ignorable. Key stakeholders involved with the theme were also identified. Additional sources of data available for this study included the six final individual site reports written by the senior project researcher and staff for each state. Discussion with the implementation monitor and trainers and, in some cases, agency program leaders were used to clarify “meaning” of some of the data.

In order to enhance trustworthiness (Lincoln and Guba 1985) (inter-rater reliability), all implementation monitors for the national project participated in monthly conference calls throughout the project and attended annual meetings to learn, discuss and clarify any process or technical issue that arose. This technical assistance was overseen by the project’s coordinating body, Psychiatric Research Center at Dartmouth College (PRC). To enhance reliability, the PRC requested that all participants take turns in submitting examples of collected data monthly. These would then be distributed to the rest of the implementation monitors, coded independently and then reviewed by PRC for reliability. Upon completion of the project and when coding was complete, Atlas.ti 5.0 software (Muhr and Friese 2004) was used to compile the necessary data into structured formats to be used in a final report for each site.

Locality Specific Re-Coding

These themes were re-analyzed inductively for this project by a primary team of three. At this stage of analysis, investigators who were not involved in the prior deductive analysis examined the data in the six site reports without preconceived notions of what they would find. This was accomplished by removing the contextual label identified for the theme. They allowed themes to emerge from the data which advanced new concepts not previously theorized. The goal was to develop a set of categories that adequately organized and accounted for the data in the local context of it. The process was iterative where deeper analysis uncovered flaws and inadequacies in a category scheme demanding reformulations of the categories. In this way, the data continued to drive the analysis.

As a new coding guide was being developed, open coding followed by axial coding (Strauss and Corbin 1990) led to the emergence of categories establishing a conceptual set of codes for continued analysis. Once a tentative coding guide was determined, two analysts independently coded the data contained from one site. In conjunction with the primary investigator, the results were compared. Through explanation and collaboration, the codes were modified and a final coding guide was established. The final coding guide was then applied to the data of all six sites. Part of the process involved going back to the raw data or the trainers to clarify, inform, and elaborate on specific data contained in the display that appeared ambiguous or lacking of context.

The final thematic categories that included barriers were:

  • Behavior of Front-Line Supervisors

  • Behavior of Front-Line Practitioners

  • Behavior of Intra-Agency Member

  • External Stakeholder Involvement

  • Funding

Findings

While five thematic categories that included barriers were identified, three emerged as most powerfully influencing implementation of the EBP across at least five sites. This was determined based on frequency of mention across sites, the difficulty in overcoming the barrier, and the severity of consequences caused by the barrier. The judgments were made by the four person research team in consultation with the two trainers. Although funding interfered, all six sites found ways to overcome it. Similarly, the National EBP project placed primacy on consumer and family involvement as external stakeholders. Despite concerted efforts at all six sites, this was not achieved but did not seem to influence implementation. The data on barriers that profoundly influence implementation coalesced around the roles, responsibilities, and behaviors of three clusters of personnel: Front-line supervisors, practitioners, and other professional staff within the agency. It is important to note that these barriers endured for 6–9 months of the project depending on the sites. Each of the sites were able to overcome these barriers and achieve high fidelity with the particular practice. The strategies used to overcome these barriers will be presented in a second paper.

Front-Line Supervisors

At each site, the front-line supervisors of pilot teams were designated the program leaders. In every instance, these supervisors and their practice were found seriously lacking and probably was the single greatest barrier to implementation. While supervisory practice deficits were numerous, there were several that were common across sites.

First, at most sites supervisors did not set expectations (EBP-related or otherwise). Practitioners developed their own sense of how to do their jobs. Although clearly understanding the administrative requirements for billable hours and documentation, consultation with supervisors around service delivery only occurred when confronted with difficulties. There were few prescriptions or structure to their practice. One supervisor saw himself as more of a “buddy” than a “boss”. They seemed to go out of their way to avoid conflict with their staff. At one SE site, practitioner rebellion was answered by placing the project and training on hold for 3 months because “the team is upset” (at which time a new program leader/supervisor was hired and some practitioners were transferred or resigned).

There were few examples in the data of workers receiving meaningful feedback on their practice. In fact, few supervisors had but the most superficial knowledge of how their workers practiced. This “laissez faire” form of leadership has been found ineffective in producing high rates of client outcomes (Corrigan et al. 2000). The introduction of the EBP’s with their specific practice guidelines required a new set of behaviors by supervisors who had great difficulty making the transformation. Each attempt to set and enforce EBP expectations was based largely on coaxing, persuasion, and the fervent hope that training and time would eventually produce conforming behavior by direct service workers. There were rarely any consequences for poor performance.

In the EBP project, team meetings were to be dominated by application of the practices to the myriad of ideosyncratic consumer situations. It was to be a primary mechanism for improving the practice of team members. A specific model for doing this, group supervision, was introduced to each agency (Rapp and Goscha 2006). At the onset, team meetings were not well run and in most cases, the initial set of supervisors never mastered the skills of group supervision. In over half the sites, team meetings were devoted to reviewing administrative matters or brief discussions of consumer crises. The CAT’s notes described this incident:

An employment specialist asked to speak about a consumer issue and the supervisor said “No, not now, I’m on a roll”. The supervisor had been reading administrative announcements from a handout that all employment specialists had received.

In other cases, the meetings were unfocused discussions often dominated by one person that never arrived at a conclusion or next step. In some meetings, direct care staff would do their paperwork or have side conversations while a case was being discussed, and in others attendance was uneven. At one site, the supervisor never mastered the skill of following a disciplined format for case reviews despite training, consultation and modeling.

Second, according to interviews and site observations two supervisors sabotaged the project in several ways. These supervisors would not follow-through on decisions/instructions emanating from the leadership team or the CAT even when they seemed to agree and endorse the decision. For example, they would follow the guidelines for team meetings when the CAT was present, but otherwise would ignore them. In another case, a supervisor refused to set EBP expectations nor did he set up opportunities for learning or practice and created a team environment of low enthusiasm for the EBP. He did not require practitioners to discuss stage specific interventions and would shorten supervision time because “nothing is going on”. At this site, supervision scores as measured by the General Organizational Index (Drake et al. 2002) were on or below two (on a five point scale) until the 18 month review period when a new program leader was hired and trained in IDDT.

The reasons for these situations were varied. In many cases, supervisors did not know the EBP skills and felt inadequate to supervise the practice. This was true in all IDDT sites. In the SE sites, job development was new to all supervisors. In some cases, supervisors were afraid of exercising their authority and had been imbedded in agencies that did not demand it. In most cases, these supervisors were not involved in the decision to be undertake the EBP project or only superficially so. In three sites, the supervisor also had significant responsibilities unrelated to the particular team’s efforts to implement the EBP. In one case, the supervisor also oversaw therapists and the day treatment program; in another, the supervisor was also the Community Support Service Director; in the third, the supervisor also oversaw the COMPEER (a program linking consumers with community volunteers for companionship) and day treatment program. In these situations, their attention was diffused and was susceptible to distractions to EBP implementation.

It should be noted that five out of six project sites enjoyed the support or active championing of the EBP by upper management. In the one exception, the executive director agreed to do the project, delegated overall responsibility for its operation to the clinical director and was never seen again. The clinical director’s contribution seemed monopolized with protecting the agency’s budget. A sample of statements that occurred in leadership meetings included:

  • I’m quite concerned about the cost of this plan

  • Consumers should receive assistance from staff with getting to groups. This will increase billable hours.

  • I want to avoid having the staff to come to one site for meetings because it eats up too much time in travel.

  • A new policy on treatment plans was defended: “The advantage of this plan include reduced travel expenses and reduced overhead expenses”.

The clinical director did not agree with all the practices imbedded in the IDDT model, had very little commitment to implementation and routinely refused to alter policies (reducing billable hours requirements to facilitate training) or to procure needed resources (attendant care position to help with medications).

Front-Line Practitioners

Data in this barrier category were defined as “Behavior observed of any agency members that have direct responsibility for the implementation of the EBP, indicating doubt or resistance toward consumer recovery or the EBP practice”. Practitioner’s resistance to the new practices was present in all six sites and formidable at five of the sites. The nature of the resistance varied from profound indifference to open hostility. Others were passive-aggressive, seeming to cooperate when the CAT was present then sabotaging implementation at other times. Often power struggles ensued between practitioner and program leaders. As one practitioner expressed:

“You are saying that we can’t work with anyone unless they want to work and then we have to get them a job in one month. We are doing bad work. Nothing we are doing is right. When we finish the project, can we go back to doing things the way we are now?

This resistance occurred in the fertile ground provided by the “laissez faire” management style of the supervisors in five of the six sites. It should be noted that this resistance was not universal. In most of these sites, there were practitioners who were enthused about the EBP and a few even acted as champions of it. The oppositional practitioners, however, created a hostile work environment that made it difficult for other practitioners to express excitement about the EBP or to actively engage.

In virtually all cases, the initial resistance was viewed by agency leaders and the CATS as emanating from the practitioners’ assumptions and lack of knowledge about the work that was contrary to EBP practice. For example, many SE practitioners believed that pre-vocational activities and volunteer work contributed to consumers becoming employed; that in areas of high unemployment employers were not likely to hire consumers; that symptoms needed to be controlled, substance abuse avoided, and hygiene attended to before a consumer could get a job.

“I’ve been doing this for so many years—I disagree with not doing groups and [placing people in] volunteer [positions]. [This change] is not going to last. It’s a big change of who we are… “Volunteering is the root of who we are. I created contract work for people not ready to work. They cleaned group homes, parks, movie theaters…For clients not ready for competitive employment they need to volunteer and to participate in groups to get ready.”

The overwhelming agency belief was with more information, training, and time these myths and the resultant resistance by staff would wane. Much time, often 6 months or more, was committed to this strategy before it was deemed a failure. In only two situations did practitioners who opposed the practice change their perceptions through training, etc. In the majority of cases, resistant practitioners either left their position or the agency reassigned them.

Intra-Agency Synergy

For a given EBP to reach high fidelity and produce the desired level of consumer outcomes requires agency personnel, beyond the staff targeted for EBP implementation (e.g., supported employment specialist), to practice in certain ways. In both EBP projects reported here, psychiatrists and other medical personnel, were important. Case managers and their supervisors in the supported employment sites were particularly important and substance abuse counselors were necessary for IDDT sites.

Psychiatrists and other medical personnel were important to the implementation of both evidence-based practices yet in four of the six sites, their practice interfered with successful implementation. At one site, the psychiatrist’s prescribing practices were contraindicated by IDDT practice and research evidence. He routinely prescribed benzodiazines for clients who had substance abuse issues despite the dangers of abuse and further addiction, and told at least one consumer that drinking in “small amounts now and then” was OK. He was also unavailable for team meetings or individual consultations with staff. Of greater concern was that when presented with information concerning these issues, he remained steadfast to refusing to change.

In all the SE sites, psychiatrists, nurses and clinicians rarely discussed work with consumers nor referred consumers to SE services. In general, the SE program was operating in a sea of indifference or hostility to work among other agency personnel. In fact, they often discouraged consumer interest in work because of beliefs such as: (1) work will increase stress and exacerbate symptoms; (2) the clients need their energy focused on “more fundamental issues”. One psychiatrist would not grant permission to work when medication regimens were changed until the client was “stabilized” on the regimen (90 days) despite the consumer’s desire to work.

Case managers were to be the principle source of referrals to the SE program at all three sites. Case managers, however, informally screened out consumers based on their beliefs about the consumer’s ability to succeed or their belief that symptoms would increase due to stress caused by employment. They had little or no knowledge of the role of employment in recovery. They did not understand how work could improve hygiene or decrease substance abuse. Rather, they assumed adequate performance in these areas must pre-date employment. Furthermore, some case managers (and other agency personnel) could not accept alternative perspectives even when presented with considerable evidence.

The integration of SE services into case management teams was difficult. While each site moved quickly to assign one SE specialist to each team, actual integration took longer. CM team leaders, case managers and even SE specialists were not sure how they were to act during these meetings or what role they were to play. At first, they just attended. The team meetings did not allow for participation of SE specialists, and work was rarely mentioned as teams reviewed particular case situations.

The integration of substance abuse counselors was critical to IDDT implementation. In two of the three IDDT sites, there were significant problems. At one site there were no SA counselors within the agency. This was a barrier throughout the implementation. The CSS program created an IDDT Liaison position, but this position served all of CSS (650 consumers). This made it impossible for the IDDT Liaison to become integrated into the IDDT team (couldn’t attend very many team meetings). There was also limited availability for individual SA counseling for IDDT clients.

At another site, the IDDT practitioners did not have contact with the agency’s substance abuse counselors who were housed in a different building across town. These counselors were not providing services to dually diagnosed consumers. One team member did assume the role of “unofficial substance abuse specialist” by virtue of facilitating a dual diagnosis group based on the “12 steps”. Other practitioners looked to him for dual diagnosis expertise. The introduction of the IDDT-EBP threatened his role of “expert” and led him to frequently question and object to key elements of the practice. His attitude and behavior contributed to “doubts” about the practice by other practitioners.

Discussion

Great care was devoted to ensuring the internal validity of the findings. As in much qualitative research, external validity is severely limited by the boundaries of the study. The study involved only six sites within one state implementing two specific EBP’s. It is likely that other states confronted a different configuration of obstacles. It is also possible that different EBP practices conjure different obstacles. However, it also seems likely that barriers emanating from supervisors, direct service staff and other agency professionals would be common occurrences (Torrey et al. 2001).

Funding and top-level administrative support are the most frequently cited elements in making program improvements and implementing new interventions (Drake et al. 2001; Goldman et al. 2001). The findings of this study suggest that these are probably necessary but insufficient conditions for successful implementation. In this study, top-level administrative support was present in five of the six sites and the few funding barriers were quickly resolved yet the EBP implementation still progressed slowly at first.

The findings suggest that implementation of EBP is a complex undertaking requiring varied groups of people to behave in ways that are different from current practice. To successfully implement EBP, there needs to be a synergy operating that involves upper level administration, program leaders/supervisors, direct service workers, and related professionals within the agency. In some cases, external stakeholders need to also be part of the synergy. This synergy is both powerful and fragile. When in place, the EBP is implemented well; yet if one party is out of synchronicity, performance can lag.

Synergy emerges when all the key players are fulfilling their necessary role and meeting expectations. The barriers to implementation were often traced back to the lack of expectations or their enforcement by one or more groups. At the team/service delivery level, the culture of low or no expectations was present in five of the six sites at the beginning of the project. Direct service workers and supervisors went about their jobs as they saw fit. There was little discipline to their practice. In fact, supervisors were shockingly unaware of the “actual” practice of their teams (e.g., how they engaged with consumers, how they interacted with employers, what methods and strategies they employed to help consumer reach their goals). While most people worked hard, there seemed to be few demands placed on workers beyond that (except insuring paperwork and billable hours standards were met). The vacuum caused by the lack of expectations led to highly ideosyncratic and reactive practice.

The difficulty of implementing EBP in an environment that has few expectations was manifest when practitioners resisted implementation. The preferred solution to the resistance was providing more information, training, and at times, individual conversations. There was a universal and profound avoidance by the supervisors to setting expectations and demanding efforts towards compliance. At times, this was due to ambivalence by the supervisor to all of the EBP elements or fear of not being “liked”. What became clear was that the project was asking supervisors (e.g., set and enforce expectations) and direct service workers to perform behaviors that they had rarely done before within an organizational culture that never demanded it. Direct service workers were given de facto authority to reject the EBP.

While the successful implementation of EBP’s requires the constructive involvement of a host of players, the front-line supervisors (project leaders) were in many ways the most critical. Structural changes were the province of upper management, but making those structural changes actually work for clients and the implementation of practice skills fell most directly on the front-line supervisors. For example, upper management created interdisciplinary teams for IDDT and assigned an employment specialist to each case management team for SE. But it was the front-line supervisors that had to ensure that these contributed to the practice. Resistance by front-line practitioners to these changes should be addressed by the supervisor. The initial set of supervisors in Kansas was passive custodians of their teams rather than leaders. Two SE sites were “getting nowhere” until the supervisor was replaced by a more committed and skilled person.

The National Project astutely identified that EBP implementation would require the constructive involvement of a variety of key actors/stakeholders. Attention was paid to and materials developed for agency administrators, families, consumers and state mental health authorities. In this state, therapists, medical staff and case managers emerged as critical actors. For SE, therapists, medical staff, and case managers often discouraged or “prohibited” people from pursuing work. These people were the source from which the SE programs should receive most of their referrals. For IDDT, medical staff and their prescribing practices, and substance abuse staff were particularly important.

The findings of this study suggest a framework for investigating and understanding barriers to EBP implementation. The approach would identify the key participants and specify the behavior necessary for implementation. Fidelity guides already specify the behavior of front-line staff and structural elements which are usually the responsibility of administration to make. Largely missing is the expectations of supervisors and the necessary role of other agency professionals. Once these guides are developed, implementation research could more completely understand the process.