Introduction

Since the early 2000s, mental health service users, practitioners, family advocates, and researchers have urged state and federal policy makers to increase the availability of evidence-based practices (EBPs) in public behavioral health systems (Drake et al. 2001). A recent study of the spread of EBPs between 2001 and 2012 and the nature and extent of state investments in EBPs found that while many states have invested in infrastructure to encourage and support EBP implementation, the impact of these efforts has been relatively small in terms of the number of service users served. In fact, EBP adoption by states showed flat or declining temporal trends (Bruns et al. 2016).

In this study we examined the national availability of evidence-based supported employment, or Individual Placement and Support (IPS). IPS is an evidence-based practice that helps people with serious mental illness achieve competitive employment (Marshall et al. 2014). Evidence-based employment services are important for people with serious mental illness; approximately two-thirds of people in the U.S. community mental health system have reported that employment is one of their goals (Bond and Drake 2014). Despite the desire to work, only about 15% are employed (Rosenheck et al. 2006; Salkever et al. 2007). Across studies, IPS improves competitive employment outcomes when compared to other vocational services, and IPS service users demonstrate substantial employment rates for years following the end of services (Drake et al. 2016).

Given the promise of IPS, it is important to understand the prevalence and quality of IPS nationwide. The Substance Abuse and Mental Health Services Administration (SAMHSA) has required state behavioral health agencies to prepare an annual report known as the Annual Mental Health Block Grant Implementation Report (Lutterman 2015); these reports have indicated which states provide supported employment services, and the data are included in SAMHSA’s National Outcome Measure on Provision of Evidence-based Services. According to this annual survey, 41 states provided supported employment for people with serious mental illness in 2014, including 17 states reporting that they assessed supported employment programs for fidelity. However, these reports did not clearly define “supported employment,” making comparison across states difficult. Furthermore, state agency web sites and other public documents often have not been current or precise (Ganju 2003).

In the last 15 years, several factors have impacted the growth of IPS. In this paper, we focus on discerning the significance of factors previously identified within the research literature (Becker et al. 2014; Bond et al. 2016; Drake et al. 2016; Ganju 2003). One such factor has been the Supreme Court’s 1999 Olmstead decision. In 2009, the U.S. Department of Justice launched an effort to enforce the Supreme Court’s decision in Olmstead v. L.C., requiring states to eliminate unnecessary segregation of persons with disabilities and ensure that persons with disabilities should have the opportunity to live like people without disabilities and to receive services in the most integrated setting appropriate to their needs (Burnim 2015). As part of furthering community integration, Olmstead settlements in numerous states have included the mandated expansion of supported employment services for people with serious mental illness (Drake et al. 2016).

A second clear factor has been the development of the IPS Learning Community. Since 2002, state and regional leaders in the IPS Learning Community have collaborated on disseminating, implementing, and sustaining IPS (Becker et al. 2014; Bond et al. 2016). The learning community has a decentralized structure with two tiers: the national leadership team at the IPS Employment Center located in Lebanon, NH, and leaders from state and regional agencies who have planned, implemented, and monitored IPS programs within their states. The IPS Employment Center has provided training and consultation to state programs regarding funding, policies and procedures around IPS, and tracking outcomes. As part of their participation in the learning community, state leaders in behavioral health and vocational rehabilitation agencies from 20 U.S. states and the District of Columbia have agreed to pursue several strategies adopted by the learning community to promote the adoption and implementation of IPS services. These are implementation strategies that state leaders can pursue to enhance EBP services (Finnerty et al. 2009). Three of these strategies, or key quality indicators, are as follows:

  • Collaboration between state behavioral health and vocational rehabilitation agencies: State implementation teams, including members of the state vocational rehabilitation and behavioral health agencies, have been instrumental in improving collaboration between state agencies involved in providing employment services to people with serious mental illness in learning community states (Drake et al. 2006).

  • Comprehensive technical assistance and training for supported employment programs: The IPS Learning Community facilitates multiple training opportunities for state leaders and IPS trainers, and fosters broader conversation about strategies for building technical assistance and training infrastructure within the state. A number of states have established evidence-based technical assistance centers to promote the development of EBPs (Salyers et al. 2007).

  • Regular, independent fidelity monitoring of IPS programs: state leaders use a standardized fidelity scale to monitor adherence to the IPS model, and regular fidelity reviews conducted by independent assessors are a requirement for membership in the IPS Learning Community. Ongoing fidelity monitoring is associated with a variety of positive program-level outcomes, such as greater staff retention (Aarons et al. 2009).

The purpose of this study was to describe the prevalence and quality of IPS nationwide. We documented the national spread of IPS, estimating the number of states implementing IPS and the number of programs per 1,000,000 people in each state. Since learning community states are advised to initially implement only about three to six IPS programs and expand gradually in subsequent years (Becker et al. 2014), we expected to find lower penetration for states joining more recently than those participating for a longer time. We then evaluated the quality of IPS programs based on three quality indicators that have played an important role in the success of the IPS Learning Community: collaboration between state behavioral health and vocational rehabilitation agencies, training and technical assistance, and independent fidelity monitoring. We also evaluated the impact of membership in the IPS Learning Community and the presence of Olmstead settlements on IPS program penetration and quality across the United States.

Methods

This study was part of a larger national survey of leaders in state behavioral health and vocational rehabilitation agencies on current and promising practices in providing employment services to people with serious mental illness. The overall interview used a semi-structured protocol. Our primary goal was to understand the type, prevalence, and quality of employment services available in each state. The Dartmouth College institutional review board, which followed the principles outlined in the Declaration of Helsinki, approved the study. Respondents gave verbal consent for participation.

Participants

We attempted to interview agency officials representing 52 jurisdictions: each of the 50 states, the District of Columbia, and Alameda County, California (a member of the learning community separate from the rest of California). For convenience, we will refer to all of these jurisdictions as states. Our plan was to interview one person from the behavioral health agency and one person from the vocational rehabilitation agency in each state.

To identify respondents, we contacted state behavioral health and vocational rehabilitation agency officials by email, using a wide variety of resources: current state liaisons to the IPS Learning Community, participant lists from prior IPS Center trainings, the National Association of State Mental Health Program Directors’ list of state commissioners, the SAMHSA directory of State Mental Health Authorities, the Council of State Administrators of Vocational Rehabilitation’s list of state vocational rehabilitation directors, and information on state behavioral health and vocational rehabilitation agency websites.

Once we made contact with a state agency, we asked to be referred to the most appropriate person at the agency. In behavioral health agencies, we sought representatives responsible for oversight of supported employment, evidence-based practices, or community services. In vocational rehabilitation agencies, we spoke with representatives in charge of supported employment, services for people with serious mental illness, or field services.

Sample

We completed full interviews with 107 respondents: 52 representatives from 48 state behavioral health agencies and 55 representatives from 51 state vocational rehabilitation agencies. Eight of these interviews were conducted with two respondents. In three of the states that were members of the IPS Learning Community, state IPS trainers participated in interviews with the behavioral health representative. One state joined the learning community in 2016 after we had interviewed representatives from that state’s agencies, and we therefore classified that state as a non-learning community state in this report.

One state vocational rehabilitation agency did not respond to multiple requests to participate in the survey. In three states, we spoke with leaders in the behavioral health agency and corresponded via email but did not complete full interviews; in two of these states representatives indicated that their agency did not provide direct employment services to people with serious mental illness, and declined to participate. In the remaining state, a representative provided answers to some questions via email.

California’s behavioral health agency did not have authority over the state’s services; it primarily served to distribute state and federal funds to the state’s 58 counties, which have sole oversight over local services. We did complete an interview with two representatives from California’s vocational rehabilitation agency, who reported that vocational rehabilitation did not fund any IPS programs in the state nor were they familiar with the availability of IPS services statewide. After several unsuccessful attempts to identify a state behavioral health official who was well-informed about employment services in California as a whole, we opted to omit California from the survey (except for Alameda County, which joined the IPS Learning Community in 2012).

Interview Procedures

The interview protocol was structured into a series of open-ended and closed-ended questions. All interviews were conducted between August 2015 and March 2016 and averaged 45 minutes to one hour. The first author, who conducted all interviews, had not met any of the respondents prior to the interview. A second interviewer, who had met a minority of the respondents at previous conferences, sometimes joined the interviews. The interviewees knew the interviewers as IPS researchers and employees of the national IPS Employment Center. During the interview, the first author took detailed notes and completed the interview template. After interview completion, the first author typed up notes and entered primary outcome variables into a spreadsheet.

Measures

  1. 1.

    IPS availability. By definition, all 19 states within the learning community at the time of the interviews offered IPS. To determine if other states offered IPS, the interview protocol suggested several alternative terms sometimes used to refer to IPS, including the “SAMHSA Toolkit” supported employment, evidence-based supported employment, and the “Dartmouth” model. The interviewer probed to ensure that the respondent understood the terminology.

  2. 2.

    Number of IPS programs. To determine the count of IPS programs nationwide, we asked each respondent to report the number of IPS programs in their state. In 12 (31%) of 38 states with IPS programs, both respondents reported the same number of IPS programs. In 6 (15%) states, both respondents deferred to statistics included in the state’s quarterly employment report compiled from data that each IPS program submits to the IPS Employment Center, in order to obtain the most accurate count of number of sites. In 20 (51%) of states, the vocational rehabilitation respondent deferred to the number of programs reported by the behavioral health respondent in their state.

  3. 3.

    Number of IPS programs per 1,000,000 people. To assess the penetration of IPS programs relative to the state’s population (as defined by the 2010 U.S. Census), we defined the IPS program penetration index by the ratio of the number of IPS programs per 1,000,000 people.

  4. 4.

    Training and technical assistance. To assess the level of training and technical assistance available to IPS programs in each state, we asked respondents whether or not their state (a) operated a technical assistance center that provided comprehensive training to IPS programs, (b) employed a state trainer for IPS programs, or (c) offered other types of trainings (e.g. benefits planning or serving people with dual diagnoses) that were relevant to all employment programs, but not specific to IPS. Because relatively few states reported operating technical assistance centers, we collapsed the first and second categories for the purpose of quantitative analysis.

  5. 5.

    Fidelity monitoring. When respondents reported that their state agency operated IPS programs, we asked whether or not those programs received regular fidelity reviews from independent assessors using a standardized IPS fidelity scale (Bond et al. 1997, 2012).

  6. 6.

    Interagency collaboration. To gauge the level of collaboration between state behavioral health and vocational rehabilitation agencies on employment services for people with serious mental illness, the interviewer transcribed detailed notes of responses to open-ended questions regarding collaboration. Responses included descriptions of the two agencies working together on developing specific employment programs for people with serious mental illness, state agencies sharing funding for employment programs (e.g., the behavioral health agency funding long-term supports for people with serious mental illness receiving vocational rehabilitation services), and state vocational rehabilitation agencies streamlining the eligibility process for people already served by the behavioral health agency. Two independent raters coded responses into one of two categories (direct collaboration vs. limited or no collaboration). In states with a limited relationship, neither respondent reported concrete examples of the two agencies working together. The two coders initially agreed on codes for 43 (84%) of 51 states. After further discussion, the two raters reached consensus on the coding of these discrepancies. We excluded two states where we only interviewed one respondent.

  7. 7.

    Department of Justice settlements. We asked each respondent whether or not their state agency had ever operated under an Olmstead or similar settlement that impacted the provision of employment services for people with serious mental illness. In states where one or more respondents indicated that a settlement had impacted services, we corroborated with outside data sources found on either the U.S. Department of Justice website (https://www.ada.gov/olmstead/olmstead_enforcement.html) or a state government website. The U.S. Department of Justice maintains a comprehensive list of Olmstead enforcement in each state, and at least one respondent in each state with an active and relevant Olmstead settlement was aware of the settlement. All of the referenced settlements mandated the expansion of community-based services to people with serious mental illness who were residing in institutions or in danger of being institutionalized. Seven settlements mandated the implementation or expansion of supported employment services; four settlements specifically referenced the expansion of the Dartmouth supported employment model, the SAMHSA toolkit, or evidence-based supported employment.

Data Analysis

We calculated descriptive statistics on the number of reported IPS programs, penetration rate, and three program quality indicators: interagency collaboration, training and technical assistance, and regular, independent fidelity monitoring.

We hypothesized that IPS Learning Community membership and the presence of Olmstead or similar settlements would correspond with greater IPS program penetration and quality. We used t tests to compare groups on IPS program penetration.

We also examined cross-tabulations of the three program quality indicators for both hypothesized predictors. We used chi square analyses to evaluate our hypotheses regarding differences between learning community and non-learning community states on the three program quality indicators.

Results

Prevalence and Penetration of IPS Nationwide

In 24 (47%) states, respondents from both state agencies reported that their state offered IPS. In 14 (27%) states, the behavioral health respondent reported that their agency offered IPS, while the vocational rehabilitation respondent referred the interviewer to the behavioral health representative. In the remaining 13 (25%) states, both respondents reported that their state did not offer IPS.

Overall, representatives from 38 (75%) of 51 states reported at least one IPS program. The 38 states with IPS services averaged 13.7 programs (SD = 16.0) for a total of 523 IPS programs nationwide. The number of IPS programs per 1,000,000 people in each state ranged from 0.05 to 16.62 (M = 3.61, SD = 3.62).

Quality of IPS Nationwide

Among the 38 states that reported having at least one IPS program, 23 (61%) states had either established a technical assistance center providing comprehensive services to IPS programs or employed at least one full-time IPS trainer. Fifteen (39%) states provided regular trainings on topics of relevance to employment specialists (for example, benefits counseling and motivational interviewing), but did not provide IPS-specific training.

Twenty-six (68%) states reported that their IPS programs received regular fidelity monitoring from independent assessors. Two (5%) conducted regular self-assessments of IPS fidelity. Ten (26%) reported that their IPS programs did not currently receive any fidelity reviews; one of these states was in the process of establishing regular, independent fidelity reviews, and two had previously conducted fidelity assessments, but had discontinued.

Twenty-one (55%) states with IPS programs reported direct collaboration between state behavioral health and vocational rehabilitation agencies on employment services for people with serious mental illness. Fifteen (40%) reported limited or no collaboration; two (5%) did not report.

Predictors of IPS Penetration and Quality

Learning Community Membership

The 38 states that reported IPS programs operating in their state split evenly between those that were participating in the learning community and those that were not, as shown in Table 1. Nineteen learning community states reported a total of 257 IPS programs; 19 non-learning community states with IPS programs reported a total of 266 IPS programs operating in their states. Five (26%) states, each with at least 20 IPS programs, accounted for 192 (72%) of the IPS programs outside of the learning community.

Table 1 IPS in learning community and non-learning community states
Full size table

Comparisons between learning community and non-learning community states on penetration and quality indicators are shown in Table 1. IPS program penetration rates did not differ between learning community states and non-learning community states with IPS. However, states within the IPS Learning Community were much more likely to have established a technical assistance center or employ at least one full-time IPS trainer. Learning community member-states were also much more likely to report that their IPS programs received regular fidelity monitoring from independent assessors and to offer examples of direct collaboration between the behavioral health and vocational rehabilitation agencies on employment services for people with SMI.

In response to questions about interagency collaboration, leaders in states without direct collaboration typically reported either that local behavioral health agencies might refer service users to vocational rehabilitation services, that little communication occurred between agencies at the state level, or that the two agencies disagreed about the best way to provide employment services.

Department of Justice legislation

As shown in Table 2, 11 (29%) of the 38 states offering IPS (seven learning community states and four non-learning community states) had been impacted by Olmstead settlements or other similar litigation (which predated the Olmstead decision) that mandated the expansion or provision of employment services to people with serious mental illness. States with relevant Olmstead settlements reported a total of 236 IPS programs operating in states, and states without relevant Olmstead settlements reported a total of 285 programs. Three learning community states with Olmstead or other settlements joined after the settlements had gone into effect, while four states implemented Olmstead settlements after joining the learning community.

Table 2 IPS in states with and without Olmstead settlements
Full size table

Compared to states without Department of Justice settlements, states with such settlements neither had a higher IPS program penetration rate, nor did they differ in level of technical assistance to IPS programs, provision of fidelity monitoring, or the level of reported collaboration between behavioral health and vocational rehabilitation agencies. However, states with Department of Justice settlements did have significantly higher numbers of IPS programs than states without relevant settlements.

Discussion

IPS programs exist in the majority of U.S. states. However, most states with IPS services reported very low numbers of IPS programs compared to the state’s population and potential demand. More than half of the states offering IPS had instituted direct collaboration between behavioral health and vocational rehabilitation agencies, comprehensive technical assistance centers or full-time IPS trainers, and regular fidelity monitoring conducted by independent assessors.

There was no difference in IPS program penetration between learning community and non-learning community member states. While we initially expected that the IPS Learning Community might act as a causal agent for the growth of IPS, these results suggest that states sometimes initiate IPS programs on their own and join the learning community later, possibly seeking to expand or sustain their existing programs.

Membership in the IPS Learning Community was strongly associated with state agency quality indicators. States within the learning community were much more likely than non-learning community states with IPS to report providing high levels of technical assistance to IPS programs (either establishing a technical assistance center for IPS or employing a full-time IPS trainer), conducting regular, independent fidelity reviews of IPS programs, and collaboration between state behavioral health and vocational rehabilitation agencies. Annual, independently conducted fidelity reviews are an essential aspect of maintaining adherence to the IPS model, which has been associated with better employment outcomes for IPS service users (Bond et al. 2012; Kim et al. 2015). Learning community training for state-level and program-level leaders may emphasize the importance of fidelity reviews in achieving good employment outcomes, and may facilitate higher levels of communication between state behavioral health and vocational rehabilitation agencies.

Although Olmstead settlements have prompted some states to implement IPS services, our analyses did not indicate any significant differences in IPS penetration or quality between states with and without Olmstead settlements. The full impact of the Supreme Court’s 1999 Olmstead decision may not yet be evident. The U.S. Department of Justice launched its effort to enforce the Olmstead decision in 2009, and continues to file suits and argue cases today. Certain states may also have changed their policies proactively, increasing community integration in order to avoid the threat of an Olmstead settlement.

In addition to the IPS Learning Community and individual state initiatives, other factors promote the growth of supported employment. In 2014 the Substance Abuse and Mental Health Services Administration (SAMHSA) funded supported employment programs in seven states through the Transforming Lives Through Supported Employment Program. These grants are designed to enhance state and community capacity to provide evidence-based supported employment programs to people with serious mental illness. Another source of growth has been non-learning community states that have moved forward aggressively, such as New York, which has 92 IPS programs. Margolies et al. (2015) describe New York’s approach to expanding IPS services in the state. Of the 19 states with IPS outside the learning community, five account for 72% of the IPS programs.

Despite the positive features of IPS and efforts to increase its availability, few service users in community behavioral health systems across the U.S. have access to IPS or other evidence-based practices. Only about 2% of clients with serious mental illness served in the community behavioral health system received any supported employment services in 2012 (Bruns et al. 2016). Another study found that less than 1% of Medicaid beneficiaries with schizophrenia had an identifiable claim for supported employment (Brown et al. 2012).

Outside of the SAMHSA surveys of evidence-based practices discussed above, national surveys of evidence-based practices are rare. For example, to our knowledge, the last published national survey of assertive community treatment was reported in 1995 (Deci et al. 1995). A recent survey of early intervention programs identified only 34 programs nationally (White et al. 2015). Unfortunately, therefore, there are no comparable studies on the national prevalence and quality of other evidence-based practices in community mental health (e.g. assertive community treatment, supported housing, medication management, illness self-management, dual diagnosis treatment, family psychoeducation). Existing data suggest that other evidence-based practices are similarly rare in community mental health settings. Over one year, between 3 and 5% of Medicaid beneficiaries with schizophrenia had identifiable claims for assertive community treatment, family therapy, psychoeducation, or skills training (Brown et al. 2012). Between 2 and 3% of clients with serious mental illness served in the community behavioral health system received any supported housing, functional family therapy, or assertive community treatment in 2012 (Bruns et al. 2016). However, current data sources for the prevalence of other evidence-based practices do not clearly define the specific evidence-based practices, estimate the prevalence of programs nationwide, or comment on the quality of existing programs.

Limitations

Data were obtained primarily from two state agency representatives from each state. Although chosen because of their role in monitoring employment services for adults with serious mental illness, they may have not been aware of IPS programs affiliated with private organizations not receiving state support, early psychosis programs, or services for transition age youth. We did not include any programs associated with the Department of Veterans Affairs. The information was collected via telephone interviews and not corroborated by any state agency reports. While all the Olmstead settlements were confirmed on the Olmstead website, additional settlements may not have been identified by state agency representatives. Therefore, we may have underestimated the prevalence of these settlements. Outside the learning community the terminology used to identify IPS is imprecise; respondents also referenced programs using the “SAMSHA toolkit,” the “Dartmouth” model, and “evidence-based supported employment.” We may have underestimated or overestimated the number of IPS programs due to a difference in terminology. States that are members of the IPS Learning Community report the number of both IPS teams and provider agencies as part of their quarterly outcomes reporting. Although most agencies participating in the learning community have a single IPS team—81% of agencies in one survey (Bond et al. 2016)—large agencies sometimes have multiple IPS teams. During our interviews, we asked each respondent to report the number of IPS programs in their states. In the learning community, state leaders understand “programs” to refer to provider agencies. When interviewing states that were not members of the learning community, we did not distinguish between “provider agency” or “team.” We assume that, in the large majority of cases, provider agencies had only one team, but there may be some minor discrepancies in IPS program numbers outside of the learning community because of differences in terminology. Finally, the measure of IPS program penetration is a crude index that gives equal weight to all IPS programs regardless of size. IPS program size varies widely; we do not know, for example, whether IPS programs are larger in learning community states compared to states outside the learning community.

Conclusions

IPS has been widely disseminated across the United States. However, IPS services are still not available to millions of people with serious mental illness who want to work. Future efforts should emphasize the role of employment in recovery and ensure that helping people gain competitive employment is an integral part of community mental health services.

States outside of the IPS Learning Community report low levels of program quality activities (including training, fidelity monitoring, and collaboration between agencies) that may be necessary to sustain and expand IPS programs in the future.