Multi-service psychiatric rehabilitation programs that operate in community settings are increasingly adopting evidence-based practices for adults with severe mental illness, including supported employment (SE), cognitive and social skill training, health promotion activities, supported education, supported housing, and supported socialization [1–5]. However, multi-service programs often find it difficult to meet the fidelity requirements established for specialty interventions. This is especially true of unstructured programs, like drop-in centers and clubhouses, in which clients are encouraged to seek out the staff workers they prefer in lieu of scheduled appointments with assigned staff. For example, supported employment standards [6, 7] mandate the exclusive provision of vocational services by trained specialist staff who carry separate client caseloads [8–10]. By contrast, many multi-service programs rely on generalist staff to ensure the continuous availability of a wide range of services, and allow clients to casually request the services they need from any staff worker they encounter in the program’s social milieu [11–13]. Are evidence-based services less effective when provided by generalist psychiatric rehabilitation staff rather than only by trained specialists, or when multiple staff are permitted to serve the same individuals instead of carrying separate caseloads?

A consideration of how supported employment is provided in multi-service programs suggests that it may be possible to comply with the intent of a fidelity standard, even while violating the explicit requirement. Supported employment experts argue that separate staff caseloads are essential because exclusive staff–client relationships build rapport and strengthen the therapeutic alliance [10, 14]. On the other hand, many psychiatric rehabilitation programs believe staff–client relationships are strongest when they are allowed to evolve naturally as clients spend time with staff in collaborative activities and conversation. Clients who interact casually with staff in a social milieu can draw on diverse staff expertise and receive help that is spontaneous and timely [11, 15, 16]. In a typical drop-in center, the same individual consumer might choose to begin an online job search with a computer instructor, discuss social security restrictions on gainful employment with a vocational specialist, and then update her resume with a social worker who knows her well. We do not yet know whether this ‘generalist staff/shared caseload’ approach is as effective as the recommended ‘specialist staff/separate caseload’ approach to supported employment because fidelity studies have not compared different methods of staffing or tested the relative contribution of particular model components to SE outcomes [17, 18]. It is possible that either approach is effective as long as program staff provide a core set of high quality services.

We investigated the viability of generalist staff provision of supported employment services in a proof-of-concept study of a single community program that allowed clients to choose to work with any staff worker, at any time, on their employment goals. Our intent was to provide preliminary evidence of a procedural link between shared caseloads (measured as count of different SE service providers per consumer) and supported employment outcomes. Because our study was non-experimental, we tested a very specific prediction that would be highly unlikely to occur by chance [19, 20].

FormalPara Hypothesis

The more staff who provide supported employment services to a participant in a socially-interactive community support program, the greater likelihood that individual participant will work a mainstream job.

We reasoned that, if this hypothesis was supported by our findings, and if the employment outcomes of this multi-service drop-in center were as good as published outcomes for high-fidelity supported employment programs that rely on specialist staff and separate caseloads, we could conclude that the shared generalist staff/separate caseload approach is a viable adaptation of SE fidelity standards that merits further study in randomized trials.

The present study was a ‘science-to-service/service-to-science’ [3] collaboration between two psychiatrists and two supported employment researchers, and the manager, staff, and members of an outpatient psychiatric rehabilitation program at McLean Hospital in Belmont, MA. These individuals worked closely together to design a study that would serve both as a theory-based research project and a quality improvement project [21–23]. No external funding was received, but all data retrieval procedures were approved by the McLean Hospital institutional review board, and by the members and staff of the program under study.

Methods

Research Site

The site for this study is an outpatient psychiatric rehabilitation program located near Boston that had been in operation for 7 years. The program is a drop-in and resource center that offers individual counseling, practical help, and daily opportunities for group activities, including a social milieu in which members (clients) talk casually with one another, professionals, and peer staff. The program follows a philosophy of consumer empowerment in which members decide what services they need and when, and help to plan and lead support, discussion, and special interest groups, as well as recreational activities. Attendance is not mandatory, and members are welcome to suggest a new activity or outing at any time, or simply sit alone and read or listen to music. Programming is organic and consumer-driven, so there is no service manual, and the delivery of individualized staff services is most often a spontaneous response to a member’s expression of need, either in private consultation or casual conversation.

Employment Services

The program under study offers individualized (one-on-one) supported employment services, including assistance with job searches, applications, and resume preparation, benefit counseling, career planning, interview practice, job travel training, on-job coaching, mediation of work conflicts, and negotiation of disability accommodations. A master’s level vocational specialist carries bottom-line responsibility for the quality of these SE services and provides as-needed consultation to other program staff. Employment services are provided either formally, as when the vocational specialist approaches an individual to inquire about work interest, or informally, as when program members seek out the help they needed from any, and as many, staff workers as they want. This member-driven process creates informal ‘provider teams’ tailored to each individual’s needs and preferences [11]. Staff caseloads are not assigned, but develop naturally during weekly service planning meetings, as each staff worker identifies the members he or she is currently assisting so that staff assisting with the same individuals can coordinate their efforts and complement each other.

Staffing

The study program was similar in client–staff ratio (16:1 FTE) and membership size (about 100 active clients per year) to most certified clubhouse programs in the USA [24] The program was staffed throughout 2007 by four full-time professional staff with master’s degrees: two social workers, one occupational therapist, and one vocational specialist (Table 1). Beginning in April 2007, there was also a part-time computer instructor who helped with internet job searches, resume preparation, and software practice needed to pass a job qualification test, e.g., spreadsheet data entry or word-processing. At varying times, the program also had one or more graduate interns from local university departments of social work or occupational therapy, and 2–4 part-time peer specialists (non-professionals with diagnoses of mental illness). All staff provided vocational services whenever such services were requested, and the staff met together once a week to review members’ progress toward employment or other personal goals and to coordinate their own efforts.

Table 1 Staff provision of one-on-one vocational services in 2007
Full size table

Sample

The study sample included all service program clients who were active in services for at least six-months during the 2007 calendar year (N = 99). ‘Active status’ was defined as having at least 3 days of service contact in any one month of 2007, and the equivalent of more than a full week of service (8 total days) over the course of the year. This definition eliminated those individuals whose only contact was an occasional drop-in or phone call, as well as those who attended intensely for a few days to try out the program but never fully engaged. To ensure that all study participants had been active in services for at least six months, we omitted everyone who enrolled in the program after July 1st (n = 12).

The sample had equivalent numbers of men and women participants (50% each gender), and about half the sample (55%) had a primary diagnosis of a schizophrenia spectrum disorder. Almost all (92%) were Caucasian, in keeping with the ethnic distribution of the surrounding community. At the time of this study, the membership had an average age of 46 years (M = 45.5 ± 10.7, range: 21–67 years). First psychiatric hospitalization occurred approximately 20 years earlier, at an average 24 years of age. Seventy-five percent had never been married, and only 7% were currently married. About three-fourths of the membership had a monthly income of less than $1200, and a third received regular financial support from their parents or other relatives. One-fourth of the sample lived with a relative, usually a mother.

Measures

This study relied entirely on data the agency routinely collected for its own reporting and quality assurance purposes so that study procedures would be replicable in other service systems.

Participant Characteristics

Table 2 shows that four background characteristics were significant, or near-significant, correlates of mainstream employment in 2007: gender, physical health, age, and family financial support. Physical health was measured as the presence (0) versus absence (1) of any potentially disabling chronic health problem (e.g., high blood pressure, COPD, congestive heart failure, rheumatoid arthritis, morbid obesity). Participant age was calculated for December 31, 2007. Family financial support was a dichotomous coding of any versus no monthly dollar amount reported as income at enrollment.

Table 2 Characteristics of employed & unemployed participants active in 2007 (N = 99)
Full size table

Age, health, and gender were strongly correlated, reflecting the fact that 84% (n = 26) of men younger than age 50 had no chronic health condition (X 2 = 8.33, df = 1, P = .004), whereas 71% (n = 17) of women age 50 or over reported a chronic health condition that limited their everyday functioning (X 2 = 4.12, df = 1, P = .042). Because this natural co-occurrence of characteristics confounded these three variables, we created a dichotomous variable for the multivariate regression analysis that compared participants whom we expected would be most likely to work (relatively healthy men under age 50) to all other study participants. Substituting this one subgroup variable for the three separate background variables in the regression analyses increased statistical power and provided protection against collinearity [25].

Program Attendance

Attendance is not mandatory, but members of the program voluntarily sign an attendance sheet kept close to the front door as part of the program’s continuous quality improvement efforts. To ensure that attendance records are accurate, the program receptionist also maintains an attendance checklist. Attendance was measured as total days during the specified study periods. It was not feasible to track total hours of attendance because members are free to come and go throughout any day.

One-on-One Employment Services

All program staff maintain daily service logs that track their activities and all one-on-one contacts with clients. The computerized software system documents the date, time of day, and duration of each one-on-one contact with a client, need addressed, and type of service rendered. Contact notes are entered as narratives that serve as clinical records. Employment services are first coded as ‘vocational’ and then subcoded by the staff worker using standard supported employment service categories developed for a multi-site employment study [26], e.g. a staff phone call on behalf of a client to a local employer to gather more information on an advertised position would be coded ‘assistance with job search.’

To improve reliability and reduce the impact of outliers (i.e., time periods in which an individual had unusually high or low program contact), all service measures were ‘moving averages’ [27], calculated as each participant’s mean count of service days or providers across all possible consecutive six-month periods during 2007 (e.g., January–June, February–July, March–August, etc.) subsequent to the participant’s enrollment in the program. ‘Frequency of program attendance’ was each participant’s mean count of attendance days, averaged across all possible six-month periods. ‘Frequency of one-on-one employment services’ was each participant’s mean count of days on which the participant received any face-to-face supported employment service, averaged across these same six month periods. ‘Count of one-on-one employment service providers’ was each participant’s mean count of different staff providing face-to-face supported employment services during the same six-month periods. Averaging attendance days and provider counts across overlapping time periods smoothed outliers and allowed us to report reliable counts for a typical six-month reporting period while taking advantage of 12 full months of data collection.

Employment Outcomes

We counted as mainstream employment any job that met the Department of Labor’s definition of competitive employment: (a) any individually-held job, (b) located in a mainstream, integrated setting, which (c) paid current minimum wage or higher [28]. We adopted an ‘intent-to-treat’ method of employment rate calculation. The number of consumers employed in a mainstream job at least one week during the 2007 calendar year was expressed as a ratio of all consumers active in the program during 2007.

Data Analysis Plan

We tested our single hypothesis of a positive association between count of SE service providers and mainstream employment in two separate hierarchical logistic regression analyses. The first analysis was a whole calendar year comparison of participants who were unemployed (n = 72) versus employed (n = 27) any time during 2007. The two predictor variables were each participant’s mean days of SE service receipt and each participant’s mean count of different SE providers over consecutive six-month periods during the 2007 calendar year.

This first hierarchical regression analysis had four blocks of variables. Block 1 contained participant background characteristics that might offer an alternative explanation for the predicted findings. A greater percentage of unemployed participants were female and over age 50 (Table 2), and this corresponded to a statistical trend (P = .12) for women over age 50 to receive supported employment services from fewer staff. To rule out the possibility that these demographic variables might partially account for any observed positive association between count of SE providers and employment status, we included the subgroup measure of inter-correlated characteristics ‘young, male, good health’ as a covariate. We included ‘any versus no receipt of family financial support’ as a second covariate to explore the strength of this rarely studied variable as a predictor of employment when age, gender, and physical health are statistically controlled. Block 2 contained a measure of each participant’s mean days of program attendance, calculated across consecutive six-month periods in 2007. This measure of program attendance controlled for the fact that participants who routinely attend Waverley Place have more opportunities to talk with staff about their lives. Block 3 contained each participant’s mean days (frequency) of one-on-one employment service receipt during the same consecutive six-month periods in 2007. Block 4 contained each participant’s mean count of one-on-one employment service providers across the same time periods.

In order to infer that the receipt of one-on-one employment services from multiple staff contributed to mainstream employment, mean count of staff providing one-on-one employment services during the designated six-month periods would need to be a significant positive predictor of mainstream employment during 2007 over and above mean frequency of one-on-one employment service receipt and all other variables in the regression model.

The second regression analysis tested the same hypothesis by comparing participants who started a new job in 2007 (n = 17) to those who either remained unemployed or continued to work jobs they already held on December 31, 2006 (n = 82). The inclusion of already-employed participants in the comparison group of this second analysis provided a more stringent test of the study hypothesis. This second analysis also allowed a test of a temporal association between start of a new job and prior receipt of employment services. For the 17 participants who started a new job in 2007, employment service frequency and count of providers were calculated as individual mean scores across the three consecutive six-month periods preceding and overlapping with the job start date. (If the job started in the first half of 2007, pre-job service measures extended back into 2006.) Service measures for the comparison group were calculated in the same way as for the first analysis, i.e., individual mean scores across all consecutive six-month periods in 2007 after the individual had become active in the program.

The design of the second hierarchical regression analysis was identical to the design of the first analysis, except that the only control variable included in Block 1 was absence versus presence of a chronic physical health condition, because this was the only background variable that distinguished new employees from all other participants and showed a trend toward a positive association with count of SE service providers (P = .07).

In order to infer that the receipt of one-on-one employment services from multiple staff facilitated the start of new jobs, mean count of staff providing one-on-one employment services during the designated time periods would need to be a significant positive predictor of whether the participant started a new job over and above mean frequency of one-on-one employment service receipt and all other variables in the regression model.

Results

Program Participants Employed in 2007

The mainstream employment rate for all participants active in the psychiatric rehabilitation program during calendar year 2007 was 27% (n = 27 out of 99). Quarterly mainstream employment rates across the four quarter periods of 2007 were 25% (Jan–Mar), 25% (Apr–Jun), 27% (Jul–Sep), and 23% (Oct–Dec), respectively, with a mean quarterly rate of 25% for 2007. Corresponding quarterly counts of active participants ranged from 90 to 107. The stability of the research program’s membership size and employment rate over the four quarters of 2007 suggests that the 25% mean quarterly employment rate is a reliable measure.

Job diversity reflected the wide variation in employees’ educational backgrounds, ranging from law clerk to nursing home recreational aide to grocery store bagger. Four participants received specialized training and served as peer staff in another rehabilitation setting as a case manager, protective payee, skill trainer, or supported education tutor. About a fourth of all employed participants (6 of 27) worked 32 to 40 h a week, and median hourly wage across all jobs was $12. By broad service type, one-on-one supported employment services provided individually to participants by program staff included assistance with job searches and applications (33%), work-focused supportive counseling (32%), job-related skill building (20%), and career planning & skill assessment (15%). The majority of participants served by a single staff-worker (n = 14) received only initial vocational assessments.

Among the 27 participants who worked a mainstream job in 2007, 10% (n = 3) were employed less than six months, but 70% (n = 19) stayed employed more than 12 months during the 2007–2008 calendar years. Seven individuals worked the entire two years. Seventy percent of those who started a new job in 2007 (n = 12 of 17) were still employed a year later, and all except one new job lasted at least 8 months (M = 432, SD = 157; Median = 443; range: 77–730 days).

Univariate Comparisons of Employed Versus Unemployed Participants

As Table 2 shows, program participants who worked a mainstream job in 2007 received one-on-one employment services more frequently than other participants, and they also received employment services from a larger number of staff workers. The mainstream employment rate for participants who received one-on-one employment services from three or more different staff workers during 2007 was 39% (n = 16 of 41), which was significantly higher than the 19% employment rate (n = 11 of 58) for all other participants (X 2 = 4.87, P < .05).

A similar comparison of participants who did versus did not start a new job in 2007 shows that the more staff providing employment services to a participant, the greater likelihood that individual would start a new job (n = 17, M = 2.5 ± 1.3 vs. n = 82, M = 1.26 ± 1.17; t = 4.03, df = 1, 97, P < .001). Study participants who started new jobs had higher counts of employment service providers in the months immediately preceding their new job (Mdn = 2.33, interquartile range = 1.67–3.50) compared to staff counts for other participants over comparable six-month intervals (Mdn = 1.14, IQR = .11–1.86).

Tests of a Positive Association Between Count of SE Providers and Participant Employment

‘Calendar Year’ Comparison of Employed Versus Unemployed Participants

The results of the first hierarchical logistic regression analysis (Table 3) support the study hypothesis that the more staff who provided supported employment services to an individual, the greater likelihood he or she would be employed. In Block 1, membership in the ‘less than age 50, male, healthy’ subgroup and receipt of family financial support are both positive predictors of mainstream employment. However, in Block 2, mean days (frequency) of program attendance is not predictive of employment status. In Block 3, mean days (frequency) of one-on-one employment services is a positive predictor of employment. When mean count of employment service providers is added to the regression model in Block 4, it is also a significant predictor of employment, over and above all other variables in the full model. Frequency of one-on-one employment service receipt remains a significant but weaker predictor of mainstream employment after the final addition of mean count of employment service providers, indicating that staff count may partially explain the positive association of SE service frequency with employment status. SE service frequency and provider count scores were only moderately correlated (r = .48; P < .001), but a visual inspection of plots of SE service delivery over time suggests that higher staff counts may have increased the frequency of service receipt because program staff tended to provide employment services to the same participant on different days.

Table 3 Logistic regression analysis of association between individualized vocational services and mainstream employment
Full size table

Two background characteristics predicted mainstream employment in 2007 in the full regression model. The subgroup of program participants who were male, younger than age 50, and in good health were more likely to work than other participants (42%; n = 11 versus 23%, n = 16; X 2 = 8.33, df = 1, P = .004). Only a single woman (6%) in the older, chronically physically ill women subgroup (n = 17) worked a mainstream job. Employed participants were also more likely to receive financial support from their parents or other relatives. A closer look at this positive correlation between family support and employment reveals it is limited to participants (n = 67) on SSI or SSDI (X 2 = 3.87, P < .05).

Comparison of Participants Starting New Jobs Versus All Other Participants

A second hierarchical regression analysis (Table 4) provided a more direct test of the same study hypothesis by comparing participants who started a new job in 2007 to participants who either remained unemployed or continued to work a job they already held. In Block 1, absence of a chronic health condition approached statistical significance (P = .08) as a predictor of new employment. In Block 2, the covariate mean days (frequency) of program attendance was again unrelated to employment. However, both mean days (frequency) of employment service receipt (Block 3) and mean count of staff providing employment services (Block 4) were significant predictors of a new job start in the full regression model. As in the first analysis, frequency of employment services remained a significant but weaker predictor of new employment after count of service providers is added to the regression model in Block 4, suggesting that service frequency tends to increase when two or more staff provide employment services to the same participant. Absence of a chronic health condition also predicted new mainstream employment in the full regression model.

Table 4 Logistic regression analysis of individualized vocational services as precursor to new mainstream employment
Full size table

An alternative explanation for these findings is that participants who had been in the program longer might have had greater familiarity with staff and more time to look for work, and so variations in program tenure might account for the positive correlation between mainstream employment and provider counts. However, most program participants had been active long enough to know every staff worker well. Nearly one-half (42%) of all study participants had been active in this rehabilitation program for four or more years, 70% had been attending for at least two years, and mean duration of membership by the end of 2007 was identical for new employees versus all other participants (M = 42 months, n = 17 vs. 82).

Discussion

Positive Association Between Employment Status and SE Provider Counts

Study findings show that the more often one-on-one supported employment services were provided to program participants, the more likely the participant was to become employed. This finding was obtained even when statistically controlling for total days of program attendance and background characteristics that distinguished employed versus unemployed participants. Also, in support of the study hypothesis, the more staff providing supported employment services to an individual participant, the greater likelihood that participant would become employed. The present study was not a randomized controlled trial, but as a proof-of-concept pilot study, our findings reveal a higher than usual frequency of SE services, and higher count of different SE providers, in the months prior to the start of a new mainstream job. This positive temporal association between quantity of services received and subsequent outcomes is rare in supported employment research, as well as in mental health services research in general [29–33].

Can Generalist Staff Provide High Quality SE?

The multi-service psychiatric rehabilitation program that served as a research site for our study performed as well as other providers of supported employment that have published quarterly employment rates [34]. The program’s 25% average quarterly employment rate approximates the 27% average quarterly rate for eight high fidelity IPS programs in New Hampshire [35], the 26% quarterly rate for a certified clubhouse program in Massachusetts [36], and the 28% quarterly rate attained by a specialized supported employment team in South Carolina that was closely integrated with a program of assertive community treatment [37]. The 24-month participant follow-up cumulative employment rate for this same South Carolina program was 64%, which exceeds the 50–55% employment rates set as benchmarks for randomized controlled trials [38, 39], so a 25–28% point-in-time calendar quarter employment rate appears to be an appropriate performance standard.

Work longevity for employed members of this psychiatric rehabilitation program (70% worked longer than a year) exceeds the work duration outcomes for two state-level performance-based contract demonstration studies [40, 41], as well as average total days or total weeks of work reported by seven randomized studies of supported employment programs [36, 37, 42]. Comparability in employment outcomes between the study program and high performance specialized employment programs suggests that multi-service programs can effectively provide supported employment services, even while they are simultaneously serving individuals who do not want employment, and that one particularly effective method of service delivery in multi-service settings may be generalist staff roles.

Association of Participant Characteristics with Mainstream Employment

Previous supported employment research suggests that older age [43–46] and poor physical health [47–49] are serious obstacles to mainstream employment, especially when these two attributes co-occur [25]. Our findings replicate the negative correlation between health problems and employment, and introduces new evidence that the natural co-occurrence of older age and poor health may sometimes be greater for women than men. This finding may be unique to our study sample, or it may help to explain contradictory findings for gender alone as a predictor of employment within the population of adults with severe mental illness [46, 50].

We also identified a positive correlation between employment and family financial support for participants with social security income that suggests the availability of monetary support might help to ameliorate worry about loss of benefits. Financially supportive families may also encourage mainstream work by helping to navigate entitlement restrictions on gainful employment. This tentative finding deserves closer scrutiny in future research.

Implications for Supported Employment as an Evidence-Based Practice

Separate staff caseloads and specialist roles may be effective ways for some programs to provide supported employment, but a generalist, shared caseload approach, wherein any staff worker can provide employment services to any client on demand, appears to be a viable alternative for multi-service rehabilitation programs. Generalist roles may sometimes be more acceptable than specialist staff roles in programs capable of full fidelity to SE standards. In a recent USA survey, 55 exemplary supported employment practitioners gave the SE fidelity component ‘exclusive vocational focus’ their next to lowest mean rating of importance [51], possibly because many SE specialists are master’s level psychologists and social workers, professions that have a strong multidisciplinary focus that fits a generalist perspective.

The most essential ingredient for effective supported employment may be the provision of a core set of individualized employment services (e.g., rapid job searches, benefit counseling, resume’ preparation, interview practice, job travel training, work-related counseling) geared toward socially-integrated mainstream employment that pays at least minimum wage. The findings of our study suggest that these practical services are effective even when provided by a psychiatric rehabilitation program that has no service manual or fidelity standards. Moreover, an outpatient program that offers easy access, drop-in services, and casual care may be more appealing to many persons with severe mental illness, and more likely to be utilized, than a technically correct but fragmented system of specialized care [52].

The demonstrated ability of our study program to track consumer outcomes and maintain daily service logs suggests that eclectic community programs could become valuable laboratories for studying how to adapt standard practices to achieve congruence with a program’s own mission and structure [53, 54], as well as useful for identifying the core ingredients of any evidence-based practice. We will have more confidence in particular fidelity requirements, and a better understanding about what makes an evidence-based practice effective [55–57], if researchers and community program stakeholders continue to work together to design replicable field studies to test the viability of program-driven adaptations of evidence-based practices [58–60]. Toward this end, we restricted our study measures to the record data tracked by many psychiatric programs in this era of managed healthcare, but more research is needed to standardize research procedures and identify potential sources for research funding [61].

Conclusions

This study builds on an earlier documentation of the natural evolution of client–staff alliances in a psychiatric rehabilitation program designed to encourage client confidence and self-determination [11]. A generalist staff approach, wherein any staff worker can provide any type of services to any client, and every consumer has the right to request any type of help from any staff worker, appears to be a viable way to integrate supported employment services into eclectic psychiatric rehabilitation agencies that represent ‘usual services’ in many communities.