Introduction

The preponderance of the research conducted to date on vocational outcomes for individuals with psychiatric disabilities has focused on the principles and structure of work interventions that are designed to promote vocational outcomes (Drake et al. 2012). Research has culminated in the development and testing of Individual Placement and Support (IPS) which is now an evidence-based supported employment practice with numerous randomized trials confirming its effectiveness (Drake et al. 2012). For the most part, examinations of the fidelity of the IPS approach have  focused on the presence and quality of structural elements such as caseload size, agency support, coordination among specialists, and the mechanics of job development, among others (Luciano et al. 2014; Individual Placement and Support Employment Center 2018). Much less is known about the competencies and characteristics of employment specialists who deliver employment interventions in terms of their ability to promote work outcomes.

Over the past few years, researchers have attempted to better understand employment specialist competencies. Whitley et al. (2010) conducted a qualitative study to examine the “desirable characteristics” and competencies of employment specialists. Using an empirically grounded framework and content analyses, employment specialists and their supervisors judged the following characteristics as important: (1) initiative, (2) outreach, (3) persistence, (4) hardiness, (5) empathy, (6) passion, (7) team orientation, and finally, (8) professionalism. However, these characteristics did not predict employment outcomes. Dreher et al. (2010) developed a 30-item self-report measure to assess 107 specialists from high-fidelity IPS programs and 59 specialists from non-IPS vocational programs in the key principles and components of IPS. The authors concluded that their scale tapped job knowledge, and that scores differentiated between high fidelity IPS versus non-IPS programs. While useful for rating knowledge of the structural elements of IPS, the scale did not examine the relationship or the working alliance between the employment specialist and client.

Using self-reports of performance along with supervisory perceptions, Taylor and Bond (2014) examined the relationship between ratings and employment outcomes among 57 employment specialists. Paradoxically, employment specialists’ self-report measures of their competencies were unrelated to outcomes. Supervisory ratings of employment specialist job performance, efficiency, percent of time spent in the community, and the frequency of contacts with service recipients were all associated with better employment outcomes. The authors concluded that direct observation of employment specialists’ job performance was needed to better identify competencies that predict employment outcomes.

Corbière et al. (2014) undertook an extensive examination of employment specialists’ competencies by developing and testing a 90-item questionnaire (Behaviors, Attitudes, and Knowledge—BAKES) using a cross-cultural, cross-model sample of employment specialists. The cross-model aspect of the study was achieved by having data from IPS and non-IPS employment specialists, while the cross-cultural aspect was addressed using samples from Canada and the Netherlands. Employment specialists completed the BAKES and provided contemporaneous employment outcome information for the clients they served. Exploratory factor analysis yielded two useful subscales which predicted employment outcomes: (1) relationships with employers and supervisors, and (2) support and a client-centered approach. The first subscale was the most predictive of positive employment outcomes. Corbière and colleagues (Corbière et al. 2014; de Pierrefeu et al. 2017) further concluded that employment specialists require and should be trained in a broad range of competencies and affirmed the importance of the working alliance, concluding that it predicted employment outcomes (Corbière et al. 2017).

Results from psychotherapy research may be particularly instructive in understanding the role of the employment specialist in promoting positive outcomes. While there is a significant emphasis on evidence-based therapeutic approaches, there is also a large body of research suggesting that the clinician-client relationship is of great importance in establishing an effective working relationship, as well as in setting and attaining goals (Wampold and Imel 2015; Wampold 2015). Numerous meta-analyses have been conducted over the past 30 years suggesting that the relationship between a helping professional and his or her client is responsible for unique variance in predicting positive outcomes over and above any particular model or approach utilized. Wampold and Imel (2015) have attempted to unbundle the effects on outcomes of specific treatment approaches, versus the non-specific or “common” factors that are more relational in nature. In a recent review of meta-analyses, Wampold (2015) concluded that there are robust effects of therapists, which leads to the question: what makes therapists using the same model or therapeutic approach more effective than others? Results of several studies suggests the following traits: effective therapists (when compared to less effective therapists) can form stronger alliances across a broad range of patients, have better facilitative interpersonal skills, express more professional self-doubt, and engage in more time outside of the actual therapy practicing various therapy skills (Wampold and Imel 2015). Two other large meta-analyses found significant “non-specific” effects in the treatment of depression (Palpacuer et al. 2017), and in a large analysis of psychotherapy outcomes (5828 patients and 158 therapists) (Goldberg et al. 2016). This research, and the characteristics identified by Wampold (2015), may be instructive in understanding the competencies that can be helpful for employment specialists and that transcend specific interventions.

Taken together, results of studies to date yield equivocal and limited information about needed specialist competencies that promote engagement in employment services and promote employment outcomes. Insufficient attention is paid to the strategic competencies of employment specialists and the working alliance they forge with individuals who have psychiatric disabilities and are served in employment programs. Furthermore, to the best of our knowledge, there are no attempts to garner the perspective of service recipients in defining these competencies. Thus, the purpose of this study was to examine, using a participatory research framework, which competencies service recipients and employment specialists consider critical to vocational recovery. Using exploratory and descriptive survey research methods, we examined these factors and validated competencies. We developed candidate items to measure these competencies and tested the scale’s psychometric properties.

Methods

All exploratory and descriptive survey research methods were carried out from 2014 to 2017. Recruitment materials and survey instruments were reviewed and approved by the University Institutional Review Board (IRB), as well as the State Department of Mental Health IRB.

Stage 1 Procedures

Our initial conceptual work began by examining literature that addressed the competencies of employment specialists, and their associated constructs and facets. We were guided by previous work at our Center to develop and validate a measure that defines competence, aiming to promote the overall recovery of persons with psychiatric disabilities (Russinova et al. 2011, 2013). Based on previous literature and work, we hypothesized that the dimensions of hope, empowerment, purpose, and self-acceptance, as well as the quality of the relationship between the employment specialist and the service recipient, would be considered critical competencies for employment specialists. We drafted candidate items tapping those constructs. We then consulted with vocational experts in the field of psychiatric vocational rehabilitation to refine those items.

Stage 2 Procedures

We began this stage with n = 64 draft items that tapped the constructs mentioned above of vocational empowerment, hope, self-acceptance, and purpose. We conducted two focus groups with employment specialists and one focus group with clients to determine if our initial set of items was comprehensive and complete. Inclusion criteria to participate in the client focus group were: (1) being an adult with a psychiatric disability, (2) being employed or wanting to work, and (3) currently receiving or having received services from an employment specialist (e.g., vocational counselor, employment specialist) in the last 2 years. A total of seven adults with psychiatric disabilities were recruited from our recovery program to attend a focus group that took approximately 90 min. A cash stipend was provided for participation in the focus group.

Inclusion criteria for the employment specialists were: (1) working as an employment specialist irrespective of the model of service delivery adhered to, (2) working with individuals who have a psychiatric disability, and (3) being employed at a local employment program funded by the State Department of Mental Health. A total of 16 employment specialists were recruited to attend 2 separate focus groups that took approximately 90 min each. A cash stipend was provided for attendance and participation at these focus groups.

Focus groups for employment specialists and clients were conducted separately, but followed a similar structure: (1) both employment specialists and service recipients were oriented to the research study and consented, (2) the researchers (CT, ESR, ZR) provided a detailed explanation of what feedback and input was being sought, and (3) the researchers asked for comments about the overarching constructs/subscales (e.g., vocational hope, acceptance, etc.) as an overview. We then asked for input item by item. In the employment specialist focus groups, we asked if there were items they perceived as unimportant in terms of the competencies they reflected. We asked for input about additional competencies not reflected in the items presented, and about items that tapped an important competency but were not worded in such a way as to convey that competency. We also queried employment specialists about specifically whether each item would be appropriate for any type of client seeking services, as we wanted to assess competencies that would be considered universal and appropriate for all individuals with psychiatric disabilities seeking employment services. Employment specialists supplied valuable input about the wording of specific items. For example, words or items they considered to be emotionally laden were recommended for modification; they examined the appropriate tense to use in items, as well as whether items would be appropriate irrespective of where a client was in choosing, getting, or keeping employment. Employment specialists also suggested items to reflect competencies not tapped in the items presented.

Clients who participated in the focus groups were also asked about additions, deletions, and edits to the items presented, but, in this focus group, individuals were asked to comment from their own experiences and not about the appropriateness of the items across client populations.

All focus groups were audio-recorded, and those recordings were transcribed by research staff at the Center. We then undertook content analyses of the transcripts using NVivo 10. Results of these analyses allowed us to expand the constructs we examined, and to use those constructs and the comments of focus group members to develop new items. We concluded these activities with 60 candidate items.

Stage 3 Procedures—Refining Candidate Items

We used a combination of feedback we received through focus groups and experts to weigh the value of each item, including the clarity and brevity of each item as well as the number of items for each domain. We began with 60 items, eliminated 14 items, modified 12, and added 4 new items, resulting in 50 candidate items for further testing. We solicited input on the importance of these items using an online survey of individuals receiving vocational or employment services. We had a very modest response to this survey, but that feedback combined with the research teams’ review of items resulted in 8 additional items being eliminated, 2 being merged, and 1 being modified. Next, we conducted 4 cognitive interviews to ensure the correct comprehension and meaning of each candidate item for our population using a semi-structured cognitive interviewing approach (Willis 2014). Inclusion criteria for selecting individuals for the cognitive testing were identical to the criteria for the client focus group. Participants in the cognitive testing were recruited from the Center recovery program. Results from this step lead to the deletion of seven items and the modification of six items.

At the conclusion of this refinement process, we agreed upon 34 items that tapped competencies in the areas of promoting hope, empowerment, self-acceptance, and purpose in relation to work, and the interpersonal skills of employment specialists.

Stage 4 Procedures—Psychometric Testing

We next sought to empirically validate the candidate items using an anonymous internet survey targeting a national sample of individuals with psychiatric disabilities who had or were receiving vocational services. This stage is described more fully below.

Research Procedures for Stage 4

Sample

Inclusion criteria were similar to those used in the previous stages of the project, and were assessed through the use of screener questions prior to beginning the data collection. In order to ensure that we were targeting individuals with serious psychiatric disabilities, we asked the following questions: (1) are you currently using medications for your psychiatric symptoms?; (2) have you ever been hospitalized for psychiatric symptoms?; (3) have you had a major disruption in your life (for example, lost a job, interrupted school, got divorced) due to psychiatric issues?; and, (4) are you currently receiving SSDI or SSI because of a psychiatric condition? Respondents were also asked if they were currently receiving employment services, or if they had received such services in the past 2 years. Individuals who responded affirmatively to any one of the screener questions about the severity of their disability, and who were (or had) receiving employment services, could access the survey portal. Individuals who did not pass through the screener questions were thanked for their interest and told they were not a match for the survey. The survey was monitored continuously from the beginning to the end of data collection. Completed surveys were downloaded from Qualtrics Software (2013; www.qualtrics.com) into SPSS 19.0 for data analysis.

Data Collection

The draft VPRS survey was constructed to be administered in Qualtrics and mounted on the Center’s website. Various strategies were used for recruitment including: an announcement in Center newsletters, banners placed on the Center’s website, and word of mouth.

Towards the end of the data collection period, respondents were asked to participate in a re-test assessment to examine the stability of responses. Data were collected from June 2016 until October 2017. A total of n = 142 usable surveys were completed for the initial survey, and 27 of these individuals were re-tested. Individuals were paid $20 for participation in each of the initial and the re-testing surveys.

Measures

  1. (1)

    Vocational Practices and Relationship Scale (VPRS) A total of 34 items of the draft VPRS were presented, and individuals were asked to endorse each candidate item based on their vocational service experience in terms of their agreement using the following 4-point Likert scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree.

  2. (2)

    Demographic-Clinical Questionnaire We used a Center-developed measure to capture information about respondents, including their demographic characteristics: age, gender, ethnicity, living status, as well as clinical characteristics: psychiatric diagnosis, use of psychotropic medications, and disability benefit status.

  3. (3)

    Working Alliance The Working Alliance Inventory (WAI), Short Form (Horvath, 2005) was used to examine concurrent validity of the VPRS. This 12-item instrument provides information about the extent to which the employment specialist and consumer have a positive working relationship and share mutual goals (e.g., “__blank__ and I are working towards mutually agreed upon goals”), tasks (e.g., “We have established a good understanding of the kind of changes that would be good for me”), and therapeutic bond (e.g., “I feel that __blank__ appreciates me”). Individuals respond on a 7-point scale ranging from Always to Never. The WAI has been demonstrated to have excellent reliability and validity (Horvath and Greenberg 1994).

Statistical Analyses

Data were downloaded from Qualtrics into SPSS and cleaned prior to analysis. We used a variety of statistical analyses to examine the psychometric properties of the VPRS items, subscales, and total scale. In addition to descriptive statistics, we used Pearson correlations to examine items and relationships among them. We conducted tests of internal consistency using coefficient alpha, and examined item-to-total correlations and alpha-if-item deleted. For the individuals on whom we collected re-test data, we conducted Pearson correlation coefficients to test for stability. Finally, to examine factorial validity, we conducted principal components analyses, all in an effort to develop a more parsimonious item set. Analyses were conducted in SPSS 19.0.

Results

Demographic characteristics of respondents to the draft VPRS survey appear in Table 1.

Table 1 Demographic characteristics (n = 142)
Full size table

The exploratory steps of developing, vetting, refining, and cognitively testing candidate items yielded 34 items for psychometric analyses, including items that tapped: Empowerment (n = 13 items), Purpose (n = 5 items), Self-Acceptance (n = 3 items), and Hope (n = 5 items)—all in the context of work–-and the Client-Specialist Relationship (n = 8 items).

First, we examined mean (SD) scores for all items within the original subscales. As can be seen in Table 2, items in the Vocational Hope and Core Relationship subscales rated higher, with the mean of all items scoring above 3.0.

Table 2 Results of psychometric testing
Full size table

Secondly, we examined coefficient alphas for each of the subscales. They were as follows: Hope .95 (n = 141), Vocational Empowerment .96 (n = 137), Acceptance, .91 (n = 139), Vocational Purpose .97 (n = 136), and Relationship .96 (n = 137) (note that the sample sizes for the subscales vary slightly due to missing data). Test–retest statistics were also excellent with n = 26 individuals from the original sample having usable data. Re-test correlations were as follows for each subscale: Hope .95, Vocational Empowerment .94, Acceptance, .93, Vocational Purpose .85, and Relationship .96, suggesting good stability over time. We examined the inter-items correlations and item-to-total scale correlations. We recommend dropping items that were deemed redundant; items suggested for retention are bolded in Table 2.

Next, we conducted analyses to determine if our initial concepts held up in the five subscales and to further understand the underlying constructs. Using principal components analysis with an oblique rotation, we requested a five-factor solution to parallel the number of factors that guided our original item development. Results of the PCA extracted 5 factors accounting for 81% of the variance. However, we did not find the solution to be consistent with our original constructs and related subscales. Subsequently, we conducted a principal components analysis with varimax rotation, allowing the procedure to default to the optimal number of factors. In doing so, we found two factors that accounted for 74% of the variance. The two factors fell largely into two subscales that we named the Empowering Relationship and Job Promoting Strategies, respectively. Items in the first factor tapped into constructs such as hope, belief, a trusting relationship, being seen as a person and not a diagnosis, choice, and self-determination. The second subscale consisted of Job Promoting Strategies, such as helping the person learn about how to cope on the job, learn to ask for accommodations, and learn about disclosure. Items that did not load clearly onto one of the two factors were marked for deletion and are not bolded in Table 2.

Finally, we examined the relationship between the VPRS subscales and the WAI to assess concurrent validity. Correlations were relatively high, .75–.88, for the subscale correlations. The total WAI correlated .91 with VPRS Relationship subscale, re-affirming the overriding importance of the relationship.

Discussion

We began this instrument development research with the intention of adding to the available empirical information about the competencies and behaviors that are important for employment specialists and that may enable them to promote vocational recovery (Corbière et al. 2014, 2017; Dreher et al. 2010; Taylor and Bond 2014). We considered it both critical and unique that in this study we sought out and incorporated the views of service recipients and specialists about those needed competencies. Research to date on vocational outcomes has focused heavily on programmatic structures and processes of providing supported employment, which has resulted in the development and large-scale testing of the IPS model (Drake et al. 2016). However, little knowledge exists about the role of the “non-specific” effects of the specialist or specialist competencies that are predictive of employment outcomes. Through a variety of conceptual and developmental activities to develop and vet candidate items, we ultimately tested 34 questions pertaining to vocational promoting strategies, relationship variables, and behaviors using an online survey. Our findings suggest a more parsimonious set of items that broadly tap and affirm the importance of the Empowering Relationship and Job Promoting Strategies.

We can infer from these data that what one might consider the more practical strategies related to working, such as being guided to ask for accommodations at work, may be perceived as less critical than the role of the specialist in instilling hope for a better vocational future. Having a trusting relationship, being seen as a person and not a diagnosis, and being open minded about clients’ vocational aspirations, are also critical. We were somewhat surprised, based on our anecdotal knowledge of individuals’ concerns, that the nature of the working relationship between specialist and client may be more important than individual strategies.

Items that were endorsed by recipients of employment services can serve as a blueprint for the development of employment specialist and vocational counselor skills and behaviors. Training efforts for employment specialists must focus not simply on the mechanics and technicalities of job development and placement, but squarely on instilling hope and empowerment. A focus on the “non-specific” aspects of the client-specialist relationship is essential, including the core conditions such as empathy, respect, and the other factors articulated decades ago by Carl Rogers (1975, 2007) and operationalized by others (e.g. Truax and Carkhuff 2007). These conditions, in addition to hope-inspiring strategies, may be particularly essential ingredients, not just in recovery overall (as noted by Anthony et al. 2002), but in vocational and employment services as well. Most importantly, they cannot be assumed to be in a specialist’s existing cache of competencies by default and must be assessed, trained to, supervised, and coached.

Conclusions

Using the input of both people with psychiatric disabilities receiving employment services, and employment specialists serving this population, we developed and refined a measure of professional competencies needed to promote better vocational outcomes. These competencies are related to the employment specialists’ capacity to promote hope for a better vocational future, empowerment in relation to one’s vocational abilities, self-acceptance of one’s vocational strengths and history, and a sense of purpose that can be derived from work. Items tapping the specialist’s core interpersonal skills and the resulting working alliance are a key component of these competencies, as was found by Corbière et al. (2017). Both can be used as blueprints for training employment specialists to provide more effective vocational services, especially because they transcend models of employment services. Further psychometric testing of this measure is warranted to determine its ability to predict employment outcomes.